immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

JS heap out of memory - library job #3899

New Issue
Closed
opened 2026-02-05 09:25:53 +03:00 by OVERLORD · 18 comments
OVERLORD commented 2026-02-05 09:25:53 +03:00
Owner
Copy Link

Originally created by @aks-cadesign on GitHub (Jul 17, 2024).

The bug

library job failed after adding external library

The OS that Immich Server is running on

Ubuntu 24.04 LTS

Version of Immich Server

v1.108.0

Version of Immich Mobile App

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    #extends:
      #file: hwaccel.transcoding.yml
      #service: quicksync # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - ${EXTERNAL_PATH_CLIENTS}:/usr/src/app/external:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

UPLOAD_LOCATION=./library
DB_DATA_LOCATION=./postgres
IMMICH_VERSION=release
DB_PASSWORD=password
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
EXTERNAL_PATH_CLIENTS=/mnt/media
NODE_OPTIONS=--max-old-space-size=8196

Reproduction steps

1. followed the guide to add external library: https://immich.app/docs/guides/external-library/
2. library job failed

Relevant log output

[Nest] 7  - 07/17/2024, 12:36:34 PM     LOG [Microservices:LibraryService] Refreshing library: 3cb44023-78b5-4ee1-97eb-e77ae17a6955
[Nest] 17  - 07/17/2024, 12:39:01 PM     LOG [Api:AuditService~ydncl54g] Found 0 original files, 0 thumbnails, 0 encoded videos, 0 profile files
[Nest] 17  - 07/17/2024, 12:39:01 PM     LOG [Api:AuditService~ydncl54g] Found 0 assets, 1 users, 0 people
node:events:497
      throw er; // Unhandled 'error' event
      ^
Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory
    at [kOnExit] (node:internal/worker:313:26)
    at Worker.<computed>.onexit (node:internal/worker:229:20)
Emitted 'error' event on Worker instance at:
    at [kOnExit] (node:internal/worker:313:12)
    at Worker.<computed>.onexit (node:internal/worker:229:20) {
  code: 'ERR_WORKER_OUT_OF_MEMORY'
}

Additional information

at first, it crashed around 4.3GB RAM usage, so I added 'NODE_OPTIONS=--max-old-space-size=8196' env variable but I get the same outcome with increased memory usage (9.4GB).

Originally created by @aks-cadesign on GitHub (Jul 17, 2024). ### The bug library job failed after adding external library ### The OS that Immich Server is running on Ubuntu 24.04 LTS ### Version of Immich Server v1.108.0 ### Version of Immich Mobile App - ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} #extends: #file: hwaccel.transcoding.yml #service: quicksync # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - ${EXTERNAL_PATH_CLIENTS}:/usr/src/app/external:ro env_file: - stack.env ports: - 2283:3001 depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache env_file: - stack.env restart: always redis: container_name: immich_redis image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b healthcheck: test: redis-cli ping || exit 1 restart: always database: container_name: immich_postgres image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' volumes: - ${DB_DATA_LOCATION}:/var/lib/postgresql/data healthcheck: test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 interval: 5m start_interval: 30s start_period: 5m command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"] restart: always volumes: model-cache: ``` ### Your .env content ```Shell UPLOAD_LOCATION=./library DB_DATA_LOCATION=./postgres IMMICH_VERSION=release DB_PASSWORD=password DB_USERNAME=postgres DB_DATABASE_NAME=immich EXTERNAL_PATH_CLIENTS=/mnt/media NODE_OPTIONS=--max-old-space-size=8196 ``` ### Reproduction steps ```bash 1. followed the guide to add external library: https://immich.app/docs/guides/external-library/ 2. library job failed ``` ### Relevant log output ```shell [Nest] 7 - 07/17/2024, 12:36:34 PM LOG [Microservices:LibraryService] Refreshing library: 3cb44023-78b5-4ee1-97eb-e77ae17a6955 [Nest] 17 - 07/17/2024, 12:39:01 PM LOG [Api:AuditService~ydncl54g] Found 0 original files, 0 thumbnails, 0 encoded videos, 0 profile files [Nest] 17 - 07/17/2024, 12:39:01 PM LOG [Api:AuditService~ydncl54g] Found 0 assets, 1 users, 0 people node:events:497 throw er; // Unhandled 'error' event ^ Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory at [kOnExit] (node:internal/worker:313:26) at Worker.<computed>.onexit (node:internal/worker:229:20) Emitted 'error' event on Worker instance at: at [kOnExit] (node:internal/worker:313:12) at Worker.<computed>.onexit (node:internal/worker:229:20) { code: 'ERR_WORKER_OUT_OF_MEMORY' } ``` ### Additional information at first, it crashed around 4.3GB RAM usage, so I added 'NODE_OPTIONS=--max-old-space-size=8196' env variable but I get the same outcome with increased memory usage (9.4GB).
OVERLORD added the 🗄️server label 2026-02-05 09:25:53 +03:00
OVERLORD commented 2026-02-05 09:25:55 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Jul 17, 2024):

Any thoughts on this @zackpollard?

@alextran1502 commented on GitHub (Jul 17, 2024): Any thoughts on this @zackpollard?
OVERLORD commented 2026-02-05 09:25:59 +03:00
Author
Owner
Copy Link

@zackpollard commented on GitHub (Jul 17, 2024):

How big is your external library? Count of assets is the main thing

@zackpollard commented on GitHub (Jul 17, 2024): How big is your external library? Count of assets is the main thing
OVERLORD commented 2026-02-05 09:26:01 +03:00
Author
Owner
Copy Link

@aks-cadesign commented on GitHub (Jul 17, 2024):

19.903 files, 1.450 folders, 145GB

takes around 6 minutes until it crashes:

image

@aks-cadesign commented on GitHub (Jul 17, 2024): 19.903 files, 1.450 folders, 145GB takes around 6 minutes until it crashes: ![image](https://github.com/user-attachments/assets/7b856427-0928-4788-99f6-c5fb26d4f7cf)
OVERLORD commented 2026-02-05 09:26:02 +03:00
Author
Owner
Copy Link

@m4ntic0r commented on GitHub (Jul 21, 2024):

not a lot of files and folders for this error. i can only share my experiences.
at the moment i have 10 libraries (~600 folders each) and 1.35 million assets.
on a system with 8gb ram i had problem if a single library gets around 250k+ assets. so be sure you are at ~200k and limit concurrent worker threads for library to 1!
i then had never the "JS heap out of memory" again. i have no special variables @.env etc.. only default stuff.

because your numbers are much smaller than that, i dont have a clue what the problem is. but try concurrent librarys workers set to 1. i dont know why default setting is 5. in my opinion its too high, because a lot of memory stuff is not fixed at the moment. and even for my over millions assets a setting of 1 is enough.

@m4ntic0r commented on GitHub (Jul 21, 2024): not a lot of files and folders for this error. i can only share my experiences. at the moment i have 10 libraries (~600 folders each) and 1.35 million assets. on a system with 8gb ram i had problem if a single library gets around 250k+ assets. so be sure you are at ~200k and limit concurrent worker threads for library to 1! i then had never the "JS heap out of memory" again. i have no special variables @.env etc.. only default stuff. because your numbers are much smaller than that, i dont have a clue what the problem is. but try concurrent librarys workers set to 1. i dont know why default setting is 5. in my opinion its too high, because a lot of memory stuff is not fixed at the moment. and even for my over millions assets a setting of 1 is enough.
OVERLORD commented 2026-02-05 09:26:05 +03:00
Author
Owner
Copy Link

@aks-cadesign commented on GitHub (Aug 5, 2024):

I moved to a different machine with more RAM and changed NODE_OPTIONS environment variable to:
--max-old-space-size=49152
and got a new error this time

node:internal/event_target:1094
process.nextTick(() => { throw err; });
^
RangeError [Error]: Map maximum size exceeded
at Map.set ()
at EntryFilter._createIndexRecord (/usr/src/app/node_modules/fast-glob/out/providers/filters/entry.js:37:20)
at EntryFilter._filter (/usr/src/app/node_modules/fast-glob/out/providers/filters/entry.js:29:18)
at /usr/src/app/node_modules/fast-glob/out/providers/filters/entry.js:13:32
at Object.isAppliedFilter (/usr/src/app/node_modules/@nodelib/fs.walk/out/readers/common.js:12:31)
at AsyncReader._handleEntry (/usr/src/app/node_modules/@nodelib/fs.walk/out/readers/async.js:86:20)
at /usr/src/app/node_modules/@nodelib/fs.walk/out/readers/async.js:65:22
at callSuccessCallback (/usr/src/app/node_modules/@nodelib/fs.scandir/out/providers/async.js:103:5)
at /usr/src/app/node_modules/@nodelib/fs.scandir/out/providers/async.js:38:13
at end (/usr/src/app/node_modules/run-parallel/index.js:21:15)
Emitted 'error' event on Worker instance at:
at [kOnErrorMessage] (node:internal/worker:326:10)
at [kOnMessage] (node:internal/worker:337:37)
at MessagePort. (node:internal/worker:232:57)
at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20)
at MessagePort. (node:internal/per_context/messageport:23:28)

now running version v1.111.0
RAM usage peaked around 42GB.

concurrent library workers has been set to 1 even before opening this issue.

another thing I noticed: nothing is being added to generate thumbnails or other jobs and it's just the 1 library job running.

@aks-cadesign commented on GitHub (Aug 5, 2024): I moved to a different machine with more RAM and changed NODE_OPTIONS environment variable to: --max-old-space-size=49152 and got a new error this time node:internal/event_target:1094 process.nextTick(() => { throw err; }); ^ RangeError [Error]: Map maximum size exceeded at Map.set (<anonymous>) at EntryFilter._createIndexRecord (/usr/src/app/node_modules/fast-glob/out/providers/filters/entry.js:37:20) at EntryFilter._filter (/usr/src/app/node_modules/fast-glob/out/providers/filters/entry.js:29:18) at /usr/src/app/node_modules/fast-glob/out/providers/filters/entry.js:13:32 at Object.isAppliedFilter (/usr/src/app/node_modules/@nodelib/fs.walk/out/readers/common.js:12:31) at AsyncReader._handleEntry (/usr/src/app/node_modules/@nodelib/fs.walk/out/readers/async.js:86:20) at /usr/src/app/node_modules/@nodelib/fs.walk/out/readers/async.js:65:22 at callSuccessCallback (/usr/src/app/node_modules/@nodelib/fs.scandir/out/providers/async.js:103:5) at /usr/src/app/node_modules/@nodelib/fs.scandir/out/providers/async.js:38:13 at end (/usr/src/app/node_modules/run-parallel/index.js:21:15) Emitted 'error' event on Worker instance at: at [kOnErrorMessage] (node:internal/worker:326:10) at [kOnMessage] (node:internal/worker:337:37) at MessagePort.<anonymous> (node:internal/worker:232:57) at [nodejs.internal.kHybridDispatch] (node:internal/event_target:820:20) at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28) now running version v1.111.0 RAM usage peaked around 42GB. concurrent library workers has been set to 1 even before opening this issue. another thing I noticed: nothing is being added to generate thumbnails or other jobs and it's just the 1 library job running.
OVERLORD commented 2026-02-05 09:26:07 +03:00
Author
Owner
Copy Link

@zackpollard commented on GitHub (Aug 5, 2024):

Could you please run the following command in the folder your library is in on the host machine and post the result here or DM it to me on discord?

ls -lR /path/to/folder | grep '^l'
@zackpollard commented on GitHub (Aug 5, 2024): Could you please run the following command in the folder your library is in on the host machine and post the result here or DM it to me on discord? ```bash ls -lR /path/to/folder | grep '^l' ```
OVERLORD commented 2026-02-05 09:26:08 +03:00
Author
Owner
Copy Link

@kdybicz commented on GitHub (Sep 5, 2024):

I'm facing the same issue with an immich-server limited to 3GB and with the external library with around 150k assets :/

@kdybicz commented on GitHub (Sep 5, 2024): I'm facing the same issue with an `immich-server` limited to 3GB and with the external library with around 150k assets :/
OVERLORD commented 2026-02-05 09:26:09 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Sep 6, 2024):

@kdybicz Are you on 1.113.0 or later? That release lowered RAM usage when scanning large libraries.

@mertalev commented on GitHub (Sep 6, 2024): @kdybicz Are you on 1.113.0 or later? That release lowered RAM usage when scanning large libraries.
OVERLORD commented 2026-02-05 09:26:10 +03:00
Author
Owner
Copy Link

@kdybicz commented on GitHub (Sep 6, 2024):

@mertalev I've tested it yesterday with 1.113.1

@kdybicz commented on GitHub (Sep 6, 2024): @mertalev I've tested it yesterday with `1.113.1`
OVERLORD commented 2026-02-05 09:26:11 +03:00
Author
Owner
Copy Link

@danieldietzler commented on GitHub (Sep 23, 2024):

cc @etnoy

@danieldietzler commented on GitHub (Sep 23, 2024): cc @etnoy
OVERLORD commented 2026-02-05 09:26:12 +03:00
Author
Owner
Copy Link

@etnoy commented on GitHub (Oct 8, 2024):

Is this still an issue after #7934 is merged? We've also refactored it with #12220 so maybe it's better now?

@etnoy commented on GitHub (Oct 8, 2024): Is this still an issue after #7934 is merged? We've also refactored it with #12220 so maybe it's better now?
OVERLORD commented 2026-02-05 09:26:13 +03:00
Author
Owner
Copy Link

@vs30217 commented on GitHub (Nov 21, 2024):

I am still facing this timeout issue, my external library is currently having 750,905 assets and now all the jobs are getting failed due to connection timeout and following error occurs-

microservices worker error: Error [ERR_WORKER_OUT_OF_MEMORY]: worker terminated due to reaching memory limit: JS heap out of memory
my system is having 128 GB ram out of which 115 GB is available.

@vs30217 commented on GitHub (Nov 21, 2024): I am still facing this timeout issue, my external library is currently having 750,905 assets and now all the jobs are getting failed due to connection timeout and following error occurs- microservices worker error: Error [ERR_WORKER_OUT_OF_MEMORY]: worker terminated due to reaching memory limit: JS heap out of memory my system is having 128 GB ram out of which 115 GB is available.
OVERLORD commented 2026-02-05 09:26:14 +03:00
Author
Owner
Copy Link

@etnoy commented on GitHub (Nov 21, 2024):

Can you retry with verbose logging? Please post the full log but check for private information first

@etnoy commented on GitHub (Nov 21, 2024): Can you retry with verbose logging? Please post the full log but check for private information first
OVERLORD commented 2026-02-05 09:26:15 +03:00
Author
Owner
Copy Link

@vk496 commented on GitHub (Dec 4, 2024):

Can confirm that server stop crashing at 8Gb of RAM usage with NODE_OPTIONS=--max-old-space-size=X where X is the limit I set (the same as docker RAM limit).

Before this, the server was crashing even when the docker limit was on 20Gi

@vk496 commented on GitHub (Dec 4, 2024): Can confirm that `server` stop crashing at 8Gb of RAM usage with `NODE_OPTIONS=--max-old-space-size=X` where X is the limit I set (the same as docker RAM limit). Before this, the `server` was crashing even when the docker limit was on 20Gi
OVERLORD commented 2026-02-05 09:26:16 +03:00
Author
Owner
Copy Link

@danieldietzler commented on GitHub (Apr 1, 2025):

This should be fixed with recent library changes. Can you verify?

@danieldietzler commented on GitHub (Apr 1, 2025): This should be fixed with recent library changes. Can you verify?
OVERLORD commented 2026-02-05 09:26:17 +03:00
Author
Owner
Copy Link

@etnoy commented on GitHub (Apr 1, 2025):

I've tested >10M assets, though with 8gb max node size (not sure if that's needed) without crashing . Closing

@etnoy commented on GitHub (Apr 1, 2025): I've tested >10M assets, though with 8gb max node size (not sure if that's needed) without crashing . Closing
OVERLORD commented 2026-02-05 09:26:17 +03:00
Author
Owner
Copy Link

@the-marenga commented on GitHub (Jul 2, 2025):

Just ran into this in the latest version (v1.135.3). Added an external library with ~11M assets and it crashed with Worker terminated due to reaching memory limit. After setting NODE_OPTIONS=--max-old-space-size=61440 (60GB), I get the same map size exceeded error as mentioned in one of the replies above:

microservices worker error: RangeError: Map maximum size exceeded, stack: RangeError: Map maximum size exceeded
    at Map.set (<anonymous>)
    at EntryFilter._createIndexRecord (/usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:41:20)
    at EntryFilter._filter (/usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:33:18)
    at /usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:21:32
    at Object.isAppliedFilter (/usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/common.js:12:31)
    at AsyncReader._handleEntry (/usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/async.js:86:20)
    at /usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/async.js:65:22
    at callSuccessCallback (/usr/lib/immich/app/server/node_modules/@nodelib/fs.scandir/out/providers/async.js:103:5)
    at /usr/lib/immich/app/server/node_modules/@nodelib/fs.scandir/out/providers/async.js:38:13
    at end (/usr/lib/immich/app/server/node_modules/run-parallel/index.js:21:15)
microservices worker exited with code 1

The server never exceeds 7GB of memory usage during the scan.

I've tested >10M assets, though with 8gb max node size (not sure if that's needed) without crashing

My best guess as to why this does not work for me is that my files are not divided into folders at all, just 11Mio files top level in the library folder. Just guessing based on "fast-glob" in the stacktracke, my instance crashes when applying the exclusion pattern. If that > 10M test was done with files in seperate folders and the exclusion check is done per folder, that would have had less files to deal with at once, thus never running into whatever limit gets exceeded here.

Setting External Libraries concurrency to 1, and removing all exclusion patterns yielded no difference.

In case anyone wants to reproduce this and this is indeed related to filenames, all files are all exactly 30 characters long with 22 extra characters preceeding for the full path.

I am aware, that this is a somewhat unusual library structure and size and I was not actually planning to use this library long term, but I just wanted to report, that this still happens

EDIT: Just had a look at the implementation of map in node/v8: https://stackoverflow.com/a/54466812 . Seems like the max limit is just 2^24, or ~16mio elements. Since maps grow before they are full, 11mio might try to grow the map beyond that cap and crash like here. fast-globs safe limit is probably < 8mio files at once, which means immichs limit is the same per folder

@the-marenga commented on GitHub (Jul 2, 2025): Just ran into this in the latest version (v1.135.3). Added an external library with ~11M assets and it crashed with ` Worker terminated due to reaching memory limit`. After setting `NODE_OPTIONS=--max-old-space-size=61440` (60GB), I get the same map size exceeded error as mentioned in one of the replies above: ``` microservices worker error: RangeError: Map maximum size exceeded, stack: RangeError: Map maximum size exceeded at Map.set (<anonymous>) at EntryFilter._createIndexRecord (/usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:41:20) at EntryFilter._filter (/usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:33:18) at /usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:21:32 at Object.isAppliedFilter (/usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/common.js:12:31) at AsyncReader._handleEntry (/usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/async.js:86:20) at /usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/async.js:65:22 at callSuccessCallback (/usr/lib/immich/app/server/node_modules/@nodelib/fs.scandir/out/providers/async.js:103:5) at /usr/lib/immich/app/server/node_modules/@nodelib/fs.scandir/out/providers/async.js:38:13 at end (/usr/lib/immich/app/server/node_modules/run-parallel/index.js:21:15) microservices worker exited with code 1 ``` The server never exceeds 7GB of memory usage during the scan. > I've tested >10M assets, though with 8gb max node size (not sure if that's needed) without crashing My best guess as to why this does not work for me is that my files are not divided into folders at all, just 11Mio files top level in the library folder. Just guessing based on "fast-glob" in the stacktracke, my instance crashes when applying the exclusion pattern. If that > 10M test was done with files in seperate folders and the exclusion check is done per folder, that would have had less files to deal with at once, thus never running into whatever limit gets exceeded here. Setting `External Libraries concurrency` to 1, and removing all exclusion patterns yielded no difference. In case anyone wants to reproduce this and this is indeed related to filenames, all files are all exactly 30 characters long with 22 extra characters preceeding for the full path. I am aware, that this is a somewhat unusual library structure and size and I was not actually planning to use this library long term, but I just wanted to report, that this still happens EDIT: Just had a look at the implementation of map in node/v8: https://stackoverflow.com/a/54466812 . Seems like the max limit is just 2^24, or ~16mio elements. Since maps grow before they are full, 11mio might try to grow the map beyond that cap and crash like here. fast-globs safe limit is probably < 8mio files at once, which means immichs limit is the same per folder
OVERLORD commented 2026-02-05 09:26:18 +03:00
Author
Owner
Copy Link

@etnoy commented on GitHub (Jul 3, 2025):

Just ran into this in the latest version (v1.135.3). Added an external library with ~11M assets and it crashed with Worker terminated due to reaching memory limit. After setting NODE_OPTIONS=--max-old-space-size=61440 (60GB), I get the same map size exceeded error as mentioned in one of the replies above:

microservices worker error: RangeError: Map maximum size exceeded, stack: RangeError: Map maximum size exceeded
    at Map.set (<anonymous>)
    at EntryFilter._createIndexRecord (/usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:41:20)
    at EntryFilter._filter (/usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:33:18)
    at /usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:21:32
    at Object.isAppliedFilter (/usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/common.js:12:31)
    at AsyncReader._handleEntry (/usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/async.js:86:20)
    at /usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/async.js:65:22
    at callSuccessCallback (/usr/lib/immich/app/server/node_modules/@nodelib/fs.scandir/out/providers/async.js:103:5)
    at /usr/lib/immich/app/server/node_modules/@nodelib/fs.scandir/out/providers/async.js:38:13
    at end (/usr/lib/immich/app/server/node_modules/run-parallel/index.js:21:15)
microservices worker exited with code 1

The server never exceeds 7GB of memory usage during the scan.

I've tested >10M assets, though with 8gb max node size (not sure if that's needed) without crashing

My best guess as to why this does not work for me is that my files are not divided into folders at all, just 11Mio files top level in the library folder. Just guessing based on "fast-glob" in the stacktracke, my instance crashes when applying the exclusion pattern. If that > 10M test was done with files in seperate folders and the exclusion check is done per folder, that would have had less files to deal with at once, thus never running into whatever limit gets exceeded here.

Setting External Libraries concurrency to 1, and removing all exclusion patterns yielded no difference.

In case anyone wants to reproduce this and this is indeed related to filenames, all files are all exactly 30 characters long with 22 extra characters preceeding for the full path.

I am aware, that this is a somewhat unusual library structure and size and I was not actually planning to use this library long term, but I just wanted to report, that this still happens

EDIT: Just had a look at the implementation of map in node/v8: https://stackoverflow.com/a/54466812 . Seems like the max limit is just 2^24, or ~16mio elements. Since maps grow before they are full, 11mio might try to grow the map beyond that cap and crash like here. fast-globs safe limit is probably < 8mio files at once, which means immichs limit is the same per folder

I don't see any way to make this work. I've tested up to 10M assets but they were not in a single dir. You are running into many limits, including those of the underlying os. You have to split things into subfolders to have any chance, 10M files in a single dir is hard.

@etnoy commented on GitHub (Jul 3, 2025): > Just ran into this in the latest version (v1.135.3). Added an external library with ~11M assets and it crashed with ` Worker terminated due to reaching memory limit`. After setting `NODE_OPTIONS=--max-old-space-size=61440` (60GB), I get the same map size exceeded error as mentioned in one of the replies above: > > ``` > microservices worker error: RangeError: Map maximum size exceeded, stack: RangeError: Map maximum size exceeded > at Map.set (<anonymous>) > at EntryFilter._createIndexRecord (/usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:41:20) > at EntryFilter._filter (/usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:33:18) > at /usr/lib/immich/app/server/node_modules/fast-glob/out/providers/filters/entry.js:21:32 > at Object.isAppliedFilter (/usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/common.js:12:31) > at AsyncReader._handleEntry (/usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/async.js:86:20) > at /usr/lib/immich/app/server/node_modules/@nodelib/fs.walk/out/readers/async.js:65:22 > at callSuccessCallback (/usr/lib/immich/app/server/node_modules/@nodelib/fs.scandir/out/providers/async.js:103:5) > at /usr/lib/immich/app/server/node_modules/@nodelib/fs.scandir/out/providers/async.js:38:13 > at end (/usr/lib/immich/app/server/node_modules/run-parallel/index.js:21:15) > microservices worker exited with code 1 > ``` > > The server never exceeds 7GB of memory usage during the scan. > > > I've tested >10M assets, though with 8gb max node size (not sure if that's needed) without crashing > > My best guess as to why this does not work for me is that my files are not divided into folders at all, just 11Mio files top level in the library folder. Just guessing based on "fast-glob" in the stacktracke, my instance crashes when applying the exclusion pattern. If that > 10M test was done with files in seperate folders and the exclusion check is done per folder, that would have had less files to deal with at once, thus never running into whatever limit gets exceeded here. > > Setting `External Libraries concurrency` to 1, and removing all exclusion patterns yielded no difference. > > In case anyone wants to reproduce this and this is indeed related to filenames, all files are all exactly 30 characters long with 22 extra characters preceeding for the full path. > > I am aware, that this is a somewhat unusual library structure and size and I was not actually planning to use this library long term, but I just wanted to report, that this still happens > > EDIT: Just had a look at the implementation of map in node/v8: https://stackoverflow.com/a/54466812 . Seems like the max limit is just 2^24, or ~16mio elements. Since maps grow before they are full, 11mio might try to grow the map beyond that cap and crash like here. fast-globs safe limit is probably < 8mio files at once, which means immichs limit is the same per folder I don't see any way to make this work. I've tested up to 10M assets but they were not in a single dir. You are running into many limits, including those of the underlying os. You have to split things into subfolders to have any chance, 10M files in a single dir is hard.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label 🗄️server
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#3899
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?