immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

[Bug] immich microservice memory leak kills host #3108

New Issue
Closed
opened 2026-02-05 07:43:50 +03:00 by OVERLORD · 35 comments
OVERLORD commented 2026-02-05 07:43:50 +03:00
Owner
Copy Link

Originally created by @Dunky-Z on GitHub (May 13, 2024).

The bug

Today, I encountered the same issue #5283 . After bulk importing around 10,000 photos(By setting the external library), I suddenly couldn't remotely connect to my NAS host. Initially, I suspected that the ML service was causing high memory usage during facial recognition, so I forcibly restarted the NAS and reconfigured concurrency settings, setting all jobs to run on a single thread. I didn't start all the jobs at once; instead, I sequentially ran FACE DETECTION and then GENERATE THUMBNAILS. There were no abnormalities during the FACE DETECTION process, indicating that the issue wasn't caused by the ML operations. After completing all FACE DETECTIONS, I initiated the GENERATE THUMBNAILS task, which ran slowly due to the large image sizes. Without continuous monitoring, upon returning to check on the NAS, I found it was unreachable remotely, with the system log displaying the following error messages:

[17452.279485] Out of memory: Killed process 24344 (immich_microser) total-vm:14375660kB, anon-rss:8827768kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:23516kB oom_score_adj:0
[17653.570542] Out of memory: Killed process 24972 (immich_microser) total-vm:13119600kB, anon-rss:8825796kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:20652kB oom_score_adj:0
[17848.345248] Out of memory: Killed process 26162 (immich_microser) total-vm:19086052kB, anon-rss:8709080kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:31688kB oom_score_adj:0
[17932.668353] Out of memory: Killed process 27348 (immich_microser) total-vm:13774836kB, anon-rss:8759228kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:22648kB oom_score_adj:0
[18001.034451] Out of memory: Killed process 27902 (immich_microser) total-vm:14749840kB, anon-rss:9111036kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:25104kB oom_score_adj:0

This suggests that there might be a memory leak issue within the immich_microser service. I would greatly appreciate any advice on how to address this problem.

The OS that Immich Server is running on

Debian(OMV)

Version of Immich Server

OpenMediaVault 5

Version of Immich Mobile App

v1.103.1

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - "/srv/share/album:/mnt/media/share/album:ro"
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - "/srv/share/album:/mnt/media/share/album:ro"
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - "/srv/appdata/immich/immich-cache:/cache"
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - "/srv/appdata/immich/immich-postgresql/data:/var/lib/postgresql/data"
    restart: always

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Add an external library that contains the over 10000 pics
2. Wait for immich to generate thumbnails for the photo

Relevant log output

No response

Additional information

No response

Originally created by @Dunky-Z on GitHub (May 13, 2024). ### The bug Today, I encountered the same issue #5283 . After bulk importing around 10,000 photos(By setting the external library), I suddenly couldn't remotely connect to my NAS host. Initially, I suspected that the ML service was causing high memory usage during facial recognition, so I forcibly restarted the NAS and reconfigured concurrency settings, setting all jobs to run on a single thread. I didn't start all the jobs at once; instead, I sequentially ran FACE DETECTION and then GENERATE THUMBNAILS. There were no abnormalities during the FACE DETECTION process, indicating that the issue wasn't caused by the ML operations. After completing all FACE DETECTIONS, I initiated the GENERATE THUMBNAILS task, which ran slowly due to the large image sizes. Without continuous monitoring, upon returning to check on the NAS, I found it was unreachable remotely, with the system log displaying the following error messages: ``` [17452.279485] Out of memory: Killed process 24344 (immich_microser) total-vm:14375660kB, anon-rss:8827768kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:23516kB oom_score_adj:0 [17653.570542] Out of memory: Killed process 24972 (immich_microser) total-vm:13119600kB, anon-rss:8825796kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:20652kB oom_score_adj:0 [17848.345248] Out of memory: Killed process 26162 (immich_microser) total-vm:19086052kB, anon-rss:8709080kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:31688kB oom_score_adj:0 [17932.668353] Out of memory: Killed process 27348 (immich_microser) total-vm:13774836kB, anon-rss:8759228kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:22648kB oom_score_adj:0 [18001.034451] Out of memory: Killed process 27902 (immich_microser) total-vm:14749840kB, anon-rss:9111036kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:25104kB oom_score_adj:0 ``` This suggests that there might be a memory leak issue within the immich_microser service. I would greatly appreciate any advice on how to address this problem. ### The OS that Immich Server is running on Debian(OMV) ### Version of Immich Server OpenMediaVault 5 ### Version of Immich Mobile App v1.103.1 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ['start.sh', 'immich'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - "/srv/share/album:/mnt/media/share/album:ro" env_file: - .env ports: - 2283:3001 depends_on: - redis - database restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding # file: hwaccel.transcoding.yml # service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding command: ['start.sh', 'microservices'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - "/srv/share/album:/mnt/media/share/album:ro" env_file: - .env depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - "/srv/appdata/immich/immich-cache:/cache" env_file: - .env restart: always redis: container_name: immich_redis image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5 restart: always database: container_name: immich_postgres image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - "/srv/appdata/immich/immich-postgresql/data:/var/lib/postgresql/data" restart: always ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=./library # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secret for postgres. You should change it to a random password DB_PASSWORD=postgres # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. Add an external library that contains the over 10000 pics 2. Wait for immich to generate thumbnails for the photo ``` ### Relevant log output _No response_ ### Additional information _No response_
OVERLORD commented 2026-02-05 07:43:55 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (May 13, 2024):

How much RAM does the server have, and how much is used just after starting thumbnail generation? An increase in memory usage for some period of time during thumbnail generation is normal because of memory fragmentation, but it should plateau after a certain point. If you have any RAW images, they will amplify this effect since they require much more memory.

@mertalev commented on GitHub (May 13, 2024): How much RAM does the server have, and how much is used just after starting thumbnail generation? An increase in memory usage for some period of time during thumbnail generation is normal because of memory fragmentation, but it should plateau after a certain point. If you have any RAW images, they will amplify this effect since they require much more memory.
OVERLORD commented 2026-02-05 07:43:59 +03:00
Author
Owner
Copy Link

@Dunky-Z commented on GitHub (May 13, 2024):

Thank you for your response. The server has a total of 16GB of RAM, and its typical daily memory usage hovers around 6GB. After initiating thumbnail generation, it doesn't immediately consume all memory but gradually fills it up until the program crashes. Your mention of RAW images did remind me that the majority of my gallery consists of RAW files. Is there a way to prevent excessive memory usage in this scenario? I have tried limiting the microservice using Cgroups, but when the limit is exceeded, the microservice restarts. Since I bind the Cgroup settings to the PID, once the microservice restarts, PID changed, the previous Cgroup configurations become ineffective.

@Dunky-Z commented on GitHub (May 13, 2024): Thank you for your response. The server has a total of 16GB of RAM, and its typical daily memory usage hovers around 6GB. After initiating thumbnail generation, it doesn't immediately consume all memory but gradually fills it up until the program crashes. Your mention of RAW images did remind me that the majority of my gallery consists of RAW files. Is there a way to prevent excessive memory usage in this scenario? I have tried limiting the microservice using Cgroups, but when the limit is exceeded, the microservice restarts. Since I bind the Cgroup settings to the PID, once the microservice restarts, PID changed, the previous Cgroup configurations become ineffective.
OVERLORD commented 2026-02-05 07:44:02 +03:00
Author
Owner
Copy Link

@Dunky-Z commented on GitHub (May 13, 2024):

Given the server's average performance, I have configured all my JOBS to use a single thread, and only one JOB is executed at a time. In theory, processing a single RAW file shouldn't require such an excessive amount of memory.

@Dunky-Z commented on GitHub (May 13, 2024): Given the server's average performance, I have configured all my JOBS to use a single thread, and only one JOB is executed at a time. In theory, processing a single RAW file shouldn't require such an excessive amount of memory.
OVERLORD commented 2026-02-05 07:44:11 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (May 13, 2024):

Hmm, that much of an increase is unexpected. It could be related to the issue behind #6542, or possibly #4391. Are there any errors in the microservices logs before the OOM error?

As far as limiting memory usage in the meantime, you can set a limit through Docker, which will force the container to be restarted after reaching a certain usage.

@mertalev commented on GitHub (May 13, 2024): Hmm, that much of an increase is unexpected. It could be related to the issue behind #6542, or possibly #4391. Are there any errors in the microservices logs before the OOM error? As far as limiting memory usage in the meantime, you can set a limit through Docker, which will force the container to be restarted after reaching a certain usage.
OVERLORD commented 2026-02-05 07:44:14 +03:00
Author
Owner
Copy Link

@Dunky-Z commented on GitHub (May 13, 2024):

Below are some logs from the microservices; perhaps they might provide some insight:

[Nest] 7  - 05/13/2024, 11:34:38 AM     LOG [EventRepository] WebSocket server initialized.
[Nest] 7  - 05/13/2024, 11:35:06 AM   ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: The input file contains an unsupported image format
[Nest] 7  - 05/13/2024, 11:35:06 AM   ERROR [JobService] Error: The input file contains an unsupported image format
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.resize (/usr/src/app/dist/repositories/media.repository.js:76:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:156:48)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:134:29)
    at async /usr/src/app/dist/services/job.service.js:149:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 05/13/2024, 11:35:06 AM   ERROR [JobService] Affected Job ID:
{
  "id": "5802ddbb-4e01-47fb-b21b-73bc0361f9e8"
}

[Nest] 7  - 05/13/2024, 11:35:07 AM   ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  Built with gcc 12 (Debian 12.2.0-14)
  Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input

[Nest] 7  - 05/13/2024, 11:35:07 AM   ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  Built with gcc 12 (Debian 12.2.0-14)
  Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header
upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input

    at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
    at ChildProcess.emit (node:events:518:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 7  - 05/13/2024, 11:35:07 AM   ERROR [JobService] Affected Job ID:
{
  "id": "ee18e5e8-5430-4fcb-aa10-ef4170e2b30e"
}

Limiting the container's resource usage has indeed helped me; it prevents my server from crashing. However, upon examining the microservices' logs, I noticed frequent restarts, indicating that the microservices are indeed exceeding the memory limit. Even after setting a 4GB memory limit, it's still insufficient.

Here's the relevant part of my Docker Compose configuration:

  immich-microservices:
    container_name: immich_microservices
    image: altran1502/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - "/root/sharedfolder/syncthing/Photo_Album:/mnt/media/Photo_Album:ro"
      - /etc/localtime:/etc/localtime:ro
    devices:
      - /dev/dri:/dev/dri
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 4G

[Nest] 7  - 05/13/2024, 11:35:19 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:35:33 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:35:45 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:35:58 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:36:11 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:36:53 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:37:06 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:37:19 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:37:34 AM     LOG [EventRepository] Initialized websocket server
[Nest] 7  - 05/13/2024, 11:37:47 AM     LOG [EventRepository] Initialized websocket server

I'm still hoping to identify and resolve the root cause entirely. If more information is needed, please let me know.

On a side note, how can I persistently save Immich logs? The log path isn't mentioned in the user documentation, so I haven't mapped any volumes, resulting in log loss upon container restarts.

Thank you very much！

@Dunky-Z commented on GitHub (May 13, 2024): Below are some logs from the microservices; perhaps they might provide some insight: ```bash [Nest] 7 - 05/13/2024, 11:34:38 AM LOG [EventRepository] WebSocket server initialized. [Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: The input file contains an unsupported image format [Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Error: The input file contains an unsupported image format at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19) at MediaRepository.resize (/usr/src/app/dist/repositories/media.repository.js:76:14) at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:156:48) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:134:29) at async /usr/src/app/dist/services/job.service.js:149:36 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) [Nest] 7 - 05/13/2024, 11:35:06 AM ERROR [JobService] Affected Job ID: { "id": "5802ddbb-4e01-47fb-b21b-73bc0361f9e8" } [Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Failed to execute job handler (thumbnailGeneration/generate-preview): Error: ffprobe exited with code 1 ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers Built with gcc 12 (Debian 12.2.0-14) Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 3.100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input [Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Error: ffprobe exited with code 1 ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers Built with gcc 12 (Debian 12.2.0-14) Configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Invalid sample size -1008 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x36da0190180] Error reading header upload/upload/98a61f7d-3957-465a-b946-79697c36d3b1/59/5d/595d905e-3bde-49ef-a699-17b677622b3c.mp4: Invalid data found when processing input at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22) at ChildProcess.emit (node:events:518:28) at ChildProcess._handle.onexit (node:internal/child_process:294:12) [Nest] 7 - 05/13/2024, 11:35:07 AM ERROR [JobService] Affected Job ID: { "id": "ee18e5e8-5430-4fcb-aa10-ef4170e2b30e" } ``` Limiting the container's resource usage has indeed helped me; it prevents my server from crashing. However, upon examining the microservices' logs, I noticed frequent restarts, indicating that the microservices are indeed exceeding the memory limit. Even after setting a 4GB memory limit, it's still insufficient. Here's the relevant part of my Docker Compose configuration: ```yaml immich-microservices: container_name: immich_microservices image: altran1502/immich-server:${IMMICH_VERSION:-release} command: ['start.sh', 'microservices'] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - "/root/sharedfolder/syncthing/Photo_Album:/mnt/media/Photo_Album:ro" - /etc/localtime:/etc/localtime:ro devices: - /dev/dri:/dev/dri env_file: - .env depends_on: - redis - database restart: always deploy: resources: limits: cpus: '2' memory: 4G ``` ```bash [Nest] 7 - 05/13/2024, 11:35:19 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:35:33 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:35:45 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:35:58 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:36:11 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:36:53 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:37:06 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:37:19 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:37:34 AM LOG [EventRepository] Initialized websocket server [Nest] 7 - 05/13/2024, 11:37:47 AM LOG [EventRepository] Initialized websocket server ``` I'm still hoping to identify and resolve the root cause entirely. If more information is needed, please let me know. On a side note, how can I persistently save Immich logs? The log path isn't mentioned in the user documentation, so I haven't mapped any volumes, resulting in log loss upon container restarts. Thank you very much！
OVERLORD commented 2026-02-05 07:44:18 +03:00
Author
Owner
Copy Link

@ceebu commented on GitHub (May 17, 2024):

I have the same problem (i5 8th gen server, running OMV 6) - am also Looking for a easy way to save immich logs -

@ceebu commented on GitHub (May 17, 2024): I have the same problem (i5 8th gen server, running OMV 6) - am also Looking for a easy way to save immich logs -
OVERLORD commented 2026-02-05 07:44:19 +03:00
Author
Owner
Copy Link

@RazerProof commented on GitHub (May 22, 2024):

screenshot
I can also confirm this issue is affecting the installation on my NAS. Limiting memory causes frequent restarts of the microservices container. I also tried setting the Generate Thumbnails Concurrency to 1, but it didn't help. I found the NAS started overutalising the drives as they were at 100% utilisation. I imagine it was paging out memory to disk; CPU utilization was around 40%. I have increased the RAM in the NAS to 32GB, which has solved the problem for me, and the microservices container now runs constantly, with no restarting at all. The microservices container stabilised at around 28GB, the NAS memory resource monitor shows almost all of that usage is cache memory, the container limit is set to 4GB.
After restarting the container, it takes about 15-20min to stabilise at this level. Disk usage is now around 40% and CPU is around 80%, generating thumbnails at around 30/min. This is with no other jobs running.
I have around 230k assets in the external library, 50%jpg, 50%RAW. I also tried this in docker on a server (TrueNAS scale) and experienced the same issue and outcome.
Overall, I love the way Immich is going. It is going to be a great product. Well done to everyone involved, and thanks for the great work.

@RazerProof commented on GitHub (May 22, 2024): ![screenshot](https://github.com/immich-app/immich/assets/166206048/c6baf535-5c4d-44b9-bc74-f853f090b6fb) I can also confirm this issue is affecting the installation on my NAS. Limiting memory causes frequent restarts of the microservices container. I also tried setting the Generate Thumbnails Concurrency to 1, but it didn't help. I found the NAS started overutalising the drives as they were at 100% utilisation. I imagine it was paging out memory to disk; CPU utilization was around 40%. I have increased the RAM in the NAS to 32GB, which has solved the problem for me, and the microservices container now runs constantly, with no restarting at all. The microservices container stabilised at around 28GB, the NAS memory resource monitor shows almost all of that usage is cache memory, the container limit is set to 4GB. After restarting the container, it takes about 15-20min to stabilise at this level. Disk usage is now around 40% and CPU is around 80%, generating thumbnails at around 30/min. This is with no other jobs running. I have around 230k assets in the external library, 50%jpg, 50%RAW. I also tried this in docker on a server (TrueNAS scale) and experienced the same issue and outcome. Overall, I love the way Immich is going. It is going to be a great product. Well done to everyone involved, and thanks for the great work.
OVERLORD commented 2026-02-05 07:44:24 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (May 22, 2024):

Thanks for the detailed info!

  • How many of those assets are RAW, if any?
  • Do you have /tmp configured to be in-memory?
  • Is the increase completely linear, or are there spikes?
  • Any errors in the logs?
@mertalev commented on GitHub (May 22, 2024): Thanks for the detailed info! - How many of those assets are RAW, if any? - Do you have `/tmp` configured to be in-memory? - Is the increase completely linear, or are there spikes? - Any errors in the logs?
OVERLORD commented 2026-02-05 07:44:26 +03:00
Author
Owner
Copy Link

@RazerProof commented on GitHub (May 22, 2024):

How many of those assets are RAW, if any? 50% of the images are RAW around 50GB each.
Do you have /tmp configured to be in-memory? No /tmp configured.
Is the increase completely linear, or are there spikes? Mostly linear overlayed with a small saw tooth shape.
image
image

I just restarted the container. That smooth section is unusual, and it will go back to the sawtooth.
Any errors in the logs? No Errors, running smoothly.

@RazerProof commented on GitHub (May 22, 2024): How many of those assets are RAW, if any? 50% of the images are RAW around 50GB each. Do you have /tmp configured to be in-memory? No /tmp configured. Is the increase completely linear, or are there spikes? Mostly linear overlayed with a small saw tooth shape. ![image](https://github.com/immich-app/immich/assets/166206048/260512a4-23f9-43f5-b0cf-4132354d1255) ![image](https://github.com/immich-app/immich/assets/166206048/8008378b-b4d8-4653-9fb0-7c39c16b34aa) I just restarted the container. That smooth section is unusual, and it will go back to the sawtooth. Any errors in the logs? No Errors, running smoothly.
OVERLORD commented 2026-02-05 07:44:28 +03:00
Author
Owner
Copy Link

@RazerProof commented on GitHub (May 22, 2024):

Below is an example of the errors I was receiving on the NAS when 4GB was installed and getting constant restarts of the microservices container.
Capture3

@RazerProof commented on GitHub (May 22, 2024): Below is an example of the errors I was receiving on the NAS when 4GB was installed and getting constant restarts of the microservices container. ![Capture3](https://github.com/immich-app/immich/assets/166206048/10f81e23-e79c-47fa-b679-28b97a6e2b19)
OVERLORD commented 2026-02-05 07:44:33 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (May 22, 2024):

I made a test image for microservices with a possible fix: ghcr.io/immich-app/immich-server:pr-9665. Would you be able to change your image to that and see if it affects RAM usage?

@mertalev commented on GitHub (May 22, 2024): I made a test image for microservices with a possible fix: `ghcr.io/immich-app/immich-server:pr-9665`. Would you be able to change your image to that and see if it affects RAM usage?
OVERLORD commented 2026-02-05 07:44:35 +03:00
Author
Owner
Copy Link

@RazerProof commented on GitHub (May 22, 2024):

Not good news I am afraid. The immich_microservices fails.

immich_microservices
date stream content
2024/05/23 01:09:52 stderr Microservices worker exited with code 1

2024/05/23 01:09:52 stderr }

2024/05/23 01:09:52 stderr routine: 'parserOpenTable'

2024/05/23 01:09:52 stderr line: '1381',
2024/05/23 01:09:52 stderr file: 'parse_relation.c',
2024/05/23 01:09:52 stderr constraint: undefined,
2024/05/23 01:09:52 stderr dataType: undefined,
2024/05/23 01:09:52 stderr column: undefined,
2024/05/23 01:09:52 stderr table: undefined,
2024/05/23 01:09:52 stderr schema: undefined,
2024/05/23 01:09:52 stderr where: undefined,
2024/05/23 01:09:52 stderr internalQuery: undefined,
2024/05/23 01:09:52 stderr internalPosition: undefined,
2024/05/23 01:09:52 stderr position: '128',
2024/05/23 01:09:52 stderr hint: undefined,
2024/05/23 01:09:52 stderr detail: undefined,
2024/05/23 01:09:52 stderr code: '42P01',
2024/05/23 01:09:52 stderr severity: 'ERROR',
2024/05/23 01:09:52 stderr length: 113,
2024/05/23 01:09:52 stderr },
2024/05/23 01:09:52 stderr routine: 'parserOpenTable'

2024/05/23 01:09:52 stderr line: '1381',
2024/05/23 01:09:52 stderr file: 'parse_relation.c',
2024/05/23 01:09:52 stderr constraint: undefined,
2024/05/23 01:09:52 stderr dataType: undefined,
2024/05/23 01:09:52 stderr column: undefined,
2024/05/23 01:09:52 stderr table: undefined,
2024/05/23 01:09:52 stderr schema: undefined,
2024/05/23 01:09:52 stderr where: undefined,
2024/05/23 01:09:52 stderr internalQuery: undefined,
2024/05/23 01:09:52 stderr internalPosition: undefined,
2024/05/23 01:09:52 stderr position: '128',
2024/05/23 01:09:52 stderr hint: undefined,
2024/05/23 01:09:52 stderr detail: undefined,
2024/05/23 01:09:52 stderr code: '42P01',
2024/05/23 01:09:52 stderr severity: 'ERROR',
2024/05/23 01:09:52 stderr length: 113,

@RazerProof commented on GitHub (May 22, 2024): Not good news I am afraid. The immich_microservices fails. immich_microservices date stream content 2024/05/23 01:09:52 stderr Microservices worker exited with code 1 2024/05/23 01:09:52 stderr } 2024/05/23 01:09:52 stderr routine: 'parserOpenTable' 2024/05/23 01:09:52 stderr line: '1381', 2024/05/23 01:09:52 stderr file: 'parse_relation.c', 2024/05/23 01:09:52 stderr constraint: undefined, 2024/05/23 01:09:52 stderr dataType: undefined, 2024/05/23 01:09:52 stderr column: undefined, 2024/05/23 01:09:52 stderr table: undefined, 2024/05/23 01:09:52 stderr schema: undefined, 2024/05/23 01:09:52 stderr where: undefined, 2024/05/23 01:09:52 stderr internalQuery: undefined, 2024/05/23 01:09:52 stderr internalPosition: undefined, 2024/05/23 01:09:52 stderr position: '128', 2024/05/23 01:09:52 stderr hint: undefined, 2024/05/23 01:09:52 stderr detail: undefined, 2024/05/23 01:09:52 stderr code: '42P01', 2024/05/23 01:09:52 stderr severity: 'ERROR', 2024/05/23 01:09:52 stderr length: 113, 2024/05/23 01:09:52 stderr }, 2024/05/23 01:09:52 stderr routine: 'parserOpenTable' 2024/05/23 01:09:52 stderr line: '1381', 2024/05/23 01:09:52 stderr file: 'parse_relation.c', 2024/05/23 01:09:52 stderr constraint: undefined, 2024/05/23 01:09:52 stderr dataType: undefined, 2024/05/23 01:09:52 stderr column: undefined, 2024/05/23 01:09:52 stderr table: undefined, 2024/05/23 01:09:52 stderr schema: undefined, 2024/05/23 01:09:52 stderr where: undefined, 2024/05/23 01:09:52 stderr internalQuery: undefined, 2024/05/23 01:09:52 stderr internalPosition: undefined, 2024/05/23 01:09:52 stderr position: '128', 2024/05/23 01:09:52 stderr hint: undefined, 2024/05/23 01:09:52 stderr detail: undefined, 2024/05/23 01:09:52 stderr code: '42P01', 2024/05/23 01:09:52 stderr severity: 'ERROR', 2024/05/23 01:09:52 stderr length: 113,
OVERLORD commented 2026-02-05 07:44:38 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (May 22, 2024):

Hmm, that error is about connecting to Postgres, not related to thumbnail generation.

@mertalev commented on GitHub (May 22, 2024): Hmm, that error is about connecting to Postgres, not related to thumbnail generation.
OVERLORD commented 2026-02-05 07:44:41 +03:00
Author
Owner
Copy Link

@RazerProof commented on GitHub (May 22, 2024):

Yeah... Something may have gone wrong with the deployment. Let me do some more testing, and maybe get some sleep :-).

@RazerProof commented on GitHub (May 22, 2024): Yeah... Something may have gone wrong with the deployment. Let me do some more testing, and maybe get some sleep :-).
OVERLORD commented 2026-02-05 07:44:47 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (May 22, 2024):

So sorry! I based that branch off of the latest release, but it turns out that main gets merged into it anyway when the image is built. The error is probably because of that.

You can either wait for the next release to get things back up or restore from a backup. (It's also possible to mess with it more to get it back up, but I think these options are safer.)

@mertalev commented on GitHub (May 22, 2024): So sorry! I based that branch off of the latest release, but it turns out that main gets merged into it anyway when the image is built. The error is probably because of that. You can either wait for the next release to get things back up or restore from a backup. (It's also possible to mess with it more to get it back up, but I think these options are safer.)
OVERLORD commented 2026-02-05 07:44:51 +03:00
Author
Owner
Copy Link

@RazerProof commented on GitHub (May 22, 2024):

Thanks for letting me know and I can confirm I am getting the same result.
image
It is working, and there are no errors in the log; it's just hungry :-)
I will wait for the next release and re-test
Thanks for all your great work.

@RazerProof commented on GitHub (May 22, 2024): Thanks for letting me know and I can confirm I am getting the same result. ![image](https://github.com/immich-app/immich/assets/166206048/f08ba7b4-9889-42c1-adf6-8848c9c5af5e) It is working, and there are no errors in the log; it's just hungry :-) I will wait for the next release and re-test Thanks for all your great work.
OVERLORD commented 2026-02-05 07:44:54 +03:00
Author
Owner
Copy Link

@DRIgnazGortngschirl commented on GitHub (Jul 11, 2024):

I recently uploaded approximately 93MB (~180 files) and noticed that as soon as the images were uploaded and I started the jobs, the memory usage spiked. When the face detection job began, my memory usage jumped from about 600MB to 4GB, which was my maximum at that time. I increased the memory allocation, thinking that Buffalo-L might need more RAM. I kept increasing it until I started receiving error logs. Prior to this, there were no error logs, likely because the memory filled up so quickly that the logs couldn't be written. Once the memory was fully utilized, the CPU usage also surged to 100%.

immich_machine_learning  | [07/11/24 19:27:57] INFO     Booting worker with pid: 54
immich_machine_learning  | [07/11/24 19:28:00] INFO     Started server process [54]
immich_machine_learning  | [07/11/24 19:28:00] INFO     Waiting for application startup.
immich_machine_learning  | [07/11/24 19:28:00] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [07/11/24 19:28:00] INFO     Initialized request thread pool with 16 threads.
immich_machine_learning  | [07/11/24 19:28:00] INFO     Application startup complete.
immich_machine_learning  | [07/11/24 19:28:00] INFO     Loading detection model 'buffalo_l' to memory
immich_machine_learning  | [07/11/24 19:28:00] INFO     Setting execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_machine_learning  | [07/11/24 19:28:01] INFO     Loading recognition model 'buffalo_l' to memory
immich_machine_learning  | [07/11/24 19:28:01] INFO     Setting execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_machine_learning  | [07/11/24 19:28:26] ERROR    Worker (pid:54) was sent SIGKILL! Perhaps out of
immich_machine_learning  |                              memory?
immich_server            | [Nest] 7  - 07/11/2024, 7:28:26 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
immich_server            | [Nest] 7  - 07/11/2024, 7:28:26 PM   ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
immich_server            |     at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
immich_server            |     at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
immich_server            |     at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
immich_server            |     at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:275:52)
immich_server            |     at async /usr/src/app/dist/services/job.service.js:148:36
immich_server            |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_server            |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_server            | [Nest] 7  - 07/11/2024, 7:28:26 PM   ERROR [Microservices:JobService] Object:
immich_server            | {
immich_server            |   "id": "be1f37e0-0390-46da-aab5-1f0864c35941"
immich_server            | }
immich_server            |
immich_machine_learning  | [07/11/24 19:28:26] INFO     Booting worker with pid: 76
immich_machine_learning  | [07/11/24 19:28:29] INFO     Started server process [76]
immich_machine_learning  | [07/11/24 19:28:29] INFO     Waiting for application startup.
immich_machine_learning  | [07/11/24 19:28:29] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [07/11/24 19:28:29] INFO     Initialized request thread pool with 16 threads.
immich_machine_learning  | [07/11/24 19:28:29] INFO     Application startup complete.
immich_machine_learning  | [07/11/24 19:28:29] INFO     Loading detection model 'buffalo_l' to memory
immich_machine_learning  | [07/11/24 19:28:29] INFO     Setting execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_machine_learning  | [07/11/24 19:28:30] INFO     Loading recognition model 'buffalo_l' to memory
immich_machine_learning  | [07/11/24 19:28:30] INFO     Setting execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_machine_learning  | [07/11/24 19:28:51] ERROR    Worker (pid:76) was sent SIGKILL! Perhaps out of
immich_machine_learning  |                              memory?
immich_server            | [Nest] 7  - 07/11/2024, 7:28:51 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
immich_server            | [Nest] 7  - 07/11/2024, 7:28:51 PM   ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
immich_server            |     at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
immich_server            |     at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
immich_server            |     at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
immich_server            |     at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:275:52)
immich_server            |     at async /usr/src/app/dist/services/job.service.js:148:36
immich_server            |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_server            |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_server            | [Nest] 7  - 07/11/2024, 7:28:51 PM   ERROR [Microservices:JobService] Object:
immich_server            | {
immich_server            |   "id": "c66e8bd6-24e7-45b8-8ba6-0e6634577e14"
immich_server            | }
immich_server            |
immich_machine_learning  | [07/11/24 19:28:51] INFO     Booting worker with pid: 98
immich_machine_learning  | [07/11/24 19:28:53] INFO     Started server process [98]
immich_machine_learning  | [07/11/24 19:28:53] INFO     Waiting for application startup.
immich_machine_learning  | [07/11/24 19:28:53] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [07/11/24 19:28:53] INFO     Initialized request thread pool with 16 threads.
immich_machine_learning  | [07/11/24 19:28:53] INFO     Application startup complete.
immich_machine_learning  | [07/11/24 19:28:53] INFO     Loading detection model 'buffalo_l' to memory
immich_machine_learning  | [07/11/24 19:28:53] INFO     Setting execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_machine_learning  | [07/11/24 19:28:54] INFO     Loading recognition model 'buffalo_l' to memory
immich_machine_learning  | [07/11/24 19:28:54] INFO     Setting execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_server            | [Nest] 7  - 07/11/2024, 7:29:13 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
immich_server            | [Nest] 7  - 07/11/2024, 7:29:13 PM   ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
immich_server            |     at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
immich_server            |     at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
immich_server            |     at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
immich_server            |     at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:275:52)
immich_server            |     at async /usr/src/app/dist/services/job.service.js:148:36
immich_server            |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_server            |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_server            | [Nest] 7  - 07/11/2024, 7:29:13 PM   ERROR [Microservices:JobService] Object:
immich_server            | {
immich_server            |   "id": "fa08cb36-ad88-44c7-ad8b-24950987bf3c"
immich_server            | }
immich_server            |
immich_machine_learning  | [07/11/24 19:29:13] ERROR    Worker (pid:98) was sent SIGKILL! Perhaps out of
immich_machine_learning  |                              memory?
immich_machine_learning  | [07/11/24 19:29:13] INFO     Booting worker with pid: 114
immich_machine_learning  | [07/11/24 19:29:16] INFO     Started server process [114]
immich_machine_learning  | [07/11/24 19:29:16] INFO     Waiting for application startup.
immich_machine_learning  | [07/11/24 19:29:16] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [07/11/24 19:29:16] INFO     Initialized request thread pool with 16 threads.
immich_machine_learning  | [07/11/24 19:29:16] INFO     Application startup complete.
immich_machine_learning  | [07/11/24 19:29:16] INFO     Loading detection model 'buffalo_l' to memory
immich_machine_learning  | [07/11/24 19:29:16] INFO     Setting execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_machine_learning  | [07/11/24 19:29:17] INFO     Loading recognition model 'buffalo_l' to memory
immich_machine_learning  | [07/11/24 19:29:17] INFO     Setting execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_server            | [Nest] 7  - 07/11/2024, 7:29:42 PM   ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
immich_server            | [Nest] 7  - 07/11/2024, 7:29:42 PM   ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed
immich_server            |     at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19
immich_server            |     at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21)
immich_server            |     at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26)
immich_server            |     at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:275:52)
immich_server            |     at async /usr/src/app/dist/services/job.service.js:148:36
immich_server            |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_server            |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_server            | [Nest] 7  - 07/11/2024, 7:29:42 PM   ERROR [Microservices:JobService] Object:
immich_server            | {
immich_server            |   "id": "3c01f06f-01aa-481f-9703-0d7e67d41bbd"
immich_server            | }
immich_server            |
immich_machine_learning  | [07/11/24 19:29:42] ERROR    Worker (pid:114) was sent SIGKILL! Perhaps out of
immich_machine_learning  |                              memory?
immich_machine_learning  | [07/11/24 19:29:42] INFO     Booting worker with pid: 136
immich_redis             | 1:M 11 Jul 2024 19:29:44.072 * 10000 changes in 60 seconds. Saving...
immich_redis             | 1:M 11 Jul 2024 19:29:44.072 * Background saving started by pid 22
immich_redis             | 22:C 11 Jul 2024 19:29:44.089 * DB saved on disk
immich_redis             | 22:C 11 Jul 2024 19:29:44.089 * RDB: 0 MB of memory used by copy-on-write
immich_redis             | 1:M 11 Jul 2024 19:29:44.190 * Background saving terminated with success
immich_machine_learning  | [07/11/24 19:29:45] INFO     Started server process [136]
immich_machine_learning  | [07/11/24 19:29:45] INFO     Waiting for application startup.
immich_machine_learning  | [07/11/24 19:29:45] INFO     Created in-memory cache with unloading after 300s
immich_machine_learning  |                              of inactivity.
immich_machine_learning  | [07/11/24 19:29:45] INFO     Initialized request thread pool with 16 threads.
immich_machine_learning  | [07/11/24 19:29:45] INFO     Application startup complete.
@DRIgnazGortngschirl commented on GitHub (Jul 11, 2024): I recently uploaded approximately 93MB (~180 files) and noticed that as soon as the images were uploaded and I started the jobs, the memory usage spiked. When the face detection job began, my memory usage jumped from about 600MB to 4GB, which was my maximum at that time. I increased the memory allocation, thinking that Buffalo-L might need more RAM. I kept increasing it until I started receiving error logs. Prior to this, there were no error logs, likely because the memory filled up so quickly that the logs couldn't be written. Once the memory was fully utilized, the CPU usage also surged to 100%. ``` immich_machine_learning | [07/11/24 19:27:57] INFO Booting worker with pid: 54 immich_machine_learning | [07/11/24 19:28:00] INFO Started server process [54] immich_machine_learning | [07/11/24 19:28:00] INFO Waiting for application startup. immich_machine_learning | [07/11/24 19:28:00] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/11/24 19:28:00] INFO Initialized request thread pool with 16 threads. immich_machine_learning | [07/11/24 19:28:00] INFO Application startup complete. immich_machine_learning | [07/11/24 19:28:00] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/11/24 19:28:00] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/11/24 19:28:01] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/11/24 19:28:01] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/11/24 19:28:26] ERROR Worker (pid:54) was sent SIGKILL! Perhaps out of immich_machine_learning | memory? immich_server | [Nest] 7 - 07/11/2024, 7:28:26 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed immich_server | [Nest] 7 - 07/11/2024, 7:28:26 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed immich_server | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19 immich_server | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21) immich_server | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26) immich_server | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:275:52) immich_server | at async /usr/src/app/dist/services/job.service.js:148:36 immich_server | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) immich_server | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) immich_server | [Nest] 7 - 07/11/2024, 7:28:26 PM ERROR [Microservices:JobService] Object: immich_server | { immich_server | "id": "be1f37e0-0390-46da-aab5-1f0864c35941" immich_server | } immich_server | immich_machine_learning | [07/11/24 19:28:26] INFO Booting worker with pid: 76 immich_machine_learning | [07/11/24 19:28:29] INFO Started server process [76] immich_machine_learning | [07/11/24 19:28:29] INFO Waiting for application startup. immich_machine_learning | [07/11/24 19:28:29] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/11/24 19:28:29] INFO Initialized request thread pool with 16 threads. immich_machine_learning | [07/11/24 19:28:29] INFO Application startup complete. immich_machine_learning | [07/11/24 19:28:29] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/11/24 19:28:29] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/11/24 19:28:30] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/11/24 19:28:30] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/11/24 19:28:51] ERROR Worker (pid:76) was sent SIGKILL! Perhaps out of immich_machine_learning | memory? immich_server | [Nest] 7 - 07/11/2024, 7:28:51 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed immich_server | [Nest] 7 - 07/11/2024, 7:28:51 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed immich_server | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19 immich_server | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21) immich_server | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26) immich_server | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:275:52) immich_server | at async /usr/src/app/dist/services/job.service.js:148:36 immich_server | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) immich_server | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) immich_server | [Nest] 7 - 07/11/2024, 7:28:51 PM ERROR [Microservices:JobService] Object: immich_server | { immich_server | "id": "c66e8bd6-24e7-45b8-8ba6-0e6634577e14" immich_server | } immich_server | immich_machine_learning | [07/11/24 19:28:51] INFO Booting worker with pid: 98 immich_machine_learning | [07/11/24 19:28:53] INFO Started server process [98] immich_machine_learning | [07/11/24 19:28:53] INFO Waiting for application startup. immich_machine_learning | [07/11/24 19:28:53] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/11/24 19:28:53] INFO Initialized request thread pool with 16 threads. immich_machine_learning | [07/11/24 19:28:53] INFO Application startup complete. immich_machine_learning | [07/11/24 19:28:53] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/11/24 19:28:53] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/11/24 19:28:54] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/11/24 19:28:54] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_server | [Nest] 7 - 07/11/2024, 7:29:13 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed immich_server | [Nest] 7 - 07/11/2024, 7:29:13 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed immich_server | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19 immich_server | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21) immich_server | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26) immich_server | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:275:52) immich_server | at async /usr/src/app/dist/services/job.service.js:148:36 immich_server | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) immich_server | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) immich_server | [Nest] 7 - 07/11/2024, 7:29:13 PM ERROR [Microservices:JobService] Object: immich_server | { immich_server | "id": "fa08cb36-ad88-44c7-ad8b-24950987bf3c" immich_server | } immich_server | immich_machine_learning | [07/11/24 19:29:13] ERROR Worker (pid:98) was sent SIGKILL! Perhaps out of immich_machine_learning | memory? immich_machine_learning | [07/11/24 19:29:13] INFO Booting worker with pid: 114 immich_machine_learning | [07/11/24 19:29:16] INFO Started server process [114] immich_machine_learning | [07/11/24 19:29:16] INFO Waiting for application startup. immich_machine_learning | [07/11/24 19:29:16] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/11/24 19:29:16] INFO Initialized request thread pool with 16 threads. immich_machine_learning | [07/11/24 19:29:16] INFO Application startup complete. immich_machine_learning | [07/11/24 19:29:16] INFO Loading detection model 'buffalo_l' to memory immich_machine_learning | [07/11/24 19:29:16] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_machine_learning | [07/11/24 19:29:17] INFO Loading recognition model 'buffalo_l' to memory immich_machine_learning | [07/11/24 19:29:17] INFO Setting execution providers to immich_machine_learning | ['CPUExecutionProvider'], in descending order of immich_machine_learning | preference immich_server | [Nest] 7 - 07/11/2024, 7:29:42 PM ERROR [Microservices:JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed immich_server | [Nest] 7 - 07/11/2024, 7:29:42 PM ERROR [Microservices:JobService] Error: Machine learning request to "http://immich-machine-learning:3003" failed with SocketError: other side closed immich_server | at /usr/src/app/dist/repositories/machine-learning.repository.js:19:19 immich_server | at async MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:18:21) immich_server | at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:33:26) immich_server | at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:275:52) immich_server | at async /usr/src/app/dist/services/job.service.js:148:36 immich_server | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) immich_server | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) immich_server | [Nest] 7 - 07/11/2024, 7:29:42 PM ERROR [Microservices:JobService] Object: immich_server | { immich_server | "id": "3c01f06f-01aa-481f-9703-0d7e67d41bbd" immich_server | } immich_server | immich_machine_learning | [07/11/24 19:29:42] ERROR Worker (pid:114) was sent SIGKILL! Perhaps out of immich_machine_learning | memory? immich_machine_learning | [07/11/24 19:29:42] INFO Booting worker with pid: 136 immich_redis | 1:M 11 Jul 2024 19:29:44.072 * 10000 changes in 60 seconds. Saving... immich_redis | 1:M 11 Jul 2024 19:29:44.072 * Background saving started by pid 22 immich_redis | 22:C 11 Jul 2024 19:29:44.089 * DB saved on disk immich_redis | 22:C 11 Jul 2024 19:29:44.089 * RDB: 0 MB of memory used by copy-on-write immich_redis | 1:M 11 Jul 2024 19:29:44.190 * Background saving terminated with success immich_machine_learning | [07/11/24 19:29:45] INFO Started server process [136] immich_machine_learning | [07/11/24 19:29:45] INFO Waiting for application startup. immich_machine_learning | [07/11/24 19:29:45] INFO Created in-memory cache with unloading after 300s immich_machine_learning | of inactivity. immich_machine_learning | [07/11/24 19:29:45] INFO Initialized request thread pool with 16 threads. immich_machine_learning | [07/11/24 19:29:45] INFO Application startup complete. ```
OVERLORD commented 2026-02-05 07:44:57 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 10, 2024):

We changed some settings that should have helped. Is this still an issue?

@jrasm91 commented on GitHub (Sep 10, 2024): We changed some settings that should have helped. Is this still an issue?
OVERLORD commented 2026-02-05 07:45:06 +03:00
Author
Owner
Copy Link

@kdybicz commented on GitHub (Sep 10, 2024):

On the latest release version server crashed without obvious error:

2024/09/10 09:35:55,stderr,api worker exited with code null
2024/09/10 09:35:42,stderr,
2024/09/10 09:35:42,stderr,}
2024/09/10 09:35:42,stderr,  "id": "74013933-6496-422e-a1ba-d7e98c6332dc"
2024/09/10 09:35:42,stderr,{
2024/09/10 09:35:42,stderr,"est] 7  - 09/10/2024, 7:35:41 AM   ERROR icroservices:JobService] Object:"
2024/09/10 09:35:42,stderr,    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
2024/09/10 09:35:42,stderr,    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
2024/09/10 09:35:42,stderr,    at async /usr/src/app/dist/services/job.service.js:148:36
2024/09/10 09:35:42,stderr,    at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:141:29)
2024/09/10 09:35:42,stderr,    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024/09/10 09:35:42,stderr,    at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:176:48)
2024/09/10 09:35:42,stderr,    at MediaRepository.generateThumbnail (/usr/src/app/dist/repositories/media.repository.js:69:14)
2024/09/10 09:35:42,stderr,    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:90:19)
2024/09/10 09:35:42,stderr,"est] 7  - 09/10/2024, 7:35:41 AM   ERROR icroservices:JobService] Error: Input file contains unsupported image format"
2024/09/10 09:35:42,stderr,"est] 7  - 09/10/2024, 7:35:41 AM   ERROR icroservices:JobService] Unable to run job handler (thumbnailGeneration/generate-preview): Error: Input file contains unsupported image format"
2024/09/10 09:35:41,stderr,
2024/09/10 09:35:41,stderr,}
...
2024/09/10 09:35:38,stderr,  "id": "3f35526b-ce3d-40a2-b54e-6a5b19773b3d"
2024/09/10 09:35:38,stderr,{
2024/09/10 09:35:38,stderr,"est] 7  - 09/10/2024, 7:35:38 AM   ERROR icroservices:JobService] Object:"
2024/09/10 09:35:38,stderr,    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
2024/09/10 09:35:38,stderr,    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
2024/09/10 09:35:38,stderr,    at async /usr/src/app/dist/services/job.service.js:148:36
2024/09/10 09:35:38,stderr,    at async MetadataService.handleMetadataExtraction (/usr/src/app/dist/services/metadata.service.js:194:40)
2024/09/10 09:35:38,stderr,    at async MetadataService.exifData (/usr/src/app/dist/services/metadata.service.js:467:23)
2024/09/10 09:35:38,stderr,    at async Object.stat (node:internal/fs/promises:1032:18)
2024/09/10 09:35:38,stderr,"est] 7  - 09/10/2024, 7:35:38 AM   ERROR icroservices:JobService] Error: ENOENT: no such file or directory, stat '/usr/src/app/external/Upload/2024/05/IMG_20240509_183102.jpg'"
2024/09/10 09:35:38,stderr,"est] 7  - 09/10/2024, 7:35:38 AM   ERROR icroservices:JobService] Unable to run job handler (metadataExtraction/metadata-extraction): Error: ENOENT: no such file or directory, stat '/usr/src/app/external/Upload/2024/05/IMG_20240509_183102.jpg'"
2024/09/10 09:35:38,stderr,
2024/09/10 09:35:38,stderr,}
...
2024/09/10 09:35:30,stderr,"est] 17  - 09/10/2024, 7:34:07 AM   ERROR pi:LibraryService] Library watcher for library 82616fd4-b0d7-4a16-98bd-a845f000f014 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/usr/src/app/external/#recycle/something/@eaDir'"
2024/09/10 09:35:30,stderr,"est] 17  - 09/10/2024, 7:33:35 AM   ERROR pi:LibraryService] Library watcher for library 82616fd4-b0d7-4a16-98bd-a845f000f014 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/usr/src/app/external/2017/something else/@eaDir/cn-tower.jpeg@SynoEAStream'
...
@kdybicz commented on GitHub (Sep 10, 2024): On the latest release version server crashed without obvious error: ``` 2024/09/10 09:35:55,stderr,api worker exited with code null 2024/09/10 09:35:42,stderr, 2024/09/10 09:35:42,stderr,} 2024/09/10 09:35:42,stderr, "id": "74013933-6496-422e-a1ba-d7e98c6332dc" 2024/09/10 09:35:42,stderr,{ 2024/09/10 09:35:42,stderr,"est] 7 - 09/10/2024, 7:35:41 AM  ERROR icroservices:JobService] Object:" 2024/09/10 09:35:42,stderr, at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) 2024/09/10 09:35:42,stderr, at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) 2024/09/10 09:35:42,stderr, at async /usr/src/app/dist/services/job.service.js:148:36 2024/09/10 09:35:42,stderr, at async MediaService.handleGeneratePreview (/usr/src/app/dist/services/media.service.js:141:29) 2024/09/10 09:35:42,stderr, at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2024/09/10 09:35:42,stderr, at MediaService.generateThumbnail (/usr/src/app/dist/services/media.service.js:176:48) 2024/09/10 09:35:42,stderr, at MediaRepository.generateThumbnail (/usr/src/app/dist/repositories/media.repository.js:69:14) 2024/09/10 09:35:42,stderr, at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:90:19) 2024/09/10 09:35:42,stderr,"est] 7 - 09/10/2024, 7:35:41 AM  ERROR icroservices:JobService] Error: Input file contains unsupported image format" 2024/09/10 09:35:42,stderr,"est] 7 - 09/10/2024, 7:35:41 AM  ERROR icroservices:JobService] Unable to run job handler (thumbnailGeneration/generate-preview): Error: Input file contains unsupported image format" 2024/09/10 09:35:41,stderr, 2024/09/10 09:35:41,stderr,} ... 2024/09/10 09:35:38,stderr, "id": "3f35526b-ce3d-40a2-b54e-6a5b19773b3d" 2024/09/10 09:35:38,stderr,{ 2024/09/10 09:35:38,stderr,"est] 7 - 09/10/2024, 7:35:38 AM  ERROR icroservices:JobService] Object:" 2024/09/10 09:35:38,stderr, at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) 2024/09/10 09:35:38,stderr, at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) 2024/09/10 09:35:38,stderr, at async /usr/src/app/dist/services/job.service.js:148:36 2024/09/10 09:35:38,stderr, at async MetadataService.handleMetadataExtraction (/usr/src/app/dist/services/metadata.service.js:194:40) 2024/09/10 09:35:38,stderr, at async MetadataService.exifData (/usr/src/app/dist/services/metadata.service.js:467:23) 2024/09/10 09:35:38,stderr, at async Object.stat (node:internal/fs/promises:1032:18) 2024/09/10 09:35:38,stderr,"est] 7 - 09/10/2024, 7:35:38 AM  ERROR icroservices:JobService] Error: ENOENT: no such file or directory, stat '/usr/src/app/external/Upload/2024/05/IMG_20240509_183102.jpg'" 2024/09/10 09:35:38,stderr,"est] 7 - 09/10/2024, 7:35:38 AM  ERROR icroservices:JobService] Unable to run job handler (metadataExtraction/metadata-extraction): Error: ENOENT: no such file or directory, stat '/usr/src/app/external/Upload/2024/05/IMG_20240509_183102.jpg'" 2024/09/10 09:35:38,stderr, 2024/09/10 09:35:38,stderr,} ... 2024/09/10 09:35:30,stderr,"est] 17 - 09/10/2024, 7:34:07 AM  ERROR pi:LibraryService] Library watcher for library 82616fd4-b0d7-4a16-98bd-a845f000f014 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/usr/src/app/external/#recycle/something/@eaDir'" 2024/09/10 09:35:30,stderr,"est] 17 - 09/10/2024, 7:33:35 AM  ERROR pi:LibraryService] Library watcher for library 82616fd4-b0d7-4a16-98bd-a845f000f014 encountered error: Error: ENOSPC: System limit for number of file watchers reached, watch '/usr/src/app/external/2017/something else/@eaDir/cn-tower.jpeg@SynoEAStream' ... ```
OVERLORD commented 2026-02-05 07:45:12 +03:00
Author
Owner
Copy Link

@danieldietzler commented on GitHub (Sep 21, 2024):

This looks like some images are just borked. You don't run into out of memory issues anymore though, correct?

@danieldietzler commented on GitHub (Sep 21, 2024): This looks like some images are just borked. You don't run into out of memory issues anymore though, correct?
OVERLORD commented 2026-02-05 07:45:16 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 22, 2024):

@kdybicz

System limit for number of file watchers reached, watch '/usr/src/app/external/#recycle/something/@eaDir
@jrasm91 commented on GitHub (Sep 22, 2024): @kdybicz ``` System limit for number of file watchers reached, watch '/usr/src/app/external/#recycle/something/@eaDir ```
OVERLORD commented 2026-02-05 07:45:23 +03:00
Author
Owner
Copy Link

@kdybicz commented on GitHub (Sep 22, 2024):

@jrasm91 I would love to try to exclude #recycle and @eaDir, if that's what you're suggesting, but... web UI doesn't load. Just tested latest version, still having same issues.

Edit: Scratch that, I've just double checked and looks like I've already added those exclusion patterns:

**/@eaDir/**
**/#recycle/**

🤔

Edit 2: Just after web UI started I paused all of the jobs and forced the removal of offline files, and even though the service stopped logging errors related to files not found or broken, it still crashed a minute later with:

microservices worker error: Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory
@kdybicz commented on GitHub (Sep 22, 2024): @jrasm91 I would love to try to exclude `#recycle` and `@eaDir`, if that's what you're suggesting, but... web UI doesn't load. Just tested latest version, still having same issues. Edit: Scratch that, I've just double checked and looks like I've already added those exclusion patterns: ``` **/@eaDir/** **/#recycle/** ``` 🤔 Edit 2: Just after web UI started I paused all of the jobs and forced the removal of offline files, and even though the service stopped logging errors related to files not found or broken, it still crashed a minute later with: ``` microservices worker error: Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory ```
OVERLORD commented 2026-02-05 07:45:26 +03:00
Author
Owner
Copy Link

@danieldietzler commented on GitHub (Sep 22, 2024):

The latest release adds those as default exclusion patterns, that's why they're there :D

@danieldietzler commented on GitHub (Sep 22, 2024): The latest release adds those as default exclusion patterns, that's why they're there :D
OVERLORD commented 2026-02-05 07:45:30 +03:00
Author
Owner
Copy Link

@kdybicz commented on GitHub (Sep 22, 2024):

@danieldietzler latest release adds **/@eaDir/** and **/._*, not **/#recycle/** and others I've added manually.

Nevertheless, there must be some other gotcha regarding re-scans and/or offline/missing files, as I just did the classic "turn it off and on again" by nuking my old configuration and processing the library from scratch, and so far so good 🤞🏼

@kdybicz commented on GitHub (Sep 22, 2024): @danieldietzler latest release adds `**/@eaDir/**` and `**/._*`, not `**/#recycle/**` and others I've added manually. Nevertheless, there must be some other gotcha regarding re-scans and/or offline/missing files, as I just did the classic "turn it off and on again" by nuking my old configuration and processing the library from scratch, and so far so good 🤞🏼
OVERLORD commented 2026-02-05 07:45:32 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 27, 2024):

The 116 release that came out today also includes significant changes to the library scanning implementation, so you should test and let us know how it goes.

@jrasm91 commented on GitHub (Sep 27, 2024): The 116 release that came out today also includes significant changes to the library scanning implementation, so you should test and let us know how it goes.
OVERLORD commented 2026-02-05 07:45:35 +03:00
Author
Owner
Copy Link

@etnoy commented on GitHub (Oct 8, 2024):

Closing this for now, please let us know if this still is an issue. This code has been improved and hopefully it's fixed

@etnoy commented on GitHub (Oct 8, 2024): Closing this for now, please let us know if this still is an issue. This code has been improved and hopefully it's fixed
OVERLORD commented 2026-02-05 07:45:39 +03:00
Author
Owner
Copy Link

@yuhuan417 commented on GitHub (Oct 19, 2024):

I also encountered this issue on a photo library of 100,000 images, where the memory usage of immich-machine-learning would spike and keep increasing. So, I performed the following steps to identify the problem.

I ran the tasks one by one and found that facial detection was causing the issue. To make it easier to pinpoint the problem, I set the memory limit of immich-machine-learning to 2G and observed from the immich-server logs which photos would cause issues when the program crashed. Some photos, which were not large in size, would cause problems with a high probability (strangely, not 100%). I found one of them, a 2M jpg file, which was a group photo of 20 people. So, I speculated that the memory usage for facial recognition is related to the number of faces.

Upon closer examination of recognition.py, I noticed that facial recognition was done in batch prediction mode. I guessed that there was an issue with the underlying code that prevented memory from being released in this mode. I tried hardcoding "self.batch = self.model_format == ModelFormat.ONNX" to "self.batch = False", and the problem disappeared.

I'm not sure if this solution would work only in my specific environment. My machine is an N4505 with 8G of memory, and I'm using openvino for hardware acceleration.

@yuhuan417 commented on GitHub (Oct 19, 2024): I also encountered this issue on a photo library of 100,000 images, where the memory usage of immich-machine-learning would spike and keep increasing. So, I performed the following steps to identify the problem. I ran the tasks one by one and found that facial detection was causing the issue. To make it easier to pinpoint the problem, I set the memory limit of immich-machine-learning to 2G and observed from the immich-server logs which photos would cause issues when the program crashed. Some photos, which were not large in size, would cause problems with a high probability (strangely, not 100%). I found one of them, a 2M jpg file, which was a group photo of 20 people. So, I speculated that the memory usage for facial recognition is related to the number of faces. Upon closer examination of recognition.py, I noticed that facial recognition was done in batch prediction mode. I guessed that there was an issue with the underlying code that prevented memory from being released in this mode. I tried hardcoding "self.batch = self.model_format == ModelFormat.ONNX" to "self.batch = False", and the problem disappeared. I'm not sure if this solution would work only in my specific environment. My machine is an N4505 with 8G of memory, and I'm using openvino for hardware acceleration.
OVERLORD commented 2026-02-05 07:45:42 +03:00
Author
Owner
Copy Link

@kdybicz commented on GitHub (Oct 19, 2024):

That's an interesting observation and might suggest why I still see my memory-limiter server crash without any clear reason 🤔

@kdybicz commented on GitHub (Oct 19, 2024): That's an interesting observation and might suggest why I still see my memory-limiter server crash without any clear reason 🤔
OVERLORD commented 2026-02-05 07:45:50 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Oct 19, 2024):

@mertalev can you replicate what @yuhuan417 is seeing?

@alextran1502 commented on GitHub (Oct 19, 2024): @mertalev can you replicate what @yuhuan417 is seeing?
OVERLORD commented 2026-02-05 07:45:51 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Oct 19, 2024):

Yes, I've tested the same thing with @1-tempest and came to the conclusion that OpenVINO allocates completely separate memory for each number of faces it encounters in an image, probably for speed reasons. This means the memory to process 3 faces is completely separate from the memory for 4 faces, etc., and this memory is cached for quick reuse.

On @1-tempest's processor, batching didn't have any positive effect on performance. On the new 155H processor, it had a 16% impact for 5 faces. The latter is notable and would likely be a bigger gap with discrete Arc cards.

I think it makes sense to limit the batch size here in general, and conservatively set it to 1 for OpenVINO by default.

@mertalev commented on GitHub (Oct 19, 2024): Yes, I've tested the same thing with @1-tempest and came to the conclusion that OpenVINO allocates completely separate memory for each number of faces it encounters in an image, probably for speed reasons. This means the memory to process 3 faces is completely separate from the memory for 4 faces, etc., and this memory is cached for quick reuse. On @1-tempest's processor, batching didn't have any positive effect on performance. On the new 155H processor, it had a 16% impact for 5 faces. The latter is notable and would likely be a bigger gap with discrete Arc cards. I think it makes sense to limit the batch size here in general, and conservatively set it to 1 for OpenVINO by default.
OVERLORD commented 2026-02-05 07:45:54 +03:00
Author
Owner
Copy Link

@lscorcia commented on GitHub (Oct 21, 2024):

I tried hardcoding "self.batch = self.model_format == ModelFormat.ONNX" to "self.batch = False", and the problem disappeared.

+1. I can confirm that the suggestion finally fixed the persistent out of memory issue on my Synology NAS. Now memory is nice and stable around 1.5 GB for the immich_machine_learning container during face detection / recognition. Awesome find!

@lscorcia commented on GitHub (Oct 21, 2024): > I tried hardcoding "self.batch = self.model_format == ModelFormat.ONNX" to "self.batch = False", and the problem disappeared. +1. I can confirm that the suggestion finally fixed the persistent out of memory issue on my Synology NAS. Now memory is nice and stable around 1.5 GB for the immich_machine_learning container during face detection / recognition. Awesome find!
OVERLORD commented 2026-02-05 07:46:00 +03:00
Author
Owner
Copy Link

@jaimetur commented on GitHub (Feb 27, 2025):

I think the memory issues is still there. Yesterday I ran a bulk deletion of all my assets (137.000) and my system got completely frozen 100% of memory usage and cpu usage. I left it working for 24h but it didn't recover.

Is there any way to limit the memory usage of the dockers?

In my case the redis service stopped several times due to lack of memory

@jaimetur commented on GitHub (Feb 27, 2025): I think the memory issues is still there. Yesterday I ran a bulk deletion of all my assets (137.000) and my system got completely frozen 100% of memory usage and cpu usage. I left it working for 24h but it didn't recover. Is there any way to limit the memory usage of the dockers? In my case the redis service stopped several times due to lack of memory
OVERLORD commented 2026-02-05 07:46:04 +03:00
Author
Owner
Copy Link

@Dunky-Z commented on GitHub (Feb 27, 2025):

@jaimetur I utilize the 'limits' attribute to restrict the resources of the container, thereby preventing the server from crashing. It would not affect its usage either. You may refer to this for your consideration.

  immich-server:
    container_name: immich_server
    image: altran1502/immich-server:${IMMICH_VERSION:-release}
    environment:
      - TZ=Asia/Shanghai  
    volumes:
      - .......................
    env_file:
      - .env
    devices:
      - /dev/dri:/dev/dri
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 4G
@Dunky-Z commented on GitHub (Feb 27, 2025): @jaimetur I utilize the 'limits' attribute to restrict the resources of the container, thereby preventing the server from crashing. It would not affect its usage either. You may refer to this for your consideration. ```yml immich-server: container_name: immich_server image: altran1502/immich-server:${IMMICH_VERSION:-release} environment: - TZ=Asia/Shanghai volumes: - ....................... env_file: - .env devices: - /dev/dri:/dev/dri ports: - '2283:2283' depends_on: - redis - database restart: always deploy: resources: limits: cpus: '2' memory: 4G ```
OVERLORD commented 2026-02-05 07:46:08 +03:00
Author
Owner
Copy Link

@Bazinga88 commented on GitHub (Oct 9, 2025):

quote previous

immich-server:
deploy:
resources:
limits:
cpus: '2'
memory: 4G

Just curious, why is INT cpus in quotes and VARCHAR memory not?

A few weeks ago postgres freaked out at my place.
Looking for a solution and found it in this topic.
Thanks all for this discussion.

In my case immich in a containermanager (SynoNAS) ran out of diskspace and for some unknown reason postgres went sideways.
Note: Immich is the only tool that i run in the containermanager on a rs1221+ with 32GB ram.

Restored a previous good working state on a larger volume and its up and running :-)

Image

Thanks again!

@Bazinga88 commented on GitHub (Oct 9, 2025): quote previous > immich-server: > deploy: > resources: > limits: > cpus: '2' > memory: 4G _Just curious, why is INT cpus in quotes and VARCHAR memory not?_ A few weeks ago postgres freaked out at my place. Looking for a solution and found it in this topic. Thanks all for this discussion. In my case immich in a containermanager (SynoNAS) ran out of diskspace and for some unknown reason postgres went sideways. Note: Immich is the only tool that i run in the containermanager on a rs1221+ with 32GB ram. Restored a previous good working state on a larger volume and its up and running :-) <img width="1528" height="388" alt="Image" src="https://github.com/user-attachments/assets/1197366a-644a-447b-a9d8-49320e247da9" /> Thanks again!
OVERLORD commented 2026-02-05 07:46:11 +03:00
Author
Owner
Copy Link

@Dunky-Z commented on GitHub (Oct 10, 2025):

quote previous

immich-server:
deploy:
resources:
limits:
cpus: '2'
memory: 4G

Just curious, why is INT cpus in quotes and VARCHAR memory not?

The issue you discovered is quite interesting and worth looking into. However, since I was also referencing someone else's approach, I didn't pay much attention to those details. I looked into it later, and the reasons are as follows (referencing Claude AI's response, hehe).

For the cpus parameter - quotes are required:

  1. Decimal support: The cpus value can be a decimal number (e.g., '0.5', '1.5', '2.0')
  2. YAML parsing behavior: Without quotes, a plain number like 2 is parsed as an integer type by the YAML parser, but Docker Compose expects it to be a string
  3. Official recommendation: Docker's official documentation recommends using quotes to ensure compatibility across different versions

While cpus: 2 (without quotes) may work in some cases, it can lead to compatibility issues or unexpected behavior.

For the memory parameter - quotes are optional:

  1. Automatic string detection: Values like 4G contain letters, so the YAML parser automatically treats them as strings
  2. Unit-based values: Memory values with units (4G, 512M, 2048M) are inherently string-formatted

Summary:

  • cpus: '2' Recommended (quotes required)
  • memory: 4G Recommended (quotes optional)
  • memory: '4G' Also acceptable (quotes don't hurt)

For consistency, you can quote both parameters:

cpus: '2'
memory: '4G'

This approach keeps the style uniform and avoids any potential parsing issues.

@Dunky-Z commented on GitHub (Oct 10, 2025): > quote previous > > > immich-server: > > deploy: > > resources: > > limits: > > cpus: '2' > > memory: 4G > > _Just curious, why is INT cpus in quotes and VARCHAR memory not?_ The issue you discovered is quite interesting and worth looking into. However, since I was also referencing someone else's approach, I didn't pay much attention to those details. I looked into it later, and the reasons are as follows (referencing Claude AI's response, hehe). **For the `cpus` parameter - quotes are required:** 1. **Decimal support**: The `cpus` value can be a decimal number (e.g., `'0.5'`, `'1.5'`, `'2.0'`) 2. **YAML parsing behavior**: Without quotes, a plain number like `2` is parsed as an integer type by the YAML parser, but Docker Compose expects it to be a string 3. **Official recommendation**: Docker's official documentation recommends using quotes to ensure compatibility across different versions While `cpus: 2` (without quotes) may work in some cases, it can lead to compatibility issues or unexpected behavior. **For the `memory` parameter - quotes are optional:** 1. **Automatic string detection**: Values like `4G` contain letters, so the YAML parser automatically treats them as strings 2. **Unit-based values**: Memory values with units (`4G`, `512M`, `2048M`) are inherently string-formatted **Summary:** - `cpus: '2'` Recommended (quotes required) - `memory: 4G` Recommended (quotes optional) - `memory: '4G'` Also acceptable (quotes don't hurt) For consistency, you can quote both parameters: ```yaml cpus: '2' memory: '4G' ``` This approach keeps the style uniform and avoids any potential parsing issues.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#3108
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?