immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 11 Packages Projects Releases 10 Wiki Activity

[BUG] Immich microservice keep crashing #1604

New Issue
Closed
opened 2026-02-05 02:37:02 +03:00 by OVERLORD · 10 comments
OVERLORD commented 2026-02-05 02:37:02 +03:00
Owner
Copy Link

Originally created by @CorentinGrard on GitHub (Nov 12, 2023).

The bug

Hi,

I installed a week ago immich and ingested the data with immich-go.
So far no problems, but then come the time to run the jobs. At first I through it was quite slow but after some investigation I find the container immich-microservices crashing with an out of memory every 2 minutes.
I have tried to give it more memory as this post suggest but it still crash as early as without this flag and just take more time to fillup the ram and crash.
I also took down in the settings the machine learning part as there was some error msg in the log related to this but it didn't have any impact on the problem.

Immich is running under a Nixos VM and the host is a Truenas scale and the immich upload folder is a NTF drive.

Log of the immich-microservice container with log set to debug

[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +24ms
[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/11/2023, 10:40:27 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +123ms
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [InstanceLoader] DomainModule dependencies initialized +13ms
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering thumbnailGeneration with a concurrency of 10
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering metadataExtraction with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering videoConversion with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering objectTagging with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering recognizeFaces with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering clipEncoding with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering backgroundTask with a concurrency of 5
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering storageTemplateMigration with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering migration with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering search with a concurrency of 5
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering sidecar with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Registering library with a concurrency of 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [StorageTemplateService] Received config, compiling storage template: {{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [JobService] Updating queue concurrency settings
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting thumbnailGeneration concurrency to 10
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting metadataExtraction concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting videoConversion concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting objectTagging concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting recognizeFaces concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting clipEncoding concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting backgroundTask concurrency to 5
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting storageTemplateMigration concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting migration concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting search concurrency to 5
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting sidecar concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM   DEBUG [JobService] Setting library concurrency to 1
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset dcf6670f-02e2-4fdd-9d13-e4791354d755
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 352473c1-6f54-490a-8730-9ee3f3eed20c
[Nest] 7  - 11/11/2023, 10:40:28 PM     LOG [MediaService] Start encoding video 18d538c9-f1f1-4777-ba2b-ba85078987f8 {"inputOptions":[],"outputOptions":["-c:v h264","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-v verbose","-vf scale=-2:720,format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}
[Nest] 7  - 11/11/2023, 10:40:29 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 67324e02-83c9-4e35-bfd5-1838bb212e8b
[Nest] 7  - 11/11/2023, 10:40:29 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 5787e921-0fda-42ea-ad9d-52f5c8674ca1
[Nest] 7  - 11/11/2023, 10:40:29 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 08a6b22c-bc3a-4631-bb84-25931a10db79
[Nest] 7  - 11/11/2023, 10:40:29 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 858292ce-2c72-4001-af39-f2e3b62eaf24
[Nest] 7  - 11/11/2023, 10:40:29 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 39cb6e73-611a-4da3-9bfa-31592fedf9d0
[Nest] 7  - 11/11/2023, 10:40:29 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset a15410ac-f6dc-4e52-9cf8-dd5d53442872
[Nest] 7  - 11/11/2023, 10:40:30 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset ef4c8d9d-563e-416f-83fc-3f9fc629a1b8
[Nest] 7  - 11/11/2023, 10:40:30 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset a59fe606-361f-4b49-9556-4f5f2400de51
[Nest] 7  - 11/11/2023, 10:40:30 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 305507fa-f1ad-4919-b2b9-0a23f5b6a80f
[Nest] 7  - 11/11/2023, 10:40:30 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 828c0398-688c-4fcb-aca6-f7c8796af6f4
[Nest] 7  - 11/11/2023, 10:40:31 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 107d0edf-5af8-4715-858f-2c506701fdb5
[Nest] 7  - 11/11/2023, 10:40:31 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset a998937b-6d5d-45b8-ab69-96b09ce82a09
[Nest] 7  - 11/11/2023, 10:40:31 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 2b109898-c927-4500-bdfe-3f44fa8872b9
[Nest] 7  - 11/11/2023, 10:40:31 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 53973ab7-865a-428f-895a-b7c37a39d4af
[Nest] 7  - 11/11/2023, 10:40:31 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset ec94298c-15e0-4b90-a370-fea5af895993
[Nest] 7  - 11/11/2023, 10:40:31 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset e748a8d7-edbe-4633-83a3-73b740776645
[Nest] 7  - 11/11/2023, 10:40:32 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset d365bef8-97ab-4d42-822f-60ba946d009a
[Nest] 7  - 11/11/2023, 10:40:32 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset b1dbb2e9-76c3-4c04-b81c-3877ffa4ab45
[Nest] 7  - 11/11/2023, 10:40:32 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 7e955d33-efa3-44e6-86ed-cfa2d2e7c7b5
[Nest] 7  - 11/11/2023, 10:40:32 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset 9452cdb2-6970-4307-aaae-d997da21460a
[Nest] 7  - 11/11/2023, 10:40:32 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 81c982ea-b615-4dc0-98b7-0a74647048fd
[Nest] 7  - 11/11/2023, 10:40:33 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 9babfaae-93ae-416b-9715-8ad3522daeee
[Nest] 7  - 11/11/2023, 10:40:33 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset c783a3da-c4e7-48a3-8b09-02b83683223d
[Nest] 7  - 11/11/2023, 10:40:33 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset f0b305ef-c2ed-4596-85a7-5220008b3fb8
[Nest] 7  - 11/11/2023, 10:40:33 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset bd7c8369-e534-443c-acde-48d38d44d3df
[Nest] 7  - 11/11/2023, 10:40:33 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset b619813f-68e5-43a4-83d3-4dea23e542ea
[Nest] 7  - 11/11/2023, 10:40:33 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset a09f2532-3a05-490e-b657-90dc61dd132c
[Nest] 7  - 11/11/2023, 10:40:33 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 3a45071b-564c-4920-bac7-36590026fcbc
[Nest] 7  - 11/11/2023, 10:40:34 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 36c668a1-f80c-4bd2-92b2-97a2ebb8a90d
[Nest] 7  - 11/11/2023, 10:40:34 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 0f002c0b-b22a-48d9-8d9d-762f29724e9d
[Nest] 7  - 11/11/2023, 10:40:34 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset ce44252b-5b69-4e0c-8974-2a12126e1262
[Nest] 7  - 11/11/2023, 10:40:35 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset c951c90f-c060-4f8c-9762-816b85c6d4b4
[Nest] 7  - 11/11/2023, 10:40:35 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset b880b344-dff8-4223-b8c6-9672a9300a68
[Nest] 7  - 11/11/2023, 10:40:35 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 75492468-6a34-44b4-a009-de3f4b852809
[Nest] 7  - 11/11/2023, 10:40:35 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset bbdad5ee-7cea-4764-9ca9-a2f6f24923a0
[Nest] 7  - 11/11/2023, 10:40:35 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset 1f21432a-178f-4cae-834c-b68acb1d6f18
[Nest] 7  - 11/11/2023, 10:40:35 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 82b236aa-c8f4-4de9-819b-63464bc7962e
[Nest] 7  - 11/11/2023, 10:40:36 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset a451de55-3c9d-4936-9fe1-fc5d97655565
[Nest] 7  - 11/11/2023, 10:40:37 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 0b6dc2b2-0970-426c-8b24-3ea0d9301d1f
[Nest] 7  - 11/11/2023, 10:40:37 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 9f69ad08-6935-418a-a270-24e6b28477de
[Nest] 7  - 11/11/2023, 10:40:38 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset 59b204e2-675c-4af7-b487-7bb8a76f3b07
[Nest] 7  - 11/11/2023, 10:40:38 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1997e16b-0824-4cae-9bf7-c1d0d3c20a9c
[Nest] 7  - 11/11/2023, 10:40:38 PM     LOG [MediaService] Encoding success 18d538c9-f1f1-4777-ba2b-ba85078987f8
[Nest] 7  - 11/11/2023, 10:40:39 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 0161c736-52e6-4d44-88c6-119d15a31753
[Nest] 7  - 11/11/2023, 10:40:39 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 318e68df-76dc-4afa-bdee-03ec00370015
[Nest] 7  - 11/11/2023, 10:40:39 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset bb01936b-3e9d-47b3-aedd-727b1ee7333a
[Nest] 7  - 11/11/2023, 10:40:39 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset d1dea1c2-44cd-4b1f-9997-4335346fc984
[Nest] 7  - 11/11/2023, 10:40:39 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset eb49ac4d-5f83-419d-995e-dcca974b619f
[Nest] 7  - 11/11/2023, 10:40:40 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset b804f7fc-313a-43e5-a2e0-7d35cc4067e6
[Nest] 7  - 11/11/2023, 10:40:40 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset 48eeb7bd-8fe6-40a1-bd0f-eeb3d53820d6
[Nest] 7  - 11/11/2023, 10:40:40 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1ab02e84-86cd-429b-aa52-89f732c9523d
[Nest] 7  - 11/11/2023, 10:40:41 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset ce2da5ba-ba28-4ceb-9c4a-8aa5428336a6
[Nest] 7  - 11/11/2023, 10:40:41 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 26a07170-f251-422f-92c0-36eb6e95e7ac
[Nest] 7  - 11/11/2023, 10:40:43 PM     LOG [MediaService] Start encoding video f0f93ba4-6bdc-45ac-863a-37a2e6299bcf {"inputOptions":[],"outputOptions":["-c:v h264","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-v verbose","-vf scale=-2:720,format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}
[Nest] 7  - 11/11/2023, 10:40:43 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset f0db521c-dc7b-4544-90d2-73086e5dadee
[Nest] 7  - 11/11/2023, 10:40:43 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 4de748f6-7043-405d-b01a-ff731db74a2c
[Nest] 7  - 11/11/2023, 10:40:43 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset d00a6417-7b5b-432d-be47-5857414d4793
[Nest] 7  - 11/11/2023, 10:40:43 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset efb58fa6-97fe-47a1-a18f-6b64b0fa41f5
[Nest] 7  - 11/11/2023, 10:40:44 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 5f9f49cc-8959-469e-aea8-f670a7f9324f
[Nest] 7  - 11/11/2023, 10:40:44 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 462d795f-1fc8-44e3-b250-9eb78bf40af5
[Nest] 7  - 11/11/2023, 10:40:44 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset d5319bde-e518-42ba-ad36-217ab72fc7f5
[Nest] 7  - 11/11/2023, 10:40:45 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset c934a1c6-05f5-4cae-b1ec-79e0adaf4633
[Nest] 7  - 11/11/2023, 10:40:45 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset 69a58bd0-36a6-4ce5-93c2-e35d35836b41
[Nest] 7  - 11/11/2023, 10:40:45 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset d7eb1c46-9607-4fb4-9456-a28e2862a123
[Nest] 7  - 11/11/2023, 10:40:45 PM     LOG [MediaService] Successfully generated JPEG video thumbnail for asset ecaf5910-bf97-4390-9ebb-2438b02d278d
[Nest] 7  - 11/11/2023, 10:40:46 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 35a85f3b-4c3e-4f54-9467-724079f6e10a
[Nest] 7  - 11/11/2023, 10:40:46 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset d70c47f0-9b15-4da6-a06e-08f116b983ca
[Nest] 7  - 11/11/2023, 10:40:47 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 6d4dd94f-2425-4488-bbee-cfe27d5f0ac5
[Nest] 7  - 11/11/2023, 10:40:47 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 7dc6e5fc-810c-4c99-8aac-fa88225e0570
[Nest] 7  - 11/11/2023, 10:40:47 PM     LOG [MediaService] Encoding success f0f93ba4-6bdc-45ac-863a-37a2e6299bcf
[Nest] 7  - 11/11/2023, 10:40:48 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 6800bdf4-45ca-41a6-aefd-e662d8df835f
<--- Last few GCs --->
[7:0x2885a310000]   148393 ms: Mark-Compact 4040.3 (4132.0) -> 4022.4 (4131.5) MB, 1987.37 / 0.00 ms  (average mu = 0.179, current mu = 0.084) allocation failure; scavenge might not succeed
[7:0x2885a310000]   150789 ms: Mark-Compact 4040.3 (4132.0) -> 4024.4 (4133.0) MB, 2341.68 / 0.00 ms  (average mu = 0.101, current mu = 0.022) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99960 node::Abort() [immich_microservices]
 2: 0xb6ffcb  [immich_microservices]
 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 5: 0x10d06a5  [immich_microservices]
 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
13: 0x7fc662d19ef6

The OS that Immich Server is running on

NixOS 23.05

Version of Immich Server

1.85.0

Version of Immich Mobile App

1.85.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env.photos
    depends_on:
      - redis
      - database
      - typesense
    restart: unless-stopped
    networks:
      - internal

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env.photos
    depends_on:
      - redis
      - database
      - typesense
    restart: unless-stopped
    networks:
      - internal

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env.photos
    restart: unless-stopped
    networks:
      - internal

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env.photos
    restart: unless-stopped
    networks:
      - internal

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: unless-stopped
    networks:
      - internal

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: unless-stopped
    networks:
      - internal

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env.photos
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: unless-stopped
    networks:
      - internal

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    depends_on:
      - immich-server
      - immich-web
    restart: unless-stopped
    labels:
      - traefik.enable=true
      - traefik.http.routers.immich.rule=Host(`xxxxx.example.com`)
      - traefik.http.routers.immich.entrypoints=websecure
      - traefik.http.routers.immich.tls.certresolver=myresolver
      - traefik.http.services.immich.loadbalancer.server.port=8080
    networks:
      - web
      - internal

volumes:
  pgdata:
  model-cache:
  tsdata:

networks:
  web:
    external:
      name: traefik_webgateway
  internal:
    internal: true

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/data/immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=xxxxx
DB_PASSWORD=xxxxxxx

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

LOG_LEVEL=debug

Reproduction steps

1. Start immich microservice

Additional information

No response

Originally created by @CorentinGrard on GitHub (Nov 12, 2023). ### The bug Hi, I installed a week ago immich and ingested the data with [immich-go](https://github.com/simulot/immich-go). So far no problems, but then come the time to run the jobs. At first I through it was quite slow but after some investigation I find the container immich-microservices crashing with an out of memory every 2 minutes. I have tried to give it more memory as [this post](https://github.com/immich-app/immich/issues/4530#issuecomment-1769868300) suggest but it still crash as early as without this flag and just take more time to fillup the ram and crash. I also took down in the settings the machine learning part as there was some error msg in the log related to this but it didn't have any impact on the problem. Immich is running under a Nixos VM and the host is a Truenas scale and the immich upload folder is a NTF drive. Log of the immich-microservice container with log set to debug ``` [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [NestFactory] Starting Nest application... [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +24ms [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [InstanceLoader] ConfigModule dependencies initialized +3ms [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/11/2023, 10:40:27 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +123ms [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [InstanceLoader] InfraModule dependencies initialized +3ms [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [InstanceLoader] DomainModule dependencies initialized +13ms [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering thumbnailGeneration with a concurrency of 10 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering metadataExtraction with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering videoConversion with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering objectTagging with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering recognizeFaces with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering clipEncoding with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering backgroundTask with a concurrency of 5 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering storageTemplateMigration with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering migration with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering search with a concurrency of 5 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering sidecar with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Registering library with a concurrency of 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [StorageTemplateService] Received config, compiling storage template: {{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}} [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [JobService] Updating queue concurrency settings [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting thumbnailGeneration concurrency to 10 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting metadataExtraction concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting videoConversion concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting objectTagging concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting recognizeFaces concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting clipEncoding concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting backgroundTask concurrency to 5 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting storageTemplateMigration concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting migration concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting search concurrency to 5 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting sidecar concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM DEBUG [JobService] Setting library concurrency to 1 [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset dcf6670f-02e2-4fdd-9d13-e4791354d755 [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 352473c1-6f54-490a-8730-9ee3f3eed20c [Nest] 7 - 11/11/2023, 10:40:28 PM LOG [MediaService] Start encoding video 18d538c9-f1f1-4777-ba2b-ba85078987f8 {"inputOptions":[],"outputOptions":["-c:v h264","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-v verbose","-vf scale=-2:720,format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false} [Nest] 7 - 11/11/2023, 10:40:29 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 67324e02-83c9-4e35-bfd5-1838bb212e8b [Nest] 7 - 11/11/2023, 10:40:29 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 5787e921-0fda-42ea-ad9d-52f5c8674ca1 [Nest] 7 - 11/11/2023, 10:40:29 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 08a6b22c-bc3a-4631-bb84-25931a10db79 [Nest] 7 - 11/11/2023, 10:40:29 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 858292ce-2c72-4001-af39-f2e3b62eaf24 [Nest] 7 - 11/11/2023, 10:40:29 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 39cb6e73-611a-4da3-9bfa-31592fedf9d0 [Nest] 7 - 11/11/2023, 10:40:29 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset a15410ac-f6dc-4e52-9cf8-dd5d53442872 [Nest] 7 - 11/11/2023, 10:40:30 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset ef4c8d9d-563e-416f-83fc-3f9fc629a1b8 [Nest] 7 - 11/11/2023, 10:40:30 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset a59fe606-361f-4b49-9556-4f5f2400de51 [Nest] 7 - 11/11/2023, 10:40:30 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 305507fa-f1ad-4919-b2b9-0a23f5b6a80f [Nest] 7 - 11/11/2023, 10:40:30 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 828c0398-688c-4fcb-aca6-f7c8796af6f4 [Nest] 7 - 11/11/2023, 10:40:31 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 107d0edf-5af8-4715-858f-2c506701fdb5 [Nest] 7 - 11/11/2023, 10:40:31 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset a998937b-6d5d-45b8-ab69-96b09ce82a09 [Nest] 7 - 11/11/2023, 10:40:31 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 2b109898-c927-4500-bdfe-3f44fa8872b9 [Nest] 7 - 11/11/2023, 10:40:31 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 53973ab7-865a-428f-895a-b7c37a39d4af [Nest] 7 - 11/11/2023, 10:40:31 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset ec94298c-15e0-4b90-a370-fea5af895993 [Nest] 7 - 11/11/2023, 10:40:31 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset e748a8d7-edbe-4633-83a3-73b740776645 [Nest] 7 - 11/11/2023, 10:40:32 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset d365bef8-97ab-4d42-822f-60ba946d009a [Nest] 7 - 11/11/2023, 10:40:32 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset b1dbb2e9-76c3-4c04-b81c-3877ffa4ab45 [Nest] 7 - 11/11/2023, 10:40:32 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 7e955d33-efa3-44e6-86ed-cfa2d2e7c7b5 [Nest] 7 - 11/11/2023, 10:40:32 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset 9452cdb2-6970-4307-aaae-d997da21460a [Nest] 7 - 11/11/2023, 10:40:32 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 81c982ea-b615-4dc0-98b7-0a74647048fd [Nest] 7 - 11/11/2023, 10:40:33 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 9babfaae-93ae-416b-9715-8ad3522daeee [Nest] 7 - 11/11/2023, 10:40:33 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset c783a3da-c4e7-48a3-8b09-02b83683223d [Nest] 7 - 11/11/2023, 10:40:33 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset f0b305ef-c2ed-4596-85a7-5220008b3fb8 [Nest] 7 - 11/11/2023, 10:40:33 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset bd7c8369-e534-443c-acde-48d38d44d3df [Nest] 7 - 11/11/2023, 10:40:33 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset b619813f-68e5-43a4-83d3-4dea23e542ea [Nest] 7 - 11/11/2023, 10:40:33 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset a09f2532-3a05-490e-b657-90dc61dd132c [Nest] 7 - 11/11/2023, 10:40:33 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 3a45071b-564c-4920-bac7-36590026fcbc [Nest] 7 - 11/11/2023, 10:40:34 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 36c668a1-f80c-4bd2-92b2-97a2ebb8a90d [Nest] 7 - 11/11/2023, 10:40:34 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 0f002c0b-b22a-48d9-8d9d-762f29724e9d [Nest] 7 - 11/11/2023, 10:40:34 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset ce44252b-5b69-4e0c-8974-2a12126e1262 [Nest] 7 - 11/11/2023, 10:40:35 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset c951c90f-c060-4f8c-9762-816b85c6d4b4 [Nest] 7 - 11/11/2023, 10:40:35 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset b880b344-dff8-4223-b8c6-9672a9300a68 [Nest] 7 - 11/11/2023, 10:40:35 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 75492468-6a34-44b4-a009-de3f4b852809 [Nest] 7 - 11/11/2023, 10:40:35 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset bbdad5ee-7cea-4764-9ca9-a2f6f24923a0 [Nest] 7 - 11/11/2023, 10:40:35 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset 1f21432a-178f-4cae-834c-b68acb1d6f18 [Nest] 7 - 11/11/2023, 10:40:35 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 82b236aa-c8f4-4de9-819b-63464bc7962e [Nest] 7 - 11/11/2023, 10:40:36 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset a451de55-3c9d-4936-9fe1-fc5d97655565 [Nest] 7 - 11/11/2023, 10:40:37 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 0b6dc2b2-0970-426c-8b24-3ea0d9301d1f [Nest] 7 - 11/11/2023, 10:40:37 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 9f69ad08-6935-418a-a270-24e6b28477de [Nest] 7 - 11/11/2023, 10:40:38 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset 59b204e2-675c-4af7-b487-7bb8a76f3b07 [Nest] 7 - 11/11/2023, 10:40:38 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1997e16b-0824-4cae-9bf7-c1d0d3c20a9c [Nest] 7 - 11/11/2023, 10:40:38 PM LOG [MediaService] Encoding success 18d538c9-f1f1-4777-ba2b-ba85078987f8 [Nest] 7 - 11/11/2023, 10:40:39 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 0161c736-52e6-4d44-88c6-119d15a31753 [Nest] 7 - 11/11/2023, 10:40:39 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 318e68df-76dc-4afa-bdee-03ec00370015 [Nest] 7 - 11/11/2023, 10:40:39 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset bb01936b-3e9d-47b3-aedd-727b1ee7333a [Nest] 7 - 11/11/2023, 10:40:39 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset d1dea1c2-44cd-4b1f-9997-4335346fc984 [Nest] 7 - 11/11/2023, 10:40:39 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset eb49ac4d-5f83-419d-995e-dcca974b619f [Nest] 7 - 11/11/2023, 10:40:40 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset b804f7fc-313a-43e5-a2e0-7d35cc4067e6 [Nest] 7 - 11/11/2023, 10:40:40 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset 48eeb7bd-8fe6-40a1-bd0f-eeb3d53820d6 [Nest] 7 - 11/11/2023, 10:40:40 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1ab02e84-86cd-429b-aa52-89f732c9523d [Nest] 7 - 11/11/2023, 10:40:41 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset ce2da5ba-ba28-4ceb-9c4a-8aa5428336a6 [Nest] 7 - 11/11/2023, 10:40:41 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 26a07170-f251-422f-92c0-36eb6e95e7ac [Nest] 7 - 11/11/2023, 10:40:43 PM LOG [MediaService] Start encoding video f0f93ba4-6bdc-45ac-863a-37a2e6299bcf {"inputOptions":[],"outputOptions":["-c:v h264","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-v verbose","-vf scale=-2:720,format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false} [Nest] 7 - 11/11/2023, 10:40:43 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset f0db521c-dc7b-4544-90d2-73086e5dadee [Nest] 7 - 11/11/2023, 10:40:43 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 4de748f6-7043-405d-b01a-ff731db74a2c [Nest] 7 - 11/11/2023, 10:40:43 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset d00a6417-7b5b-432d-be47-5857414d4793 [Nest] 7 - 11/11/2023, 10:40:43 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset efb58fa6-97fe-47a1-a18f-6b64b0fa41f5 [Nest] 7 - 11/11/2023, 10:40:44 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 5f9f49cc-8959-469e-aea8-f670a7f9324f [Nest] 7 - 11/11/2023, 10:40:44 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 462d795f-1fc8-44e3-b250-9eb78bf40af5 [Nest] 7 - 11/11/2023, 10:40:44 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset d5319bde-e518-42ba-ad36-217ab72fc7f5 [Nest] 7 - 11/11/2023, 10:40:45 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset c934a1c6-05f5-4cae-b1ec-79e0adaf4633 [Nest] 7 - 11/11/2023, 10:40:45 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset 69a58bd0-36a6-4ce5-93c2-e35d35836b41 [Nest] 7 - 11/11/2023, 10:40:45 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset d7eb1c46-9607-4fb4-9456-a28e2862a123 [Nest] 7 - 11/11/2023, 10:40:45 PM LOG [MediaService] Successfully generated JPEG video thumbnail for asset ecaf5910-bf97-4390-9ebb-2438b02d278d [Nest] 7 - 11/11/2023, 10:40:46 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 35a85f3b-4c3e-4f54-9467-724079f6e10a [Nest] 7 - 11/11/2023, 10:40:46 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset d70c47f0-9b15-4da6-a06e-08f116b983ca [Nest] 7 - 11/11/2023, 10:40:47 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 6d4dd94f-2425-4488-bbee-cfe27d5f0ac5 [Nest] 7 - 11/11/2023, 10:40:47 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 7dc6e5fc-810c-4c99-8aac-fa88225e0570 [Nest] 7 - 11/11/2023, 10:40:47 PM LOG [MediaService] Encoding success f0f93ba4-6bdc-45ac-863a-37a2e6299bcf [Nest] 7 - 11/11/2023, 10:40:48 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 6800bdf4-45ca-41a6-aefd-e662d8df835f <--- Last few GCs ---> [7:0x2885a310000] 148393 ms: Mark-Compact 4040.3 (4132.0) -> 4022.4 (4131.5) MB, 1987.37 / 0.00 ms (average mu = 0.179, current mu = 0.084) allocation failure; scavenge might not succeed [7:0x2885a310000] 150789 ms: Mark-Compact 4040.3 (4132.0) -> 4024.4 (4133.0) MB, 2341.68 / 0.00 ms (average mu = 0.101, current mu = 0.022) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 13: 0x7fc662d19ef6 ``` ### The OS that Immich Server is running on NixOS 23.05 ### Version of Immich Server 1.85.0 ### Version of Immich Mobile App 1.85.0 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ["start.sh", "immich"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env.photos depends_on: - redis - database - typesense restart: unless-stopped networks: - internal immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: ["start.sh", "microservices"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env.photos depends_on: - redis - database - typesense restart: unless-stopped networks: - internal immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env.photos restart: unless-stopped networks: - internal immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release} env_file: - .env.photos restart: unless-stopped networks: - internal typesense: container_name: immich_typesense image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data # remove this to get debug messages - GLOG_minloglevel=1 volumes: - tsdata:/data restart: unless-stopped networks: - internal redis: container_name: immich_redis image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3 restart: unless-stopped networks: - internal database: container_name: immich_postgres image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441 env_file: - .env.photos environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: unless-stopped networks: - internal immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release} depends_on: - immich-server - immich-web restart: unless-stopped labels: - traefik.enable=true - traefik.http.routers.immich.rule=Host(`xxxxx.example.com`) - traefik.http.routers.immich.entrypoints=websecure - traefik.http.routers.immich.tls.certresolver=myresolver - traefik.http.services.immich.loadbalancer.server.port=8080 networks: - web - internal volumes: pgdata: model-cache: tsdata: networks: web: external: name: traefik_webgateway internal: internal: true ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=/mnt/data/immich # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secrets for postgres and typesense. You should change these to random passwords TYPESENSE_API_KEY=xxxxx DB_PASSWORD=xxxxxxx # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis LOG_LEVEL=debug ``` ### Reproduction steps ```bash 1. Start immich microservice ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 02:37:08 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Nov 12, 2023):

I've seen a lot of reports lately of this issue when running on the TrueNas scale environment, I am not sure what is the root issue.

If you can add this to the VM's environment variables that you are running Immich

NODE_OPTIONS="--max-old-space-size=8192"
@alextran1502 commented on GitHub (Nov 12, 2023): I've seen a lot of reports lately of this issue when running on the TrueNas scale environment, I am not sure what is the root issue. If you can add this to the VM's environment variables that you are running Immich ``` NODE_OPTIONS="--max-old-space-size=8192" ```
OVERLORD commented 2026-02-05 02:37:13 +03:00
Author
Owner
Copy Link

@CorentinGrard commented on GitHub (Nov 12, 2023):

I've tried that but the container still crash after eating lot more memory

<--- Last few GCs --->
[7:0x3efd6310000]   950404 ms: Scavenge 8056.3 (8222.7) -> 8042.9 (8222.7) MB, 26.43 / 0.00 ms  (average mu = 0.059, current mu = 0.010) allocation failure;
[7:0x3efd6310000]   950528 ms: Scavenge 8057.9 (8222.7) -> 8046.2 (8223.5) MB, 39.88 / 0.00 ms  (average mu = 0.059, current mu = 0.010) allocation failure;
[7:0x3efd6310000]  1073056 ms: Mark-Compact 8061.9 (8225.2) -> 8044.0 (8225.7) MB, 120888.41 / 0.00 ms  (average mu = 0.038, current mu = 0.017) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99960 node::Abort() [immich_microservices]
 2: 0xb6ffcb  [immich_microservices]
 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 5: 0x10d06a5  [immich_microservices]
 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
13: 0x7ff8aca99ef6
image
@CorentinGrard commented on GitHub (Nov 12, 2023): I've tried that but the container still crash after eating lot more memory ``` <--- Last few GCs ---> [7:0x3efd6310000] 950404 ms: Scavenge 8056.3 (8222.7) -> 8042.9 (8222.7) MB, 26.43 / 0.00 ms (average mu = 0.059, current mu = 0.010) allocation failure; [7:0x3efd6310000] 950528 ms: Scavenge 8057.9 (8222.7) -> 8046.2 (8223.5) MB, 39.88 / 0.00 ms (average mu = 0.059, current mu = 0.010) allocation failure; [7:0x3efd6310000] 1073056 ms: Mark-Compact 8061.9 (8225.2) -> 8044.0 (8225.7) MB, 120888.41 / 0.00 ms (average mu = 0.038, current mu = 0.017) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 13: 0x7ff8aca99ef6 ``` <img width="888" alt="image" src="https://github.com/immich-app/immich/assets/25911136/cb05dced-aef8-4533-8948-b9c0a6dbab49">
OVERLORD commented 2026-02-05 02:37:17 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Nov 12, 2023):

Does anything change if you turn off reverse geocoding?

@jrasm91 commented on GitHub (Nov 12, 2023): Does anything change if you turn off reverse geocoding?
OVERLORD commented 2026-02-05 02:37:21 +03:00
Author
Owner
Copy Link

@CorentinGrard commented on GitHub (Nov 12, 2023):

Disabling reverse geocoding solved the issue and now the thumbnails generation can go through ! Any idea of what is happening and how I could turn it back on without crashing ?

@CorentinGrard commented on GitHub (Nov 12, 2023): Disabling reverse geocoding solved the issue and now the thumbnails generation can go through ! Any idea of what is happening and how I could turn it back on without crashing ?
OVERLORD commented 2026-02-05 02:37:23 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Nov 12, 2023):

Suuuuucks, no. Probably something dumb though. We use a 3rd party implementation and it seems to be the bane of my existence.

@jrasm91 commented on GitHub (Nov 12, 2023): Suuuuucks, no. Probably something dumb though. We use a 3rd party implementation and it seems to be the bane of my existence.
OVERLORD commented 2026-02-05 02:37:26 +03:00
Author
Owner
Copy Link

@CorentinGrard commented on GitHub (Nov 12, 2023):

Alright, at least it's now my instance is usable. Thank you for the help !

@CorentinGrard commented on GitHub (Nov 12, 2023): Alright, at least it's now my instance is usable. Thank you for the help !
OVERLORD commented 2026-02-05 02:37:30 +03:00
Author
Owner
Copy Link

@yodatak commented on GitHub (Nov 16, 2023):

Got the same issue in k3s with helm chart

@yodatak commented on GitHub (Nov 16, 2023): Got the same issue in k3s with helm chart
OVERLORD commented 2026-02-05 02:37:36 +03:00
Author
Owner
Copy Link

@flymin commented on GitHub (Nov 24, 2023):

Does anything change if you turn off reverse geocoding?

Same problem here on truenas. Turning reverse geocoding off helps. Why cannot enable this feature?

@flymin commented on GitHub (Nov 24, 2023): > Does anything change if you turn off reverse geocoding? Same problem here on truenas. Turning reverse geocoding off helps. Why cannot enable this feature?
OVERLORD commented 2026-02-05 02:37:44 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Nov 24, 2023):

We use a third party library for this and it has lots of issues. We're actually rewriting it from scratch into immich right now because of that. Once merged it should be possible to use the feature reliably.

@jrasm91 commented on GitHub (Nov 24, 2023): We use a third party library for this and it has lots of issues. We're actually rewriting it from scratch into immich right now because of that. Once merged it should be possible to use the feature reliably.
OVERLORD commented 2026-02-05 02:37:49 +03:00
Author
Owner
Copy Link

@zackpollard commented on GitHub (Nov 26, 2023):

In the next release we have completely reworked how reverse geocoding will function. We are replacing the old library with our own implementation that uses postgres for doing the reverse geocoding. This should result in much lower memory usage overall, and also should eliminate the microservices container running out of memory. We believe this was happening due to some issue with the logic in the old geocoding implementation which was not written by us. If you still continue to experience these issues after the next release (1.89.0) please open a new issue describing the problem. Cheers!

@zackpollard commented on GitHub (Nov 26, 2023): In the next release we have completely reworked how reverse geocoding will function. We are replacing the old library with our own implementation that uses postgres for doing the reverse geocoding. This should result in much lower memory usage overall, and also should eliminate the microservices container running out of memory. We believe this was happening due to some issue with the logic in the old geocoding implementation which was not written by us. If you still continue to experience these issues after the next release (1.89.0) please open a new issue describing the problem. Cheers!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1604
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?