immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 11 Packages Projects Releases 10 Wiki Activity

[BUG] Memory Leak kills Host due to full Swap #1685

New Issue
Closed
opened 2026-02-05 03:07:57 +03:00 by OVERLORD · 6 comments
OVERLORD commented 2026-02-05 03:07:57 +03:00
Owner
Copy Link

Originally created by @haldi4803 on GitHub (Nov 23, 2023).

The bug

Hello,

I've setup an Immich Container with an existing Library of about 150gb so i was expecting heavy load. But after 1-2 Days i realised that my whole system was running on overload and not a single docker container could be reached. The system was busy to such an extend that an SSH connection took minutes to respond!
After a reboot i realized that the immich-microservices filled the RAM and the Swap.
Whenever this happened again i simply killed the immich-microservices docker and after a few minutes the system came back alive.
Tried to limit RAM usage im Docker Container. That did not Help... the RAM in the Docker Container itself does NOT go over the Limit! But the System Still uses shittons of RAM anyway.
So i upgraded from 8 to 16gb RAM... tought that would Help. But nope. It just took a little Longer until it happened.

Here is a screenshot that shows docker stats which are within Limits, hTop which shows full Sawp and CPU usage. And the Synology DSM RAM usage. Shorty before the system becomes unresponsive.
image

Here is a Screenshot one minute prior.
image

Here is a snipet from the Log after i used docker kill to shutdown the container.


    at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP. (node:net:337:12)
[Nest] 7  - 11/23/2023, 3:47:03 PM   ERROR [JobService] Object:
{
  "id": "aa7f1567-9ee5-4e62-827e-c87e4dd002d4"
}
[Nest] 7  - 11/23/2023, 3:47:06 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 49e29a06-2f75-445c-ac92-f9e88ce6d21f
[Nest] 7  - 11/23/2023, 3:47:15 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset b08975b2-91f7-4d55-9064-f30ccc34852a
[Nest] 7  - 11/23/2023, 3:47:18 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 0ad7c605-2932-4c8d-9999-a1a18ab8d357
[Nest] 7  - 11/23/2023, 3:47:26 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 2f8140d7-f522-480d-939e-125397ff4839
[Nest] 7  - 11/23/2023, 3:47:27 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 4be850f0-9c88-447d-9e7c-5993a73f2e97
[Nest] 7  - 11/23/2023, 3:47:37 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 11/23/2023, 3:47:37 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:385:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:572:24)
[Nest] 7  - 11/23/2023, 3:47:37 PM   ERROR [JobService] Object:
{
  "id": "ba257e05-6b84-427d-b0fc-064271cd6896"
}
[Nest] 7  - 11/23/2023, 3:47:37 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 11/23/2023, 3:47:37 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:183:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:385:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:572:24)
[Nest] 7  - 11/23/2023, 3:47:37 PM   ERROR [JobService] Object:
{
  "id": "9cd8ee26-41ad-4743-9733-b1a86a232304"
}
[Nest] 7  - 11/23/2023, 3:47:37 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1e6e02f2-270a-41da-be0b-42d67fdb8113
[Nest] 7  - 11/23/2023, 3:47:38 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: Connection terminated due to connection timeout
[Nest] 7  - 11/23/2023, 3:47:38 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
    at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP. (node:net:337:12)
[Nest] 7  - 11/23/2023, 3:47:38 PM   ERROR [JobService] Object:
{
  "id": "aa7f1567-9ee5-4e62-827e-c87e4dd002d4"
}
[Nest] 7  - 11/23/2023, 3:47:42 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset b1743549-4ef2-4717-94bb-d03dc787ce7f
[Nest] 7  - 11/23/2023, 3:47:48 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: Connection terminated due to connection timeout
[Nest] 7  - 11/23/2023, 3:47:48 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
    at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP. (node:net:337:12)
[Nest] 7  - 11/23/2023, 3:47:48 PM   ERROR [JobService] Object:
{
  "id": "ba257e05-6b84-427d-b0fc-064271cd6896"
}
[Nest] 7  - 11/23/2023, 3:47:48 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: Connection terminated due to connection timeout
[Nest] 7  - 11/23/2023, 3:47:48 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
    at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:628:28)
    at Connection.emit (node:events:514:28)
    at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
    at Socket.emit (node:events:514:28)
    at TCP. (node:net:337:12)
[Nest] 7  - 11/23/2023, 3:47:48 PM   ERROR [JobService] Object:
{
  "id": "9d1bffa4-e9de-4fdc-a892-234bcd9c50b2"
}
[Nest] 7  - 11/23/2023, 3:47:51 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset a2bfb5e4-6013-4b79-af59-cbdbbc94805b
[Nest] 7  - 11/23/2023, 3:47:51 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset e4410be2-406e-416b-b0ba-8f1195875bc3
[Nest] 7  - 11/23/2023, 3:47:57 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 75bce33f-32b5-4570-b9b4-094e79e4acd9
[Nest] 7  - 11/23/2023, 3:48:08 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 49668ae5-a15e-44bc-92a5-4850ae11b850
[Nest] 7  - 11/23/2023, 3:48:16 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1e8f6bb4-04ea-4d6d-a81e-35eac4fa8cfb
[Nest] 7  - 11/23/2023, 3:48:16 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1786fd75-69a6-418e-960a-3b4eb46377da
[Nest] 7  - 11/23/2023, 3:48:18 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 5c5c4ad0-9eac-49fc-9c66-13724a7b19ae
[Nest] 7  - 11/23/2023, 3:48:24 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset da8592fb-ea05-4a33-8982-9bb7ade2582d
[Nest] 7  - 11/23/2023, 3:49:07 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset c55346ff-f7f3-4c0b-b931-711b80c71be5
[Nest] 7  - 11/23/2023, 3:49:07 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 2818ffd2-b88e-4849-a794-edcafee7a9e6
[Nest] 7  - 11/23/2023, 3:49:09 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 39da27ce-56de-4059-ab21-089ec7cb3b3a
[Nest] 7  - 11/23/2023, 3:49:09 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 27c871f3-fea0-4b59-b8f7-b4350de37d30
[Nest] 7  - 11/23/2023, 3:49:31 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset d76b2e43-43d1-4d41-b681-e85184ea041c
[Nest] 7  - 11/23/2023, 3:49:41 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 7dcc731b-45de-43b6-afcd-a9e8dc420f9d
[Nest] 7  - 11/23/2023, 3:49:50 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1796d8f3-a7c9-4b14-a812-1cf741ecc71c

The Memory Limitation on the Container itself seems to be working.
But is it the interContainer transport of such Huge data that causes issues?

Log from Machine-Learning.


[11/23/23 15:32:57] INFO     Starting gunicorn 21.2.0                           
[11/23/23 15:32:57] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/23/23 15:32:57] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/23/23 15:32:58] INFO     Booting worker with pid: 17                        
[11/23/23 15:34:58] CRITICAL WORKER TIMEOUT (pid:17)                            
[11/23/23 15:34:59] ERROR    Worker (pid:17) was sent SIGKILL! Perhaps out of   
                             memory?                                            
[11/23/23 15:34:59] INFO     Booting worker with pid: 26                        
[11/23/23 15:36:40] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[11/23/23 15:36:40] INFO     Initialized request thread pool with 8 threads.    
[11/23/23 15:45:47] INFO     Loading facial recognition model 'buffalo_l'       
[11/23/23 15:45:52] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                              
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNextImageProcessor instead.
  warnings.warn(
[11/23/23 15:45:54] INFO     Loading clip model 'ViT-B-32__openai'              
[11/23/23 15:54:14] INFO     Shutting down due to inactivity.                   
[11/23/23 15:54:22] ERROR    Worker (pid:26) exited with code 1                 
[11/23/23 15:54:22] ERROR    Worker (pid:26) exited with code 1.                
[11/23/23 15:54:22] INFO     Booting worker with pid: 70                        
[11/23/23 15:54:47] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[11/23/23 15:54:47] INFO     Initialized request thread pool with 8 threads.

Log from Server


[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +88ms
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +8ms
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 3:36:02 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 3:36:05 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +3299ms
[Nest] 7  - 11/23/2023, 3:36:05 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 7  - 11/23/2023, 3:36:05 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 3:36:05 PM     LOG [InstanceLoader] InfraModule dependencies initialized +5ms
[Nest] 7  - 11/23/2023, 3:36:05 PM     LOG [InstanceLoader] AppModule dependencies initialized +10ms
[Nest] 7  - 11/23/2023, 3:36:05 PM     LOG [InstanceLoader] DomainModule dependencies initialized +25ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] ActivityController {/api/activity}: +790ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/activity, GET} route +5ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/activity/statistics, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/activity, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/activity/:id, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] AssetsController {/api/assets}: +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/assets, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] AssetController {/api/asset}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/map-marker, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/memory-lane, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/random, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/download/info, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/download/archive, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/download/:id, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/statistics, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/time-buckets, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/time-bucket, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/jobs, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset, PUT} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/restore, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/trash/empty, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/trash/restore, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/stack/parent, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/:id, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] AssetController {/api/asset}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/upload, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/file/:id, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/thumbnail/:id, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/curated-objects, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/curated-locations, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/search-terms, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/:deviceId, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/assetById/:id, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/exist, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/asset/bulk-upload-check, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] AppController {/api}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/.well-known/immich, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/custom.css, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/refresh-config, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] AlbumController {/api/album}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album/count, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album/:id, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album/:id, PATCH} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album/:id, DELETE} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album/:id/assets, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album/:id/assets, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album/:id/users, PUT} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/album/:id/user/:userId, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] APIKeyController {/api/api-key}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/api-key, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/api-key, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, PUT} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] AuditController {/api/audit}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/audit/deletes, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report/checksum, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report/fix, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] AuthController {/api/auth}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/auth/login, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/auth/admin-sign-up, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/auth/devices, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/auth/devices, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/auth/devices/:id, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/auth/validateToken, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/auth/change-password, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/auth/logout, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] JobController {/api/jobs}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/jobs, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/jobs/:id, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] LibraryController {/api/library}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/library, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/library, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/library/:id, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/library/:id, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/library/:id, DELETE} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/library/:id/statistics, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/library/:id/scan, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/library/:id/removeOffline, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] OAuthController {/api/oauth}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/oauth/mobile-redirect, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/oauth/config, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/oauth/authorize, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/oauth/callback, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/oauth/link, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/oauth/unlink, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] PartnerController {/api/partner}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/partner, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, PUT} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] SearchController {/api/search}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/search/person, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/search, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/search/explore, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] ServerInfoController {/api/server-info}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/server-info, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/server-info/ping, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/server-info/version, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/server-info/features, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/server-info/theme, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/server-info/config, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/server-info/statistics, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/server-info/media-types, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] SharedLinkController {/api/shared-link}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/shared-link, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/shared-link/me, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/shared-link, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, PATCH} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, DELETE} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] SystemConfigController {/api/system-config}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/system-config, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/system-config/defaults, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/system-config, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/system-config/map/style.json, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] TagController {/api/tag}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/tag, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/tag, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, PATCH} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] UserController {/api/user}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user/info/:id, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user/me, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user/:id, DELETE} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user/:id/restore, POST} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image/:id, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RoutesResolver] PersonController {/api/person}: +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/person, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/person, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/person/:id, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/person/:id, PUT} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/person/:id/statistics, GET} route +1ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/person/:id/thumbnail, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/person/:id/assets, GET} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [RouterExplorer] Mapped {/api/person/:id/merge, POST} route +0ms
[Nest] 7  - 11/23/2023, 3:36:06 PM     LOG [SearchService] Running bootstrap
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753766863: Sleeping for 4s and then retrying request...
/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23
        var _this = _super.call(this, message) || this;
                           ^
ServerError: Request failed with HTTP code 503 | Server said: Not Ready or Lagging
    at ServerError.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ServerError (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ServerError.js:25:42)
    at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:376:21)
    at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:225:58
    at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
    at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
    at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
    at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
    at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  httpStatus: 503
}
Node.js v20.8.1
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [NestFactory] Starting Nest application...
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +95ms
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +2ms
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +9ms
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 6  - 11/23/2023, 3:37:29 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +403ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [InstanceLoader] InfraModule dependencies initialized +5ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [InstanceLoader] AppModule dependencies initialized +12ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [InstanceLoader] DomainModule dependencies initialized +25ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] ActivityController {/api/activity}: +243ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/activity, GET} route +5ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/activity/statistics, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/activity, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/activity/:id, DELETE} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] AssetsController {/api/assets}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/assets, GET} route +2ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] AssetController {/api/asset}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/map-marker, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/memory-lane, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/random, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/download/info, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/download/archive, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/download/:id, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/statistics, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/time-buckets, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/time-bucket, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/jobs, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset, PUT} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/restore, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/trash/empty, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/trash/restore, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/stack/parent, PUT} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/:id, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] AssetController {/api/asset}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/upload, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/file/:id, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/thumbnail/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/curated-objects, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/curated-locations, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/search-terms, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/:deviceId, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/assetById/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/exist, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/asset/bulk-upload-check, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] AppController {/api}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/.well-known/immich, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/custom.css, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/refresh-config, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] AlbumController {/api/album}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album/count, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album/:id, PATCH} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album/:id, DELETE} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album/:id/assets, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album/:id/assets, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album/:id/users, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/album/:id/user/:userId, DELETE} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] APIKeyController {/api/api-key}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/api-key, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/api-key, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, PUT} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] AuditController {/api/audit}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/audit/deletes, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report/checksum, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report/fix, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] AuthController {/api/auth}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/auth/login, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/auth/admin-sign-up, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/auth/devices, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/auth/devices, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/auth/devices/:id, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/auth/validateToken, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/auth/change-password, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/auth/logout, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] JobController {/api/jobs}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/jobs, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/jobs/:id, PUT} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] LibraryController {/api/library}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/library, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/library, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/library/:id, PUT} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/library/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/library/:id, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/library/:id/statistics, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/library/:id/scan, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/library/:id/removeOffline, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] OAuthController {/api/oauth}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/oauth/mobile-redirect, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/oauth/config, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/oauth/authorize, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/oauth/callback, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/oauth/link, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/oauth/unlink, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] PartnerController {/api/partner}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/partner, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] SearchController {/api/search}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/search/person, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/search, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/search/explore, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] ServerInfoController {/api/server-info}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/server-info, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/server-info/ping, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/server-info/version, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/server-info/features, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/server-info/theme, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/server-info/config, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/server-info/statistics, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/server-info/media-types, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] SharedLinkController {/api/shared-link}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/shared-link, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/shared-link/me, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/shared-link, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, PATCH} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, DELETE} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] SystemConfigController {/api/system-config}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/system-config, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/system-config/defaults, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/system-config, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/system-config/map/style.json, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] TagController {/api/tag}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/tag, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/tag, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, PATCH} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] UserController {/api/user}: +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user/info/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user/me, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image, DELETE} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user/:id, DELETE} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user/:id/restore, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image, POST} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RoutesResolver] PersonController {/api/person}: +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/person, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/person, PUT} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/person/:id, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/person/:id, PUT} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/person/:id/statistics, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/person/:id/thumbnail, GET} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/person/:id/assets, GET} route +1ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [RouterExplorer] Mapped {/api/person/:id/merge, POST} route +0ms
[Nest] 6  - 11/23/2023, 3:37:30 PM     LOG [SearchService] Running bootstrap
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700753850461: Sleeping for 4s and then retrying request...
[Nest] 6  - 11/23/2023, 3:38:18 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
[Nest] 6  - 11/23/2023, 3:38:18 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
[Nest] 6  - 11/23/2023, 3:38:18 PM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1


Any other logs i should provide ?

The OS that Immich Server is running on

Synology DSM 7.2.1

Version of Immich Server

1.88.2

Version of Immich Mobile App

v0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${EXISTING_LIBRARY}:/media/existinglibrary
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - 2283:8080
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${EXISTING_LIBRARY}:/media/existinglibrary
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:3995fe6ea6a619313e31046bd3c8643f9e70f8f2b294ff82659d409b47d06abb
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:874f566dd512d79cf74f59754833e869ae76ece96716d153b0fa3e64aec88d92
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

Portainer...

Reproduction steps

1.Install Docker on Synology
2. Import a huge existing library of Fotos
3. Microservices Containert uses HUGE Loads of RAM and Swap.
...

Additional information

No response

Originally created by @haldi4803 on GitHub (Nov 23, 2023). ### The bug Hello, I've setup an Immich Container with an existing Library of about 150gb so i was expecting heavy load. But after 1-2 Days i realised that my whole system was running on overload and not a single docker container could be reached. The system was busy to such an extend that an SSH connection took minutes to respond! After a reboot i realized that the immich-microservices filled the RAM and the Swap. Whenever this happened again i simply killed the immich-microservices docker and after a few minutes the system came back alive. Tried to limit RAM usage im Docker Container. That did not Help... the RAM in the Docker Container itself does NOT go over the Limit! But the System Still uses shittons of RAM anyway. So i upgraded from 8 to 16gb RAM... tought that would Help. But nope. It just took a little Longer until it happened. Here is a screenshot that shows docker stats which are within Limits, hTop which shows full Sawp and CPU usage. And the Synology DSM RAM usage. Shorty before the system becomes unresponsive. ![image](https://github.com/immich-app/immich/assets/17104473/cdcba855-38d0-45ab-8d86-b4ce0bfb5c47) Here is a Screenshot one minute prior. ![image](https://github.com/immich-app/immich/assets/17104473/446cad20-27bc-4988-91a1-5abdca6947a1) Here is a snipet from the Log after i used docker kill to shutdown the container. <details> <pre><code> at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) [Nest] 7 - 11/23/2023, 3:47:03 PM ERROR [JobService] Object: { "id": "aa7f1567-9ee5-4e62-827e-c87e4dd002d4" } [Nest] 7 - 11/23/2023, 3:47:06 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 49e29a06-2f75-445c-ac92-f9e88ce6d21f [Nest] 7 - 11/23/2023, 3:47:15 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset b08975b2-91f7-4d55-9064-f30ccc34852a [Nest] 7 - 11/23/2023, 3:47:18 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 0ad7c605-2932-4c8d-9999-a1a18ab8d357 [Nest] 7 - 11/23/2023, 3:47:26 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 2f8140d7-f522-480d-939e-125397ff4839 [Nest] 7 - 11/23/2023, 3:47:27 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 4be850f0-9c88-447d-9e7c-5993a73f2e97 [Nest] 7 - 11/23/2023, 3:47:37 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 11/23/2023, 3:47:37 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:385:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:572:24) [Nest] 7 - 11/23/2023, 3:47:37 PM ERROR [JobService] Object: { "id": "ba257e05-6b84-427d-b0fc-064271cd6896" } [Nest] 7 - 11/23/2023, 3:47:37 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 11/23/2023, 3:47:37 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:183:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:385:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:572:24) [Nest] 7 - 11/23/2023, 3:47:37 PM ERROR [JobService] Object: { "id": "9cd8ee26-41ad-4743-9733-b1a86a232304" } [Nest] 7 - 11/23/2023, 3:47:37 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1e6e02f2-270a-41da-be0b-42d67fdb8113 [Nest] 7 - 11/23/2023, 3:47:38 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: Connection terminated due to connection timeout [Nest] 7 - 11/23/2023, 3:47:38 PM ERROR [JobService] Error: Connection terminated due to connection timeout at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) [Nest] 7 - 11/23/2023, 3:47:38 PM ERROR [JobService] Object: { "id": "aa7f1567-9ee5-4e62-827e-c87e4dd002d4" } [Nest] 7 - 11/23/2023, 3:47:42 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset b1743549-4ef2-4717-94bb-d03dc787ce7f [Nest] 7 - 11/23/2023, 3:47:48 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: Connection terminated due to connection timeout [Nest] 7 - 11/23/2023, 3:47:48 PM ERROR [JobService] Error: Connection terminated due to connection timeout at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) [Nest] 7 - 11/23/2023, 3:47:48 PM ERROR [JobService] Object: { "id": "ba257e05-6b84-427d-b0fc-064271cd6896" } [Nest] 7 - 11/23/2023, 3:47:48 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: Connection terminated due to connection timeout [Nest] 7 - 11/23/2023, 3:47:48 PM ERROR [JobService] Error: Connection terminated due to connection timeout at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) [Nest] 7 - 11/23/2023, 3:47:48 PM ERROR [JobService] Object: { "id": "9d1bffa4-e9de-4fdc-a892-234bcd9c50b2" } [Nest] 7 - 11/23/2023, 3:47:51 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset a2bfb5e4-6013-4b79-af59-cbdbbc94805b [Nest] 7 - 11/23/2023, 3:47:51 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset e4410be2-406e-416b-b0ba-8f1195875bc3 [Nest] 7 - 11/23/2023, 3:47:57 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 75bce33f-32b5-4570-b9b4-094e79e4acd9 [Nest] 7 - 11/23/2023, 3:48:08 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 49668ae5-a15e-44bc-92a5-4850ae11b850 [Nest] 7 - 11/23/2023, 3:48:16 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1e8f6bb4-04ea-4d6d-a81e-35eac4fa8cfb [Nest] 7 - 11/23/2023, 3:48:16 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1786fd75-69a6-418e-960a-3b4eb46377da [Nest] 7 - 11/23/2023, 3:48:18 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 5c5c4ad0-9eac-49fc-9c66-13724a7b19ae [Nest] 7 - 11/23/2023, 3:48:24 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset da8592fb-ea05-4a33-8982-9bb7ade2582d [Nest] 7 - 11/23/2023, 3:49:07 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset c55346ff-f7f3-4c0b-b931-711b80c71be5 [Nest] 7 - 11/23/2023, 3:49:07 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 2818ffd2-b88e-4849-a794-edcafee7a9e6 [Nest] 7 - 11/23/2023, 3:49:09 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 39da27ce-56de-4059-ab21-089ec7cb3b3a [Nest] 7 - 11/23/2023, 3:49:09 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 27c871f3-fea0-4b59-b8f7-b4350de37d30 [Nest] 7 - 11/23/2023, 3:49:31 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset d76b2e43-43d1-4d41-b681-e85184ea041c [Nest] 7 - 11/23/2023, 3:49:41 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 7dcc731b-45de-43b6-afcd-a9e8dc420f9d [Nest] 7 - 11/23/2023, 3:49:50 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 1796d8f3-a7c9-4b14-a812-1cf741ecc71c </pre></code> </details> The Memory Limitation on the Container itself seems to be working. But is it the interContainer transport of such Huge data that causes issues? Log from Machine-Learning. <details> <pre><code> [11/23/23 15:32:57] INFO Starting gunicorn 21.2.0 [11/23/23 15:32:57] INFO Listening at: http://0.0.0.0:3003 (9) [11/23/23 15:32:57] INFO Using worker: uvicorn.workers.UvicornWorker [11/23/23 15:32:58] INFO Booting worker with pid: 17 [11/23/23 15:34:58] CRITICAL WORKER TIMEOUT (pid:17) [11/23/23 15:34:59] ERROR Worker (pid:17) was sent SIGKILL! Perhaps out of memory? [11/23/23 15:34:59] INFO Booting worker with pid: 26 [11/23/23 15:36:40] INFO Created in-memory cache with unloading after 300s of inactivity. [11/23/23 15:36:40] INFO Initialized request thread pool with 8 threads. [11/23/23 15:45:47] INFO Loading facial recognition model 'buffalo_l' [11/23/23 15:45:52] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNextImageProcessor instead. warnings.warn( [11/23/23 15:45:54] INFO Loading clip model 'ViT-B-32__openai' [11/23/23 15:54:14] INFO Shutting down due to inactivity. [11/23/23 15:54:22] ERROR Worker (pid:26) exited with code 1 [11/23/23 15:54:22] ERROR Worker (pid:26) exited with code 1. [11/23/23 15:54:22] INFO Booting worker with pid: 70 [11/23/23 15:54:47] INFO Created in-memory cache with unloading after 300s of inactivity. [11/23/23 15:54:47] INFO Initialized request thread pool with 8 threads. </pre></code> </details> Log from Server <details> <pre><code> [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [NestFactory] Starting Nest application... [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +88ms [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] ConfigModule dependencies initialized +8ms [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 3:36:02 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 3:36:05 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +3299ms [Nest] 7 - 11/23/2023, 3:36:05 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms [Nest] 7 - 11/23/2023, 3:36:05 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 3:36:05 PM LOG [InstanceLoader] InfraModule dependencies initialized +5ms [Nest] 7 - 11/23/2023, 3:36:05 PM LOG [InstanceLoader] AppModule dependencies initialized +10ms [Nest] 7 - 11/23/2023, 3:36:05 PM LOG [InstanceLoader] DomainModule dependencies initialized +25ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] ActivityController {/api/activity}: +790ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/activity, GET} route +5ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/activity/statistics, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/activity, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/activity/:id, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] AssetsController {/api/assets}: +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/assets, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] AssetController {/api/asset}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/map-marker, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/memory-lane, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/random, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/download/info, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/download/archive, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/download/:id, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/statistics, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/time-buckets, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/time-bucket, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/jobs, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset, PUT} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/restore, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/trash/empty, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/trash/restore, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/stack/parent, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/:id, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] AssetController {/api/asset}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/upload, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/file/:id, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/thumbnail/:id, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/curated-objects, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/curated-locations, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/search-terms, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/:deviceId, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/assetById/:id, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/exist, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/asset/bulk-upload-check, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] AppController {/api}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/.well-known/immich, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/custom.css, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/refresh-config, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] AlbumController {/api/album}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album/count, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album/:id, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album/:id, PATCH} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album/:id, DELETE} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album/:id/assets, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album/:id/assets, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album/:id/users, PUT} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/album/:id/user/:userId, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] APIKeyController {/api/api-key}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/api-key, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/api-key, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/api-key/:id, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/api-key/:id, PUT} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/api-key/:id, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] AuditController {/api/audit}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/audit/deletes, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/audit/file-report, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/audit/file-report/checksum, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/audit/file-report/fix, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] AuthController {/api/auth}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/auth/login, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/auth/admin-sign-up, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/auth/devices, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/auth/devices, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/auth/devices/:id, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/auth/validateToken, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/auth/change-password, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/auth/logout, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] JobController {/api/jobs}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/jobs, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/jobs/:id, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] LibraryController {/api/library}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/library, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/library, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/library/:id, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/library/:id, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/library/:id, DELETE} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/library/:id/statistics, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/library/:id/scan, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/library/:id/removeOffline, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] OAuthController {/api/oauth}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/oauth/mobile-redirect, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/oauth/config, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/oauth/authorize, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/oauth/callback, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/oauth/link, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/oauth/unlink, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] PartnerController {/api/partner}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/partner, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/partner/:id, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/partner/:id, PUT} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/partner/:id, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] SearchController {/api/search}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/search/person, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/search, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/search/explore, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] ServerInfoController {/api/server-info}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/server-info, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/server-info/ping, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/server-info/version, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/server-info/features, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/server-info/theme, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/server-info/config, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/server-info/statistics, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/server-info/media-types, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] SharedLinkController {/api/shared-link}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/shared-link, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/shared-link/me, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/shared-link, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id, PATCH} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id, DELETE} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] SystemConfigController {/api/system-config}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/system-config, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/system-config/defaults, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/system-config, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/system-config/map/style.json, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] TagController {/api/tag}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/tag, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/tag, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/tag/:id, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/tag/:id, PATCH} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/tag/:id, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/tag/:id/assets, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/tag/:id/assets, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/tag/:id/assets, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] UserController {/api/user}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user/info/:id, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user/me, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user/profile-image, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user/:id, DELETE} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user/:id/restore, POST} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user/profile-image, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/user/profile-image/:id, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RoutesResolver] PersonController {/api/person}: +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/person, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/person, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/person/:id, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/person/:id, PUT} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/person/:id/statistics, GET} route +1ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/person/:id/thumbnail, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/person/:id/assets, GET} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [RouterExplorer] Mapped {/api/person/:id/merge, POST} route +0ms [Nest] 7 - 11/23/2023, 3:36:06 PM LOG [SearchService] Running bootstrap Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... Request #1700753766863: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753766863: Sleeping for 4s and then retrying request... /usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23 var _this = _super.call(this, message) || this; ^ ServerError: Request failed with HTTP code 503 | Server said: Not Ready or Lagging at ServerError.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) at new ServerError (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ServerError.js:25:42) at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:376:21) at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:225:58 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { httpStatus: 503 } Node.js v20.8.1 [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [NestFactory] Starting Nest application... [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +95ms [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +2ms [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] ConfigModule dependencies initialized +9ms [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 6 - 11/23/2023, 3:37:29 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +403ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [InstanceLoader] InfraModule dependencies initialized +5ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [InstanceLoader] AppModule dependencies initialized +12ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [InstanceLoader] DomainModule dependencies initialized +25ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] ActivityController {/api/activity}: +243ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/activity, GET} route +5ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/activity/statistics, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/activity, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/activity/:id, DELETE} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] AssetsController {/api/assets}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/assets, GET} route +2ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] AssetController {/api/asset}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/map-marker, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/memory-lane, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/random, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/download/info, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/download/archive, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/download/:id, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/statistics, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/time-buckets, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/time-bucket, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/jobs, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset, PUT} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/restore, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/trash/empty, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/trash/restore, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/stack/parent, PUT} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/:id, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] AssetController {/api/asset}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/upload, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/file/:id, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/thumbnail/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/curated-objects, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/curated-locations, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/search-terms, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/:deviceId, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/assetById/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/exist, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/asset/bulk-upload-check, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] AppController {/api}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/.well-known/immich, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/custom.css, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/refresh-config, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] AlbumController {/api/album}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album/count, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album/:id, PATCH} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album/:id, DELETE} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album/:id/assets, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album/:id/assets, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album/:id/users, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/album/:id/user/:userId, DELETE} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] APIKeyController {/api/api-key}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/api-key, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/api-key, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/api-key/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/api-key/:id, PUT} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/api-key/:id, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] AuditController {/api/audit}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/audit/deletes, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/audit/file-report, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/audit/file-report/checksum, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/audit/file-report/fix, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] AuthController {/api/auth}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/auth/login, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/auth/admin-sign-up, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/auth/devices, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/auth/devices, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/auth/devices/:id, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/auth/validateToken, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/auth/change-password, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/auth/logout, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] JobController {/api/jobs}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/jobs, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/jobs/:id, PUT} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] LibraryController {/api/library}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/library, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/library, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/library/:id, PUT} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/library/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/library/:id, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/library/:id/statistics, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/library/:id/scan, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/library/:id/removeOffline, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] OAuthController {/api/oauth}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/oauth/mobile-redirect, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/oauth/config, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/oauth/authorize, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/oauth/callback, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/oauth/link, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/oauth/unlink, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] PartnerController {/api/partner}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/partner, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/partner/:id, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/partner/:id, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/partner/:id, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] SearchController {/api/search}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/search/person, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/search, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/search/explore, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] ServerInfoController {/api/server-info}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/server-info, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/server-info/ping, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/server-info/version, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/server-info/features, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/server-info/theme, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/server-info/config, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/server-info/statistics, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/server-info/media-types, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] SharedLinkController {/api/shared-link}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/shared-link, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/shared-link/me, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/shared-link, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id, PATCH} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id, DELETE} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] SystemConfigController {/api/system-config}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/system-config, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/system-config/defaults, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/system-config, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/system-config/map/style.json, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] TagController {/api/tag}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/tag, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/tag, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/tag/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/tag/:id, PATCH} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/tag/:id, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/tag/:id/assets, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/tag/:id/assets, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/tag/:id/assets, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] UserController {/api/user}: +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user/info/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user/me, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user/profile-image, DELETE} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user/:id, DELETE} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user/:id/restore, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user/profile-image, POST} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/user/profile-image/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RoutesResolver] PersonController {/api/person}: +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/person, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/person, PUT} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/person/:id, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/person/:id, PUT} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/person/:id/statistics, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/person/:id/thumbnail, GET} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/person/:id/assets, GET} route +1ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [RouterExplorer] Mapped {/api/person/:id/merge, POST} route +0ms [Nest] 6 - 11/23/2023, 3:37:30 PM LOG [SearchService] Running bootstrap Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... Request #1700753850461: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1700753850461: Sleeping for 4s and then retrying request... [Nest] 6 - 11/23/2023, 3:38:18 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 [Nest] 6 - 11/23/2023, 3:38:18 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 [Nest] 6 - 11/23/2023, 3:38:18 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1  </pre></code> </details> Any other logs i should provide ? ### The OS that Immich Server is running on Synology DSM 7.2.1 ### Version of Immich Server 1.88.2 ### Version of Immich Mobile App v0 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - ${EXISTING_LIBRARY}:/media/existinglibrary - /etc/localtime:/etc/localtime:ro env_file: - stack.env ports: - 2283:8080 depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - ${EXISTING_LIBRARY}:/media/existinglibrary - /etc/localtime:/etc/localtime:ro env_file: - stack.env depends_on: - redis - database - typesense restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - stack.env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data # remove this to get debug messages - GLOG_minloglevel=1 volumes: - tsdata:/data restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:3995fe6ea6a619313e31046bd3c8643f9e70f8f2b294ff82659d409b47d06abb restart: always database: container_name: immich_postgres image: postgres:14-alpine@sha256:874f566dd512d79cf74f59754833e869ae76ece96716d153b0fa3e64aec88d92 env_file: - stack.env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always volumes: pgdata: model-cache: tsdata: ``` ### Your .env content ```Shell Portainer... ``` ### Reproduction steps ```bash 1.Install Docker on Synology 2. Import a huge existing library of Fotos 3. Microservices Containert uses HUGE Loads of RAM and Swap. ... ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 03:08:03 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Nov 23, 2023):

There is obviously some type of memory issue. The fetch failed also usually means machine learning url is misconfigured or the container is unreachable. Can you confirm that the container is up and machine learning works for a single photo, when you regenerate the thumbnail for it?

@jrasm91 commented on GitHub (Nov 23, 2023): There is obviously some type of memory issue. The fetch failed also usually means machine learning url is misconfigured or the container is unreachable. Can you confirm that the container is up and machine learning works for a single photo, when you regenerate the thumbnail for it?
OVERLORD commented 2026-02-05 03:08:06 +03:00
Author
Owner
Copy Link

@haldi4803 commented on GitHub (Nov 23, 2023):

uhm... okay...

I've paused everything in the Job Status.
Everything is still normal now.
Then i continue the "Generate Thumbnails" and boom. ram goes full.

image

Server:

[Nest] 6  - 11/23/2023, 4:19:08 PM     LOG [CommunicationRepository] Websocket Disconnect: wXLrI6T_AHBx-iU1AAAD
[Nest] 6  - 11/23/2023, 5:50:58 PM     LOG [CommunicationRepository] Websocket Connect:    qhRn9z0Uzv7ljOtWAAAF
[Nest] 6  - 11/23/2023, 5:51:58 PM     LOG [CommunicationRepository] Websocket Disconnect: qhRn9z0Uzv7ljOtWAAAF
[Nest] 6  - 11/23/2023, 5:51:58 PM     LOG [CommunicationRepository] Websocket Connect:    rPCHd_E5BrfkQJxyAAAH
[Nest] 6  - 11/23/2023, 5:52:56 PM     LOG [CommunicationRepository] Websocket Disconnect: rPCHd_E5BrfkQJxyAAAH
[Nest] 6  - 11/23/2023, 5:52:56 PM     LOG [CommunicationRepository] Websocket Connect:    meMbMJLW3yn23yz7AAAJ

Microservices:

[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +46ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +7ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +334ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] DomainModule dependencies initialized +22ms
[Nest] 7  - 11/23/2023, 5:52:35 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [MetadataService] Initialized local reverse geocoder with cities500
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [SearchService] Running bootstrap
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
     
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [NestApplication] Nest application successfully started +65ms
[Nest] 7  - 11/23/2023, 5:52:48 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.88.2] [PRODUCTION] 
[Nest] 7  - 11/23/2023, 5:55:46 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 7f1266f4-98c6-4b61-8eaa-7d53765fc806
[Nest] 7  - 11/23/2023, 5:55:47 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset fc3e14ea-5230-45d9-a4e1-0577075231c6
[Nest] 7  - 11/23/2023, 5:55:56 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset bdfd3b89-79a5-41b4-a2ca-aaf10bb055ef
[Nest] 7  - 11/23/2023, 5:55:56 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 4fd3beca-27b4-4114-b4bd-cbd7958e0c20
[Nest] 7  - 11/23/2023, 5:56:10 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 78a51376-2434-49f5-8439-0105c66d7a1f
[Nest] 7  - 11/23/2023, 5:56:11 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset a5086c9c-2ebc-48e7-89eb-8b4680368fb2
[Nest] 7  - 11/23/2023, 5:56:55 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 912da659-c434-4aa8-b4ec-92933b15cb23
[Nest] 7  - 11/23/2023, 5:56:59 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset aa59021e-c7a7-4329-ac79-3abfe42a8134
[Nest] 7  - 11/23/2023, 5:56:59 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 5f8a2c1d-de13-43f9-b049-16f789179ef0
[Nest] 7  - 11/23/2023, 5:57:07 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset d5950662-7efb-49b2-b842-267d43707471
[Nest] 7  - 11/23/2023, 5:57:22 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 9bdd0b9d-2455-479f-9934-796438877054

Machine-Learning

[11/23/23 15:32:57] INFO     Starting gunicorn 21.2.0                           
[11/23/23 15:32:57] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/23/23 15:32:57] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/23/23 15:32:58] INFO     Booting worker with pid: 17                        
[11/23/23 15:34:58] CRITICAL WORKER TIMEOUT (pid:17)                            
[11/23/23 15:34:59] ERROR    Worker (pid:17) was sent SIGKILL! Perhaps out of   
                             memory?                                            
[11/23/23 15:34:59] INFO     Booting worker with pid: 26                        
[11/23/23 15:36:40] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[11/23/23 15:36:40] INFO     Initialized request thread pool with 8 threads.    
[11/23/23 15:45:47] INFO     Loading facial recognition model 'buffalo_l'       
[11/23/23 15:45:52] INFO     Loading image classification model                 
                             'microsoft/resnet-50'                              
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
/opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNextImageProcessor instead.
  warnings.warn(
[11/23/23 15:45:54] INFO     Loading clip model 'ViT-B-32__openai'              
[11/23/23 15:54:14] INFO     Shutting down due to inactivity.                   
[11/23/23 15:54:22] ERROR    Worker (pid:26) exited with code 1                 
[11/23/23 15:54:22] ERROR    Worker (pid:26) exited with code 1.                
[11/23/23 15:54:22] INFO     Booting worker with pid: 70                        
[11/23/23 15:54:47] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[11/23/23 15:54:47] INFO     Initialized request thread pool with 8 threads.

Anything usefull in there?

@haldi4803 commented on GitHub (Nov 23, 2023): uhm... okay... I've paused everything in the Job Status. Everything is still normal now. Then i continue the "Generate Thumbnails" and boom. ram goes full. ![image](https://github.com/immich-app/immich/assets/17104473/72a85195-697f-4abb-87bd-9ca862121f06) Server: ``` [Nest] 6 - 11/23/2023, 4:19:08 PM LOG [CommunicationRepository] Websocket Disconnect: wXLrI6T_AHBx-iU1AAAD [Nest] 6 - 11/23/2023, 5:50:58 PM LOG [CommunicationRepository] Websocket Connect: qhRn9z0Uzv7ljOtWAAAF [Nest] 6 - 11/23/2023, 5:51:58 PM LOG [CommunicationRepository] Websocket Disconnect: qhRn9z0Uzv7ljOtWAAAF [Nest] 6 - 11/23/2023, 5:51:58 PM LOG [CommunicationRepository] Websocket Connect: rPCHd_E5BrfkQJxyAAAH [Nest] 6 - 11/23/2023, 5:52:56 PM LOG [CommunicationRepository] Websocket Disconnect: rPCHd_E5BrfkQJxyAAAH [Nest] 6 - 11/23/2023, 5:52:56 PM LOG [CommunicationRepository] Websocket Connect: meMbMJLW3yn23yz7AAAJ ``` Microservices: ``` [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [NestFactory] Starting Nest application... [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +46ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] ConfigModule dependencies initialized +7ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +334ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] InfraModule dependencies initialized +3ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] DomainModule dependencies initialized +22ms [Nest] 7 - 11/23/2023, 5:52:35 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [MetadataService] Initialized local reverse geocoder with cities500 [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [SearchService] Running bootstrap [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1   [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}] [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false} [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [NestApplication] Nest application successfully started +65ms [Nest] 7 - 11/23/2023, 5:52:48 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.88.2] [PRODUCTION] [Nest] 7 - 11/23/2023, 5:55:46 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 7f1266f4-98c6-4b61-8eaa-7d53765fc806 [Nest] 7 - 11/23/2023, 5:55:47 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset fc3e14ea-5230-45d9-a4e1-0577075231c6 [Nest] 7 - 11/23/2023, 5:55:56 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset bdfd3b89-79a5-41b4-a2ca-aaf10bb055ef [Nest] 7 - 11/23/2023, 5:55:56 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 4fd3beca-27b4-4114-b4bd-cbd7958e0c20 [Nest] 7 - 11/23/2023, 5:56:10 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 78a51376-2434-49f5-8439-0105c66d7a1f [Nest] 7 - 11/23/2023, 5:56:11 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset a5086c9c-2ebc-48e7-89eb-8b4680368fb2 [Nest] 7 - 11/23/2023, 5:56:55 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 912da659-c434-4aa8-b4ec-92933b15cb23 [Nest] 7 - 11/23/2023, 5:56:59 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset aa59021e-c7a7-4329-ac79-3abfe42a8134 [Nest] 7 - 11/23/2023, 5:56:59 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 5f8a2c1d-de13-43f9-b049-16f789179ef0 [Nest] 7 - 11/23/2023, 5:57:07 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset d5950662-7efb-49b2-b842-267d43707471 [Nest] 7 - 11/23/2023, 5:57:22 PM LOG [MediaService] Successfully generated JPEG image thumbnail for asset 9bdd0b9d-2455-479f-9934-796438877054 ``` Machine-Learning ``` [11/23/23 15:32:57] INFO Starting gunicorn 21.2.0 [11/23/23 15:32:57] INFO Listening at: http://0.0.0.0:3003 (9) [11/23/23 15:32:57] INFO Using worker: uvicorn.workers.UvicornWorker [11/23/23 15:32:58] INFO Booting worker with pid: 17 [11/23/23 15:34:58] CRITICAL WORKER TIMEOUT (pid:17) [11/23/23 15:34:59] ERROR Worker (pid:17) was sent SIGKILL! Perhaps out of memory? [11/23/23 15:34:59] INFO Booting worker with pid: 26 [11/23/23 15:36:40] INFO Created in-memory cache with unloading after 300s of inactivity. [11/23/23 15:36:40] INFO Initialized request thread pool with 8 threads. [11/23/23 15:45:47] INFO Loading facial recognition model 'buffalo_l' [11/23/23 15:45:52] INFO Loading image classification model 'microsoft/resnet-50' Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. /opt/venv/lib/python3.11/site-packages/transformers/models/convnext/feature_extraction_convnext.py:28: FutureWarning: The class ConvNextFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ConvNextImageProcessor instead. warnings.warn( [11/23/23 15:45:54] INFO Loading clip model 'ViT-B-32__openai' [11/23/23 15:54:14] INFO Shutting down due to inactivity. [11/23/23 15:54:22] ERROR Worker (pid:26) exited with code 1 [11/23/23 15:54:22] ERROR Worker (pid:26) exited with code 1. [11/23/23 15:54:22] INFO Booting worker with pid: 70 [11/23/23 15:54:47] INFO Created in-memory cache with unloading after 300s of inactivity. [11/23/23 15:54:47] INFO Initialized request thread pool with 8 threads. ``` Anything usefull in there?
OVERLORD commented 2026-02-05 03:08:08 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 24, 2023):

I'm also a user and my library is around 600gb.

From your logs, it looks like ML is still only working on image categorization. What I often run into is face recognition and merging when the load will be huge.

Thumbnail work shouldn't cause the memory to run out, I've run into repeated restarts of typesense and massive memory usage before.

Suggestion:

  • Check the CPU and memory consumption of each VM independently.
  • Reduce the number of jobs appropriately and see if the load drops.
  • Stop the ML host, delete the typesense host and the volume (tsdata), then rebuild the typesense and limit the number of concurrency.
@davidpan commented on GitHub (Nov 24, 2023): I'm also a user and my library is around 600gb. From your logs, it looks like ML is still only working on image categorization. What I often run into is face recognition and merging when the load will be huge. Thumbnail work shouldn't cause the memory to run out, I've run into repeated restarts of typesense and massive memory usage before. Suggestion: - Check the CPU and memory consumption of each VM independently. - Reduce the number of jobs appropriately and see if the load drops. - Stop the ML host, delete the typesense host and the volume (tsdata), then rebuild the typesense and limit the number of concurrency.
OVERLORD commented 2026-02-05 03:08:13 +03:00
Author
Owner
Copy Link

@haldi4803 commented on GitHub (Dec 2, 2023):

i Think that might have happened because while immich was creating thumbnails i deleted the @eaDir folders from my NAS.
I had 114'000 images in the library instead of the 18'000 actually existing.

Might that bug happen if you try to access a file that does not exist anymore?

Did a full wipe and clean start, does not seem to happen anymore.

@haldi4803 commented on GitHub (Dec 2, 2023): i Think that might have happened because while immich was creating thumbnails i deleted the @eaDir folders from my NAS. I had 114'000 images in the library instead of the 18'000 actually existing. Might that bug happen if you try to access a file that does not exist anymore? Did a full wipe and clean start, does not seem to happen anymore.
OVERLORD commented 2026-02-05 03:08:16 +03:00
Author
Owner
Copy Link

@Dunky-Z commented on GitHub (May 12, 2024):

Today, I encountered the same issue. After bulk importing around 10,000 photos, I suddenly couldn't remotely connect to my NAS host. Initially, I suspected that the ML service was causing high memory usage during facial recognition, so I forcibly restarted the NAS and reconfigured concurrency settings, setting all jobs to run on a single thread. I didn't start all the jobs at once; instead, I sequentially ran FACE DETECTION and then GENERATE THUMBNAILS. There were no abnormalities during the FACE DETECTION process, indicating that the issue wasn't caused by the ML operations. After completing all FACE DETECTIONS, I initiated the GENERATE THUMBNAILS task, which ran slowly due to the large image sizes. Without continuous monitoring, upon returning to check on the NAS, I found it was unreachable remotely, with the system log displaying the following error messages:

[17452.279485] Out of memory: Killed process 24344 (immich_microser) total-vm:14375660kB, anon-rss:8827768kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:23516kB oom_score_adj:0
[17653.570542] Out of memory: Killed process 24972 (immich_microser) total-vm:13119600kB, anon-rss:8825796kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:20652kB oom_score_adj:0
[17848.345248] Out of memory: Killed process 26162 (immich_microser) total-vm:19086052kB, anon-rss:8709080kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:31688kB oom_score_adj:0
[17932.668353] Out of memory: Killed process 27348 (immich_microser) total-vm:13774836kB, anon-rss:8759228kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:22648kB oom_score_adj:0
[18001.034451] Out of memory: Killed process 27902 (immich_microser) total-vm:14749840kB, anon-rss:9111036kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:25104kB oom_score_adj:0

This suggests that there might be a memory leak issue within the immich_microser service. I would greatly appreciate any advice on how to address this problem.

@Dunky-Z commented on GitHub (May 12, 2024): Today, I encountered the same issue. After bulk importing around 10,000 photos, I suddenly couldn't remotely connect to my NAS host. Initially, I suspected that the ML service was causing high memory usage during facial recognition, so I forcibly restarted the NAS and reconfigured concurrency settings, setting all jobs to run on a single thread. I didn't start all the jobs at once; instead, I sequentially ran FACE DETECTION and then GENERATE THUMBNAILS. There were no abnormalities during the FACE DETECTION process, indicating that the issue wasn't caused by the ML operations. After completing all FACE DETECTIONS, I initiated the GENERATE THUMBNAILS task, which ran slowly due to the large image sizes. Without continuous monitoring, upon returning to check on the NAS, I found it was unreachable remotely, with the system log displaying the following error messages: ```bash [17452.279485] Out of memory: Killed process 24344 (immich_microser) total-vm:14375660kB, anon-rss:8827768kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:23516kB oom_score_adj:0 [17653.570542] Out of memory: Killed process 24972 (immich_microser) total-vm:13119600kB, anon-rss:8825796kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:20652kB oom_score_adj:0 [17848.345248] Out of memory: Killed process 26162 (immich_microser) total-vm:19086052kB, anon-rss:8709080kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:31688kB oom_score_adj:0 [17932.668353] Out of memory: Killed process 27348 (immich_microser) total-vm:13774836kB, anon-rss:8759228kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:22648kB oom_score_adj:0 [18001.034451] Out of memory: Killed process 27902 (immich_microser) total-vm:14749840kB, anon-rss:9111036kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:25104kB oom_score_adj:0 ``` This suggests that there might be a memory leak issue within the `immich_microser` service. I would greatly appreciate any advice on how to address this problem.
OVERLORD commented 2026-02-05 03:08:20 +03:00
Author
Owner
Copy Link

@PaeyMoopy commented on GitHub (Oct 2, 2024):

Hello,
I've been having this same issue for a couple of weeks.
I learned you could tweak facial recognition, so on a fresh install I let all the jobs finish, then tweaked the settings and re ran the jobs one at a time.

Now, I run into this exact issue every couple of days.. my RAM and SWAP trickle up to 100% when a few photos are uploaded, then the Immich container stops functioning until restarted. I've tried assigning more SWAP and RAM, but it only prolongs the issue.

My fix for now is just restarting the container when it happens, I don't have anything else in the container so it's an isolated issue, the rest of my self hosted stuff keeps operating normally.

Happy to share logs or my config files, they're mostly default. I'm not sure how to access the logs.

Cheers

@PaeyMoopy commented on GitHub (Oct 2, 2024): Hello, I've been having this same issue for a couple of weeks. I learned you could tweak facial recognition, so on a fresh install I let all the jobs finish, then tweaked the settings and re ran the jobs one at a time. Now, I run into this exact issue every couple of days.. my RAM and SWAP trickle up to 100% when a few photos are uploaded, then the Immich container stops functioning until restarted. I've tried assigning more SWAP and RAM, but it only prolongs the issue. My fix for now is just restarting the container when it happens, I don't have anything else in the container so it's an isolated issue, the rest of my self hosted stuff keeps operating normally. Happy to share logs or my config files, they're mostly default. I'm not sure how to access the logs. Cheers
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1685
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?