immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 08:41:28 +03:00
Code Issues 28 Packages Projects Releases 10 Wiki Activity

[BUG] After upgrading to 1.84, my Docker container "immich_microservices" kept restarting #1545

New Issue
Closed
opened 2026-02-05 02:18:41 +03:00 by OVERLORD · 16 comments
OVERLORD commented 2026-02-05 02:18:41 +03:00
Owner
Copy Link

Originally created by @nodis on GitHub (Nov 1, 2023).

The bug

My Docker container 'immich_microservices' keeps restarting, and the error log is as follows:

/


[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "b873495b-a5a6-4230-82fb-cb091374f6c9"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "c2366caa-73d4-405b-bf88-1a77451dd859"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "59cceeab-0870-400c-a65e-933d6aaa3235"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "55746629-751a-4842-ab92-6b7a7d027e90",
  "source": "upload"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "ee05681a-b166-475c-a345-8e45bc34bb8c",
  "source": "upload"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "a9aa0971-026d-4352-b61b-071d9b6c4e0c",
  "source": "upload"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "98eba74c-1622-429e-956c-b6856766f06a",
  "source": "upload"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "717f03d6-81f5-4804-bd61-9d5daa63dd17",
  "source": "upload"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "33034fdb-cc3e-4706-a03e-8c915fbe96f2"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "87be390d-b2b1-4a86-9cad-f6d8aca88156",
  "source": "upload"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "c7f1e4e6-045a-4d1b-a342-b2245296652e",
  "source": "upload"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "5e7b5144-744f-4720-9f51-6a2171c7cb71"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "45546c2e-3027-487c-acb4-36de72a2dad0"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "53e6dcb6-c39c-4fa9-a7c1-30908eb2ca1b"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "f63f1f94-26ec-4ad9-8689-477126ff8dc5"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "fc2280a3-f940-42b2-98a0-7656e46ca9f3"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "7acdbc45-01b5-4e83-ac12-d064aa7de0b6"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Object:
{
  "id": "26e2b938-648e-42cd-96b7-91221e6f37d3"
}

[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed
[Nest] 7  - 11/01/2023, 5:21:01 PM   ERROR [JobService] TypeError: fetch failed

<--- Last few GCs --->

[7:0x43882310000]  3867693 ms: Mark-Compact 4045.4 (4134.6) -> 4031.0 (4136.1) MB, 3143.26 / 0.00 ms  (average mu = 0.144, current mu = 0.018) allocation failure; scavenge might not succeed
[7:0x43882310000]  3870775 ms: Mark-Compact 4046.8 (4136.1) -> 4032.4 (4137.6) MB, 3027.55 / 0.00 ms  (average mu = 0.086, current mu = 0.018) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99960 node::Abort() [immich_microservices]
 2: 0xb6ffcb  [immich_microservices]
 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 5: 0x10d06a5  [immich_microservices]
 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
13: 0x7fa776499ef6 
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +130ms
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +5ms
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/01/2023, 5:54:07 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 7  - 11/01/2023, 5:54:08 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +1115ms
[Nest] 7  - 11/01/2023, 5:54:08 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 11/01/2023, 5:54:08 PM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
[Nest] 7  - 11/01/2023, 5:54:08 PM     LOG [InstanceLoader] DomainModule dependencies initialized +16ms
[Nest] 7  - 11/01/2023, 5:54:08 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 7  - 11/01/2023, 5:54:19 PM     LOG [JobService] Updating queue concurrency settings
[Nest] 7  - 11/01/2023, 5:54:19 PM     LOG [MetadataService] Initialized local reverse geocoder with cities500
[Nest] 7  - 11/01/2023, 5:54:28 PM     LOG [SearchService] Running bootstrap
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305213: Sleeping for 4s and then retrying request...
Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305352: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861306543: Sleeping for 4s and then retrying request...
Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305213: Sleeping for 4s and then retrying request...
Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305352: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861306543: Sleeping for 4s and then retrying request...
Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305213: Sleeping for 4s and then retrying request...
Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305352: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861306543: Sleeping for 4s and then retrying request...
Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861315921: Sleeping for 4s and then retrying request...
Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305213: Sleeping for 4s and then retrying request...
Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305352: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861306543: Sleeping for 4s and then retrying request...
Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861315921: Sleeping for 4s and then retrying request...
Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305213: Sleeping for 4s and then retrying request...
Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305352: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861322482: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861322482: Sleeping for 4s and then retrying request...
Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861306543: Sleeping for 4s and then retrying request...
Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861315921: Sleeping for 4s and then retrying request...
Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305213: Sleeping for 4s and then retrying request...
Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305352: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861322482: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861322482: Sleeping for 4s and then retrying request...
Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861306543: Sleeping for 4s and then retrying request...
Request #1698861327402: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861327402: Sleeping for 4s and then retrying request...
Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861315921: Sleeping for 4s and then retrying request...
Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305213: Sleeping for 4s and then retrying request...
Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305352: Sleeping for 4s and then retrying request...
Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861268187: Sleeping for 4s and then retrying request...
Request #1698861322482: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861322482: Sleeping for 4s and then retrying request...
Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861306543: Sleeping for 4s and then retrying request...
[Nest] 7  - 11/01/2023, 5:55:30 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Object:
{
  "id": "589ccf96-b697-40b0-81c7-3fdcfbe99444"
}

[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Object:
{
  "id": "e0b10c3c-f03d-4c75-8a1a-2939414d8a65"
}

[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Object:
{
  "id": "a3bb9ede-a2a2-41bd-86db-97fb22b41f5d"
}

[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Object:
{
  "id": "c0b9a5ee-1cdd-422a-ae80-0419ed94c940"
}

Request #1698861327402: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861327402: Sleeping for 4s and then retrying request...
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Object:
{
  "id": "7ac5327d-554f-4b1b-96ee-d6a172c190a4"
}

[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/01/2023, 5:55:31 PM   ERROR [JobService] Object:
{
  "id": "23549c50-34ce-4c35-a7ad-225e52a2b303"
}

Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861315921: Sleeping for 4s and then retrying request...
Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305213: Sleeping for 4s and then retrying request...
Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1698861305352: Sleeping for 4s and then retrying request...

<--- Last few GCs --->

[7:0x56dd0310000]   396548 ms: Mark-Compact 4045.1 (4135.0) -> 4030.7 (4136.5) MB, 1711.74 / 0.00 ms  (average mu = 0.175, current mu = 0.036) allocation failure; scavenge might not succeed
[7:0x56dd0310000]   399848 ms: Mark-Compact 4046.6 (4136.5) -> 4032.1 (4137.8) MB, 3239.63 / 0.00 ms  (average mu = 0.091, current mu = 0.018) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99960 node::Abort() [immich_microservices]
 2: 0xb6ffcb  [immich_microservices]
 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 5: 0x10d06a5  [immich_microservices]
 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
13: 0x7f62cd119ef6 
root@DSM7_PVE_106:/volume2/docker/immich#

The OS that Immich Server is running on

Synology

Version of Immich Server

1.84

Version of Immich Mobile App

1.84

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /volume2/photo:/media/volume2/photo:ro
      - /volume1:/media/volume1:ro
      - /volume2:/media/volume2:ro
      - /volume3:/media/volume3:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:rw
      - /volume2/photo:/media/volume2/photo:ro
      - /volume1:/media/volume1:ro
      - /volume2:/media/volume2:ro
      - /volume3:/media/volume3:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /volume2/docker/immich/machine/model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - /volume2/docker/immich/tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /volume2/docker/immich/pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/volume2/Immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=nodishNqo26NFRsFBmYymuv+KLcGbNBSKHQE99w2QD8=
DB_PASSWORD=mypassword

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=immich
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1.After upgrading to 1.84, my Docker container "immich_microservices" kept restarting
2.
3.
...

Additional information

No response

Originally created by @nodis on GitHub (Nov 1, 2023). ### The bug My Docker container 'immich_microservices' keeps restarting, and the error log is as follows: / ```typescript [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "b873495b-a5a6-4230-82fb-cb091374f6c9" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "c2366caa-73d4-405b-bf88-1a77451dd859" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "59cceeab-0870-400c-a65e-933d6aaa3235" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "55746629-751a-4842-ab92-6b7a7d027e90", "source": "upload" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "ee05681a-b166-475c-a345-8e45bc34bb8c", "source": "upload" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "a9aa0971-026d-4352-b61b-071d9b6c4e0c", "source": "upload" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "98eba74c-1622-429e-956c-b6856766f06a", "source": "upload" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "717f03d6-81f5-4804-bd61-9d5daa63dd17", "source": "upload" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "33034fdb-cc3e-4706-a03e-8c915fbe96f2" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "87be390d-b2b1-4a86-9cad-f6d8aca88156", "source": "upload" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:177:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "c7f1e4e6-045a-4d1b-a342-b2245296652e", "source": "upload" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "5e7b5144-744f-4720-9f51-6a2171c7cb71" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "45546c2e-3027-487c-acb4-36de72a2dad0" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "53e6dcb6-c39c-4fa9-a7c1-30908eb2ca1b" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "f63f1f94-26ec-4ad9-8689-477126ff8dc5" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "fc2280a3-f940-42b2-98a0-7656e46ca9f3" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "7acdbc45-01b5-4e83-ac12-d064aa7de0b6" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:85:31) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:350:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:535:24) [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Object: { "id": "26e2b938-648e-42cd-96b7-91221e6f37d3" } [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): TypeError: fetch failed [Nest] 7 - 11/01/2023, 5:21:01 PM ERROR [JobService] TypeError: fetch failed <--- Last few GCs ---> [7:0x43882310000] 3867693 ms: Mark-Compact 4045.4 (4134.6) -> 4031.0 (4136.1) MB, 3143.26 / 0.00 ms (average mu = 0.144, current mu = 0.018) allocation failure; scavenge might not succeed [7:0x43882310000] 3870775 ms: Mark-Compact 4046.8 (4136.1) -> 4032.4 (4137.6) MB, 3027.55 / 0.00 ms (average mu = 0.086, current mu = 0.018) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 13: 0x7fa776499ef6 [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [NestFactory] Starting Nest application... [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +130ms [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [InstanceLoader] ConfigModule dependencies initialized +5ms [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/01/2023, 5:54:07 PM LOG [InstanceLoader] BullModule dependencies initialized +1ms [Nest] 7 - 11/01/2023, 5:54:08 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +1115ms [Nest] 7 - 11/01/2023, 5:54:08 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms [Nest] 7 - 11/01/2023, 5:54:08 PM LOG [InstanceLoader] InfraModule dependencies initialized +2ms [Nest] 7 - 11/01/2023, 5:54:08 PM LOG [InstanceLoader] DomainModule dependencies initialized +16ms [Nest] 7 - 11/01/2023, 5:54:08 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 7 - 11/01/2023, 5:54:19 PM LOG [JobService] Updating queue concurrency settings [Nest] 7 - 11/01/2023, 5:54:19 PM LOG [MetadataService] Initialized local reverse geocoder with cities500 [Nest] 7 - 11/01/2023, 5:54:28 PM LOG [SearchService] Running bootstrap Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305213: Sleeping for 4s and then retrying request... Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305352: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861306543: Sleeping for 4s and then retrying request... Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305213: Sleeping for 4s and then retrying request... Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305352: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861306543: Sleeping for 4s and then retrying request... Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305213: Sleeping for 4s and then retrying request... Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305352: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861306543: Sleeping for 4s and then retrying request... Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861315921: Sleeping for 4s and then retrying request... Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305213: Sleeping for 4s and then retrying request... Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305352: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861306543: Sleeping for 4s and then retrying request... Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861315921: Sleeping for 4s and then retrying request... Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305213: Sleeping for 4s and then retrying request... Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305352: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861322482: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861322482: Sleeping for 4s and then retrying request... Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861306543: Sleeping for 4s and then retrying request... Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861315921: Sleeping for 4s and then retrying request... Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305213: Sleeping for 4s and then retrying request... Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305352: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861322482: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861322482: Sleeping for 4s and then retrying request... Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861306543: Sleeping for 4s and then retrying request... Request #1698861327402: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861327402: Sleeping for 4s and then retrying request... Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861315921: Sleeping for 4s and then retrying request... Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305213: Sleeping for 4s and then retrying request... Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305352: Sleeping for 4s and then retrying request... Request #1698861268187: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861268187: Sleeping for 4s and then retrying request... Request #1698861322482: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861322482: Sleeping for 4s and then retrying request... Request #1698861306543: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861306543: Sleeping for 4s and then retrying request... [Nest] 7 - 11/01/2023, 5:55:30 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Object: { "id": "589ccf96-b697-40b0-81c7-3fdcfbe99444" } [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Object: { "id": "e0b10c3c-f03d-4c75-8a1a-2939414d8a65" } [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Object: { "id": "a3bb9ede-a2a2-41bd-86db-97fb22b41f5d" } [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Object: { "id": "c0b9a5ee-1cdd-422a-ae80-0419ed94c940" } Request #1698861327402: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861327402: Sleeping for 4s and then retrying request... [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Object: { "id": "7ac5327d-554f-4b1b-96ee-d6a172c190a4" } [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/01/2023, 5:55:31 PM ERROR [JobService] Object: { "id": "23549c50-34ce-4c35-a7ad-225e52a2b303" } Request #1698861315921: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861315921: Sleeping for 4s and then retrying request... Request #1698861305213: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305213: Sleeping for 4s and then retrying request... Request #1698861305352: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1698861305352: Sleeping for 4s and then retrying request... <--- Last few GCs ---> [7:0x56dd0310000] 396548 ms: Mark-Compact 4045.1 (4135.0) -> 4030.7 (4136.5) MB, 1711.74 / 0.00 ms (average mu = 0.175, current mu = 0.036) allocation failure; scavenge might not succeed [7:0x56dd0310000] 399848 ms: Mark-Compact 4046.6 (4136.5) -> 4032.1 (4137.8) MB, 3239.63 / 0.00 ms (average mu = 0.091, current mu = 0.018) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 13: 0x7f62cd119ef6 root@DSM7_PVE_106:/volume2/docker/immich# ``` ### The OS that Immich Server is running on Synology ### Version of Immich Server 1.84 ### Version of Immich Mobile App 1.84 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ["start.sh", "immich"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - /volume2/photo:/media/volume2/photo:ro - /volume1:/media/volume1:ro - /volume2:/media/volume2:ro - /volume3:/media/volume3:ro env_file: - .env depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: ["start.sh", "microservices"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:rw - /volume2/photo:/media/volume2/photo:ro - /volume1:/media/volume1:ro - /volume2:/media/volume2:ro - /volume3:/media/volume3:ro env_file: - .env depends_on: - redis - database - typesense restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - /volume2/docker/immich/machine/model-cache:/cache env_file: - .env restart: always immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release} env_file: - .env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data # remove this to get debug messages - GLOG_minloglevel=1 volumes: - /volume2/docker/immich/tsdata:/data restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3 restart: always database: container_name: immich_postgres image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441 env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - /volume2/docker/immich/pgdata:/var/lib/postgresql/data restart: always immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release} environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 depends_on: - immich-server - immich-web restart: always volumes: pgdata: model-cache: tsdata: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=/volume2/Immich # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secrets for postgres and typesense. You should change these to random passwords TYPESENSE_API_KEY=nodishNqo26NFRsFBmYymuv+KLcGbNBSKHQE99w2QD8= DB_PASSWORD=mypassword # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=immich DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1.After upgrading to 1.84, my Docker container "immich_microservices" kept restarting 2. 3. ... ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 02:18:44 +03:00
Author
Owner
Copy Link

@sgithuku commented on GitHub (Nov 2, 2023):

Same issue here. Reverted to 1.83.0 (on my UnRaid install) for now

@sgithuku commented on GitHub (Nov 2, 2023): Same issue here. Reverted to 1.83.0 (on my UnRaid install) for now
OVERLORD commented 2026-02-05 02:18:49 +03:00
Author
Owner
Copy Link

@nodis commented on GitHub (Nov 2, 2023):

So far, my synology NAS has notified me more than 50 times: "The Container Manager container immich_microservices unexpectedly stopped. Please select immich_microservices on the container page, click the Details button, and then go to the Log tab for more information

@nodis commented on GitHub (Nov 2, 2023): So far, my synology NAS has notified me more than 50 times: "The Container Manager container immich_microservices unexpectedly stopped. Please select immich_microservices on the container page, click the Details button, and then go to the Log tab for more information
OVERLORD commented 2026-02-05 02:18:52 +03:00
Author
Owner
Copy Link

@dvbthien commented on GitHub (Nov 2, 2023):

So far, my synology NAS has notified me more than 50 times: "The Container Manager container immich_microservices unexpectedly stopped. Please select immich_microservices on the container page, click the Details button, and then go to the Log tab for more information

I use Synolgy, but I don't have this issue. When I upgrade to Immich v1.84, everything is normal. You may try to delete all models in the machine-cache folder and restart all services.

@dvbthien commented on GitHub (Nov 2, 2023): > So far, my synology NAS has notified me more than 50 times: "The Container Manager container immich_microservices unexpectedly stopped. Please select immich_microservices on the container page, click the Details button, and then go to the Log tab for more information I use Synolgy, but I don't have this issue. When I upgrade to Immich v1.84, everything is normal. You may try to delete all models in the machine-cache folder and restart all services.
OVERLORD commented 2026-02-05 02:18:59 +03:00
Author
Owner
Copy Link

@nodis commented on GitHub (Nov 2, 2023):

So far, my synology NAS has notified me more than 50 times: "The Container Manager container immich_microservices unexpectedly stopped. Please select immich_microservices on the container page, click the Details button, and then go to the Log tab for more information

I use Synolgy, but I don't have this issue. When I upgrade to Immich v1.84, everything is normal. You may try to delete all models in the machine-cache folder and restart all services.

I rebuilt 'immich', but 'immich_microservices' still stopped unexpectedly. Could you please take a look at my error log and provide me with some suggestions? thanks

@nodis commented on GitHub (Nov 2, 2023): > > So far, my synology NAS has notified me more than 50 times: "The Container Manager container immich_microservices unexpectedly stopped. Please select immich_microservices on the container page, click the Details button, and then go to the Log tab for more information > > I use Synolgy, but I don't have this issue. When I upgrade to Immich v1.84, everything is normal. You may try to delete all models in the machine-cache folder and restart all services. I rebuilt 'immich', but 'immich_microservices' still stopped unexpectedly. Could you please take a look at my error log and provide me with some suggestions? thanks
OVERLORD commented 2026-02-05 02:19:03 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Nov 2, 2023):

What is the ram are you allocating for Immich?

@alextran1502 commented on GitHub (Nov 2, 2023): What is the ram are you allocating for Immich?
OVERLORD commented 2026-02-05 02:19:06 +03:00
Author
Owner
Copy Link

@nodis commented on GitHub (Nov 2, 2023):

What is the ram are you allocating for Immich?

My Synology memory is 16GB, and the memory allocated to 'immich' is' unlimited '

@nodis commented on GitHub (Nov 2, 2023): > What is the ram are you allocating for Immich? My Synology memory is 16GB, and the memory allocated to 'immich' is' unlimited '
OVERLORD commented 2026-02-05 02:19:09 +03:00
Author
Owner
Copy Link

@dvbthien commented on GitHub (Nov 2, 2023):

I rebuilt 'immich', but 'immich_microservices' still stopped unexpectedly. Could you please take a look at my error log and provide me with some suggestions? thanks

Can you capture the container immich-machine-learning logs for me?

@dvbthien commented on GitHub (Nov 2, 2023): > I rebuilt 'immich', but 'immich_microservices' still stopped unexpectedly. Could you please take a look at my error log and provide me with some suggestions? thanks Can you capture the container immich-machine-learning logs for me?
OVERLORD commented 2026-02-05 02:19:13 +03:00
Author
Owner
Copy Link

@nodis commented on GitHub (Nov 2, 2023):

I rebuilt 'immich', but 'immich_microservices' still stopped unexpectedly. Could you please take a look at my error log and provide me with some suggestions? thanks

Can you capture the container immich-machine-learning logs for me?

root@DSM7_PVE_106:/volume2/docker/immich# docker logs immich_machine_learning
[11/01/23 16:47:09] INFO     Starting gunicorn 21.2.0                           
[11/01/23 16:47:09] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/01/23 16:47:09] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/01/23 16:47:09] INFO     Booting worker with pid: 13                        
[11/01/23 16:49:09] CRITICAL WORKER TIMEOUT (pid:13)                            
[11/01/23 16:49:10] ERROR    Worker (pid:13) was sent SIGKILL! Perhaps out of   
                             memory?                                            
[11/01/23 16:49:10] INFO     Booting worker with pid: 20                        
[11/01/23 16:49:42] INFO     Created in-memory cache with unloading disabled.   
[11/01/23 16:49:42] INFO     Initialized request thread pool with 4 threads.    
[11/01/23 16:49:42] INFO     Loading image classification model                 
                             'google/vit-base-patch16-224'                      
/opt/venv/lib/python3.11/site-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead.
  warnings.warn(
[11/01/23 16:49:52] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 16:52:00] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 16:52:00] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 16:52:00] INFO     Loading facial recognition model 'buffalo_l'       
[11/01/23 16:52:00] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 16:54:07] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 16:54:07] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 16:54:07] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 16:56:14] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 16:56:14] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 16:56:14] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 16:58:21] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 16:58:21] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 16:58:21] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 16:58:21] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 16:58:21] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 16:58:21] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:00:29] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:00:29] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:00:29] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:02:36] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:02:36] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:02:36] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:04:43] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:04:43] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:04:43] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:06:50] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:06:50] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:06:50] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:08:58] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:08:58] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:08:58] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:11:05] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:11:05] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:11:05] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:13:12] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:13:12] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:13:12] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:15:19] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:15:19] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:15:19] WARNING  Attempted to clear cache for model                 
                             'XLM-Roberta-Large-Vit-B-16Plus' but cache         
                             directory does not exist.                          
[11/01/23 17:17:26] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:17:26] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:17:26] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:19:34] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:19:34] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:19:34] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:21:41] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:21:41] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:21:41] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:23:48] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:23:48] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:23:48] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:25:56] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:25:56] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:25:56] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:28:03] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:28:03] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:28:03] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:30:11] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:30:11] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:30:11] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:32:18] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
Exception in ASGI application
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 203, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 492, in _make_request
    raise new_e
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    self._validate_conn(conn)
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1097, in _validate_conn
    conn.connect()
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 611, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 212, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f61d28614d0>, 'Connection to huggingface.co timed out. (connect timeout=None)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 845, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d28614d0>, 'Connection to huggingface.co timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/app/main.py", line 101, in load
    await loop.run_in_executor(app.state.thread_pool, _load)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 94, in _load
    model.load()
  File "/usr/src/app/models/base.py", line 63, in load
    self.download()
  File "/usr/src/app/models/base.py", line 58, in download
    self._download()
  File "/usr/src/app/models/clip.py", line 123, in _download
    snapshot_download(
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info
    return method(
           ^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send
    return super().send(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d28614d0>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: f00a7a6b-890c-4e27-b6c9-7471f4ea35d4)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 75, in predict
    model = await load(await app.state.model_cache.get(model_name, model_type, **kwargs))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 114, in load
    await loop.run_in_executor(app.state.thread_pool, _load)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 94, in _load
    model.load()
  File "/usr/src/app/models/base.py", line 63, in load
    self.download()
  File "/usr/src/app/models/base.py", line 58, in download
    self._download()
  File "/usr/src/app/models/clip.py", line 123, in _download
    snapshot_download(
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info
    return method(
           ^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send
    return super().send(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d26a4b50>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: 9761d87b-1d2e-4988-8d5b-95a0a0b4b0aa)')
[11/01/23 17:34:25] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
Exception in ASGI application
Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 203, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 492, in _make_request
    raise new_e
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 468, in _make_request
    self._validate_conn(conn)
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1097, in _validate_conn
    conn.connect()
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 611, in connect
    self.sock = sock = self._new_conn()
                       ^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 212, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f61d0583750>, 'Connection to huggingface.co timed out. (connect timeout=None)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 845, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d0583750>, 'Connection to huggingface.co timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/app/main.py", line 101, in load
    await loop.run_in_executor(app.state.thread_pool, _load)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 94, in _load
    model.load()
  File "/usr/src/app/models/base.py", line 63, in load
    self.download()
  File "/usr/src/app/models/base.py", line 58, in download
    self._download()
  File "/usr/src/app/models/clip.py", line 123, in _download
    snapshot_download(
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info
    return method(
           ^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send
    return super().send(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d0583750>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: caa5587f-b798-45d4-8bef-2b4447a3c49a)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 75, in predict
    model = await load(await app.state.model_cache.get(model_name, model_type, **kwargs))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 114, in load
    await loop.run_in_executor(app.state.thread_pool, _load)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/main.py", line 94, in _load
    model.load()
  File "/usr/src/app/models/base.py", line 63, in load
    self.download()
  File "/usr/src/app/models/base.py", line 58, in download
    self._download()
  File "/usr/src/app/models/clip.py", line 123, in _download
    snapshot_download(
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info
    return method(
           ^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info
    r = get_session().get(path, headers=headers, timeout=timeout, params=params)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send
    return super().send(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 507, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61c9bd3bd0>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: 05ab7f0c-5837-4b93-bd02-5a80957624ae)')
[11/01/23 17:36:33] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:36:33] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:36:33] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:38:40] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:38:40] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:38:40] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:40:47] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:40:47] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:40:47] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:42:54] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:42:54] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:42:54] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:45:01] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:45:01] INFO     Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus'
[11/01/23 17:45:01] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:45:05] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:45:06] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:45:06] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:47:12] INFO     Downloading clip model                             
                             'XLM-Roberta-Large-Vit-B-16Plus'.This may take a   
                             while.                                             
[11/01/23 17:47:12] WARNING  Failed to load clip model                          
                             'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and
                             retrying.                                          
[11/01/23 17:47:12] INFO     Cleared cache directory for model                  
                             'XLM-Roberta-Large-Vit-B-16Plus'.                  
[11/01/23 17:51:58] INFO     Starting gunicorn 21.2.0                           
[11/01/23 17:51:58] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/01/23 17:51:58] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/01/23 17:51:58] INFO     Booting worker with pid: 13                        
[11/01/23 17:53:58] CRITICAL WORKER TIMEOUT (pid:13)                            
[11/01/23 17:53:59] ERROR    Worker (pid:13) was sent SIGKILL! Perhaps out of   
                             memory?                                            
[11/01/23 17:53:59] INFO     Booting worker with pid: 18                        
[11/01/23 17:54:11] INFO     Created in-memory cache with unloading disabled.   
[11/01/23 17:54:11] INFO     Initialized request thread pool with 4 threads.    
[11/01/23 17:54:29] INFO     Loading image classification model                 
                             'google/vit-base-patch16-224'                      
/opt/venv/lib/python3.11/site-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead.
  warnings.warn(
[11/01/23 17:54:41] INFO     Loading facial recognition model 'buffalo_l'       
[11/01/23 17:54:46] INFO     Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus'
[11/02/23 00:23:09] INFO     Starting gunicorn 21.2.0                           
[11/02/23 00:23:09] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/02/23 00:23:09] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/02/23 00:23:09] INFO     Booting worker with pid: 13                        
[11/02/23 00:25:09] CRITICAL WORKER TIMEOUT (pid:13)                            
[11/02/23 00:25:10] ERROR    Worker (pid:13) was sent SIGKILL! Perhaps out of   
                             memory?                                            
[11/02/23 00:25:10] INFO     Booting worker with pid: 18                        
[11/02/23 00:27:59] INFO     Starting gunicorn 21.2.0                           
[11/02/23 00:27:59] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/02/23 00:27:59] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/02/23 00:27:59] INFO     Booting worker with pid: 13                        
[11/02/23 00:28:11] INFO     Created in-memory cache with unloading disabled.   
[11/02/23 00:28:11] INFO     Initialized request thread pool with 4 threads.    
[11/02/23 01:16:53] INFO     Loading image classification model                 
                             'google/vit-base-patch16-224'                      
/opt/venv/lib/python3.11/site-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead.
  warnings.warn(
[11/02/23 01:17:12] INFO     Loading facial recognition model 'buffalo_l'       
[11/02/23 01:17:24] INFO     Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus'
root@DSM7_PVE_106:/volume2/docker/immich#
@nodis commented on GitHub (Nov 2, 2023): > > I rebuilt 'immich', but 'immich_microservices' still stopped unexpectedly. Could you please take a look at my error log and provide me with some suggestions? thanks > > Can you capture the container immich-machine-learning logs for me? ```dart root@DSM7_PVE_106:/volume2/docker/immich# docker logs immich_machine_learning [11/01/23 16:47:09] INFO Starting gunicorn 21.2.0 [11/01/23 16:47:09] INFO Listening at: http://0.0.0.0:3003 (9) [11/01/23 16:47:09] INFO Using worker: uvicorn.workers.UvicornWorker [11/01/23 16:47:09] INFO Booting worker with pid: 13 [11/01/23 16:49:09] CRITICAL WORKER TIMEOUT (pid:13) [11/01/23 16:49:10] ERROR Worker (pid:13) was sent SIGKILL! Perhaps out of memory? [11/01/23 16:49:10] INFO Booting worker with pid: 20 [11/01/23 16:49:42] INFO Created in-memory cache with unloading disabled. [11/01/23 16:49:42] INFO Initialized request thread pool with 4 threads. [11/01/23 16:49:42] INFO Loading image classification model 'google/vit-base-patch16-224' /opt/venv/lib/python3.11/site-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead. warnings.warn( [11/01/23 16:49:52] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 16:52:00] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 16:52:00] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 16:52:00] INFO Loading facial recognition model 'buffalo_l' [11/01/23 16:52:00] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 16:54:07] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 16:54:07] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 16:54:07] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 16:56:14] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 16:56:14] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 16:56:14] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 16:58:21] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 16:58:21] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 16:58:21] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 16:58:21] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 16:58:21] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 16:58:21] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:00:29] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:00:29] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:00:29] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:02:36] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:02:36] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:02:36] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:04:43] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:04:43] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:04:43] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:06:50] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:06:50] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:06:50] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:08:58] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:08:58] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:08:58] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:11:05] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:11:05] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:11:05] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:13:12] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:13:12] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:13:12] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:15:19] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:15:19] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:15:19] WARNING Attempted to clear cache for model 'XLM-Roberta-Large-Vit-B-16Plus' but cache directory does not exist. [11/01/23 17:17:26] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:17:26] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:17:26] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:19:34] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:19:34] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:19:34] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:21:41] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:21:41] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:21:41] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:23:48] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:23:48] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:23:48] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:25:56] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:25:56] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:25:56] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:28:03] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:28:03] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:28:03] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:30:11] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:30:11] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:30:11] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:32:18] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. Exception in ASGI application Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 492, in _make_request raise new_e File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 468, in _make_request self._validate_conn(conn) File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1097, in _validate_conn conn.connect() File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 611, in connect self.sock = sock = self._new_conn() ^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 212, in _new_conn raise ConnectTimeoutError( urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f61d28614d0>, 'Connection to huggingface.co timed out. (connect timeout=None)') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d28614d0>, 'Connection to huggingface.co timed out. (connect timeout=None)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/src/app/main.py", line 101, in load await loop.run_in_executor(app.state.thread_pool, _load) File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 94, in _load model.load() File "/usr/src/app/models/base.py", line 63, in load self.download() File "/usr/src/app/models/base.py", line 58, in download self._download() File "/usr/src/app/models/clip.py", line 123, in _download snapshot_download( File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info return method( ^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info r = get_session().get(path, headers=headers, timeout=timeout, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send return super().send(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d28614d0>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: f00a7a6b-890c-4e27-b6c9-7471f4ea35d4)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__ await super().__call__(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__ await self.middleware_stack(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__ raise exc File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__ await self.app(scope, receive, _send) File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__ raise exc File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__ await self.app(scope, receive, sender) File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__ raise e File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__ await self.app(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__ await route.handle(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 75, in predict model = await load(await app.state.model_cache.get(model_name, model_type, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 114, in load await loop.run_in_executor(app.state.thread_pool, _load) File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 94, in _load model.load() File "/usr/src/app/models/base.py", line 63, in load self.download() File "/usr/src/app/models/base.py", line 58, in download self._download() File "/usr/src/app/models/clip.py", line 123, in _download snapshot_download( File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info return method( ^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info r = get_session().get(path, headers=headers, timeout=timeout, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send return super().send(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d26a4b50>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: 9761d87b-1d2e-4988-8d5b-95a0a0b4b0aa)') [11/01/23 17:34:25] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. Exception in ASGI application Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 203, in _new_conn sock = connection.create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 492, in _make_request raise new_e File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 468, in _make_request self._validate_conn(conn) File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1097, in _validate_conn conn.connect() File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 611, in connect self.sock = sock = self._new_conn() ^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 212, in _new_conn raise ConnectTimeoutError( urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7f61d0583750>, 'Connection to huggingface.co timed out. (connect timeout=None)') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d0583750>, 'Connection to huggingface.co timed out. (connect timeout=None)')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/src/app/main.py", line 101, in load await loop.run_in_executor(app.state.thread_pool, _load) File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 94, in _load model.load() File "/usr/src/app/models/base.py", line 63, in load self.download() File "/usr/src/app/models/base.py", line 58, in download self._download() File "/usr/src/app/models/clip.py", line 123, in _download snapshot_download( File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info return method( ^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info r = get_session().get(path, headers=headers, timeout=timeout, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send return super().send(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61d0583750>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: caa5587f-b798-45d4-8bef-2b4447a3c49a)') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/applications.py", line 276, in __call__ await super().__call__(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__ await self.middleware_stack(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__ raise exc File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__ await self.app(scope, receive, _send) File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__ raise exc File "/opt/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__ await self.app(scope, receive, sender) File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__ raise e File "/opt/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__ await self.app(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__ await route.handle(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/opt/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 75, in predict model = await load(await app.state.model_cache.get(model_name, model_type, **kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 114, in load await loop.run_in_executor(app.state.thread_pool, _load) File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/app/main.py", line 94, in _load model.load() File "/usr/src/app/models/base.py", line 63, in load self.download() File "/usr/src/app/models/base.py", line 58, in download self._download() File "/usr/src/app/models/clip.py", line 123, in _download snapshot_download( File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info return method( ^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info r = get_session().get(path, headers=headers, timeout=timeout, params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send return super().send(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 507, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/XLM-Roberta-Large-Vit-B-16Plus/revision/main (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f61c9bd3bd0>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: 05ab7f0c-5837-4b93-bd02-5a80957624ae)') [11/01/23 17:36:33] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:36:33] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:36:33] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:38:40] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:38:40] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:38:40] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:40:47] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:40:47] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:40:47] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:42:54] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:42:54] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:42:54] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:45:01] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:45:01] INFO Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus' [11/01/23 17:45:01] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:45:05] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:45:06] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:45:06] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:47:12] INFO Downloading clip model 'XLM-Roberta-Large-Vit-B-16Plus'.This may take a while. [11/01/23 17:47:12] WARNING Failed to load clip model 'XLM-Roberta-Large-Vit-B-16Plus'.Clearing cache and retrying. [11/01/23 17:47:12] INFO Cleared cache directory for model 'XLM-Roberta-Large-Vit-B-16Plus'. [11/01/23 17:51:58] INFO Starting gunicorn 21.2.0 [11/01/23 17:51:58] INFO Listening at: http://0.0.0.0:3003 (9) [11/01/23 17:51:58] INFO Using worker: uvicorn.workers.UvicornWorker [11/01/23 17:51:58] INFO Booting worker with pid: 13 [11/01/23 17:53:58] CRITICAL WORKER TIMEOUT (pid:13) [11/01/23 17:53:59] ERROR Worker (pid:13) was sent SIGKILL! Perhaps out of memory? [11/01/23 17:53:59] INFO Booting worker with pid: 18 [11/01/23 17:54:11] INFO Created in-memory cache with unloading disabled. [11/01/23 17:54:11] INFO Initialized request thread pool with 4 threads. [11/01/23 17:54:29] INFO Loading image classification model 'google/vit-base-patch16-224' /opt/venv/lib/python3.11/site-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead. warnings.warn( [11/01/23 17:54:41] INFO Loading facial recognition model 'buffalo_l' [11/01/23 17:54:46] INFO Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus' [11/02/23 00:23:09] INFO Starting gunicorn 21.2.0 [11/02/23 00:23:09] INFO Listening at: http://0.0.0.0:3003 (9) [11/02/23 00:23:09] INFO Using worker: uvicorn.workers.UvicornWorker [11/02/23 00:23:09] INFO Booting worker with pid: 13 [11/02/23 00:25:09] CRITICAL WORKER TIMEOUT (pid:13) [11/02/23 00:25:10] ERROR Worker (pid:13) was sent SIGKILL! Perhaps out of memory? [11/02/23 00:25:10] INFO Booting worker with pid: 18 [11/02/23 00:27:59] INFO Starting gunicorn 21.2.0 [11/02/23 00:27:59] INFO Listening at: http://0.0.0.0:3003 (9) [11/02/23 00:27:59] INFO Using worker: uvicorn.workers.UvicornWorker [11/02/23 00:27:59] INFO Booting worker with pid: 13 [11/02/23 00:28:11] INFO Created in-memory cache with unloading disabled. [11/02/23 00:28:11] INFO Initialized request thread pool with 4 threads. [11/02/23 01:16:53] INFO Loading image classification model 'google/vit-base-patch16-224' /opt/venv/lib/python3.11/site-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead. warnings.warn( [11/02/23 01:17:12] INFO Loading facial recognition model 'buffalo_l' [11/02/23 01:17:24] INFO Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus' root@DSM7_PVE_106:/volume2/docker/immich# ```
OVERLORD commented 2026-02-05 02:19:18 +03:00
Author
Owner
Copy Link

@nodis commented on GitHub (Nov 2, 2023):

Same issue here. Reverted to 1.83.0 (on my UnRaid install) for now

May I ask how you recover to 1.83.0?

@nodis commented on GitHub (Nov 2, 2023): > Same issue here. Reverted to 1.83.0 (on my UnRaid install) for now May I ask how you recover to 1.83.0?
OVERLORD commented 2026-02-05 02:19:23 +03:00
Author
Owner
Copy Link

@dvbthien commented on GitHub (Nov 2, 2023):

May I ask how you recover to 1.83.0?

In your .env file, change the following line: IMMICH_VERSION=release to IMMICH_VERSION=v1.83.0
Then repull image.
About the cause of error: I think the file model for machine learning is incorrect. Have you tried deleting all files in /volume2/docker/immich/machine/model-cache?

@dvbthien commented on GitHub (Nov 2, 2023): > May I ask how you recover to 1.83.0? In your .env file, change the following line: IMMICH_VERSION=release to IMMICH_VERSION=v1.83.0 Then repull image. About the cause of error: I think the file model for machine learning is incorrect. Have you tried deleting all files in /volume2/docker/immich/machine/model-cache?
OVERLORD commented 2026-02-05 02:19:26 +03:00
Author
Owner
Copy Link

@nodis commented on GitHub (Nov 2, 2023):

May I ask how you recover to 1.83.0?

In your .env file, change the following line: IMMICH_VERSION=release to IMMICH_VERSION=v1.83.0 Then repull image. About the cause of error: I think the file model for machine learning is incorrect. Have you tried deleting all files in /volume2/docker/immich/machine/model-cache?

I tried deleting all files in /volume2/docker/immich/machine/model-cache, but the problem still exists

log：

docker logs immich_machine_learning：

[11/01/23 17:54:41] INFO     Loading facial recognition model 'buffalo_l'       
[11/01/23 17:54:46] INFO     Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus'
[11/02/23 00:23:09] INFO     Starting gunicorn 21.2.0                           
[11/02/23 00:23:09] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/02/23 00:23:09] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/02/23 00:23:09] INFO     Booting worker with pid: 13                        
[11/02/23 00:25:09] CRITICAL WORKER TIMEOUT (pid:13)                            
[11/02/23 00:25:10] ERROR    Worker (pid:13) was sent SIGKILL! Perhaps out of   
                             memory?                                            
[11/02/23 00:25:10] INFO     Booting worker with pid: 18                        
[11/02/23 00:27:59] INFO     Starting gunicorn 21.2.0                           
[11/02/23 00:27:59] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/02/23 00:27:59] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/02/23 00:27:59] INFO     Booting worker with pid: 13                        
[11/02/23 00:28:11] INFO     Created in-memory cache with unloading disabled.   
[11/02/23 00:28:11] INFO     Initialized request thread pool with 4 threads.    
[11/02/23 01:16:53] INFO     Loading image classification model                 
                             'google/vit-base-patch16-224'                      
/opt/venv/lib/python3.11/site-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead.
  warnings.warn(
[11/02/23 01:17:12] INFO     Loading facial recognition model 'buffalo_l'       
[11/02/23 01:17:24] INFO     Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus'
[11/02/23 02:51:54] INFO     Starting gunicorn 21.2.0                           
[11/02/23 02:51:54] INFO     Listening at: http://0.0.0.0:3003 (9)              
[11/02/23 02:51:54] INFO     Using worker: uvicorn.workers.UvicornWorker        
[11/02/23 02:51:54] INFO     Booting worker with pid: 13                        
[11/02/23 02:53:55] CRITICAL WORKER TIMEOUT (pid:13)                            
[11/02/23 02:53:57] ERROR    Worker (pid:13) was sent SIGKILL! Perhaps out of   
                             memory?                                            
[11/02/23 02:53:57] INFO     Booting worker with pid: 18                        
[11/02/23 02:55:49] INFO     Created in-memory cache with unloading disabled.   
[11/02/23 02:55:49] INFO     Initialized request thread pool with 4 threads.

docker logs immich_microservices：



<--- Last few GCs --->

[8:0x2d6b0310000]   208029 ms: Mark-Compact 4040.7 (4130.0) -> 4022.8 (4130.5) MB, 4719.06 / 0.00 ms  (average mu = 0.133, current mu = 0.053) allocation failure; scavenge might not succeed
[8:0x2d6b0310000]   212890 ms: Mark-Compact 4040.6 (4130.5) -> 4024.8 (4131.0) MB, 4703.15 / 0.00 ms  (average mu = 0.083, current mu = 0.033) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99960 node::Abort() [immich_microservices]
 2: 0xb6ffcb  [immich_microservices]
 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
 5: 0x10d06a5  [immich_microservices]
 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices]
12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
13: 0x7fd57aa59ef6 
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +30ms
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +1ms
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +4ms
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/02/2023, 3:00:09 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/02/2023, 3:00:11 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +2087ms
[Nest] 7  - 11/02/2023, 3:00:11 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 11/02/2023, 3:00:11 AM     LOG [InstanceLoader] InfraModule dependencies initialized +2ms
[Nest] 7  - 11/02/2023, 3:00:11 AM     LOG [InstanceLoader] DomainModule dependencies initialized +17ms
[Nest] 7  - 11/02/2023, 3:00:11 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 7  - 11/02/2023, 3:00:23 AM     LOG [MetadataService] Initialized local reverse geocoder with cities500
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "4fc163b4-6fca-4726-8254-42c6722a96b4"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "9f1c480b-1b1e-40bc-844a-604412dcee02"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "68ae0bd1-249f-4172-a2cc-20e6f7c12e57"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "e4f4d8ad-eb26-4972-9d36-8be33f76604d"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "16fa4286-0e24-491e-bb79-97d25d1ff7b9"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "fff95cef-2df0-4920-b2e0-f8c4512e9065"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "df6f789e-422b-4b8e-b605-7a65e8eb07c2"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "d4a3bf64-54e4-4e13-a5a0-acef5a38e17d"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "0945cd10-5884-4fee-a46e-9156dbfa245e"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "4e645858-9bf0-47fc-b894-25c9045019bd"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "995a2c7e-f7b0-456d-bf3e-491e0a54455e"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "ce4a143a-a385-455e-87a7-de0a1f73fbc2"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "66a5b3b4-9da2-4b48-8411-c6957444e5e6"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "412b97df-50bb-4687-9f14-583c3807f2ad"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "1f5af953-223c-4046-8f59-e7f7a44e6f71"
}

[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:32 AM   ERROR [JobService] Object:
{
  "id": "fba524c4-3f47-43fc-8e9f-f66ba9f2a10e"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "93ea45b3-ed6a-4777-9caa-a45f46d14eba"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "5bde26a6-8098-49b8-828f-6742341aecb8"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "83ccd519-3d43-4aff-b800-f774de6989c6"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "555d88a4-9819-4f14-acb2-47a8428483df"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "fae0b256-1d13-4463-b5b3-a770e4ccc09d"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "41b630ba-b85e-4026-b59b-f3be82f525da"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "989fe15f-8d30-49f0-98a6-da7b0b535da3"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "2178c9fb-444d-40e1-8800-db14e92807f6"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "56d175c1-fe01-4f05-a0ec-e2ffdf29dccf"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "8bb8c1f3-5aaf-4101-ba67-2e5177be43ed"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "ed08f393-85b3-4525-b98a-b57ae1745c62"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "f4941c73-e6b9-419c-a5f0-ebadcaa97058"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "62460f21-f60f-4194-9f2b-fe1903677c29"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "c02f152f-8e1e-4149-93be-1696ccbe73c1"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "8d76603f-8546-4cab-97cd-f21f1468b64b"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "d73a2526-6678-4fe7-a21e-0d65384f2a5f"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "eb66ea24-eeec-4388-95b1-9337a7f407ca"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "eeea3f45-0de2-457a-a349-46843b62bc4a"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "90280801-88c5-44d6-8ab3-f825c3c8d419"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "ed8b421e-8dd0-40ea-b641-a683dcc688c2"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "2e2fda63-bdbe-4495-b364-ee1615c1d9e9"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "bf9f4586-c921-482f-a60c-b3417b6cac82"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "e8cae0b1-73d6-4ee7-8be9-5840e09c09a9"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "1cbd0d89-7d2f-49ae-a2ae-345cb66e5c13"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "f2b0f50d-2605-48a8-8b72-729b707f5dd3"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "038fc9dc-80a7-4454-9df5-eb2ed871e78f"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "6efe68d6-72cd-47bc-8425-4fa4626961cd"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "f34c4404-fd80-40b0-b008-6f06801892a0"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "82e5a8eb-7b49-4491-9c71-eeaf7e42e6dc"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "1428ead5-caf9-4fb5-a464-251a0d102f94"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Object:
{
  "id": "dd3865d9-212e-40ba-8ad5-7a1260299a1e"
}

[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect
[Nest] 7  - 11/02/2023, 3:00:33 AM   ERROR [JobService] Error: timeout exceeded when trying to connect
    at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27)
    at listOnTimeout (node:internal/timers:573:17)
    at process.processTimers (node:internal/timers:514:7)
@nodis commented on GitHub (Nov 2, 2023): > > May I ask how you recover to 1.83.0? > > In your .env file, change the following line: IMMICH_VERSION=release to IMMICH_VERSION=v1.83.0 Then repull image. About the cause of error: I think the file model for machine learning is incorrect. Have you tried deleting all files in /volume2/docker/immich/machine/model-cache? I tried deleting all files in /volume2/docker/immich/machine/model-cache, but the problem still exists log： docker logs immich_machine_learning： ```typescript [11/01/23 17:54:41] INFO Loading facial recognition model 'buffalo_l' [11/01/23 17:54:46] INFO Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus' [11/02/23 00:23:09] INFO Starting gunicorn 21.2.0 [11/02/23 00:23:09] INFO Listening at: http://0.0.0.0:3003 (9) [11/02/23 00:23:09] INFO Using worker: uvicorn.workers.UvicornWorker [11/02/23 00:23:09] INFO Booting worker with pid: 13 [11/02/23 00:25:09] CRITICAL WORKER TIMEOUT (pid:13) [11/02/23 00:25:10] ERROR Worker (pid:13) was sent SIGKILL! Perhaps out of memory? [11/02/23 00:25:10] INFO Booting worker with pid: 18 [11/02/23 00:27:59] INFO Starting gunicorn 21.2.0 [11/02/23 00:27:59] INFO Listening at: http://0.0.0.0:3003 (9) [11/02/23 00:27:59] INFO Using worker: uvicorn.workers.UvicornWorker [11/02/23 00:27:59] INFO Booting worker with pid: 13 [11/02/23 00:28:11] INFO Created in-memory cache with unloading disabled. [11/02/23 00:28:11] INFO Initialized request thread pool with 4 threads. [11/02/23 01:16:53] INFO Loading image classification model 'google/vit-base-patch16-224' /opt/venv/lib/python3.11/site-packages/transformers/models/vit/feature_extraction_vit.py:28: FutureWarning: The class ViTFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use ViTImageProcessor instead. warnings.warn( [11/02/23 01:17:12] INFO Loading facial recognition model 'buffalo_l' [11/02/23 01:17:24] INFO Loading clip model 'XLM-Roberta-Large-Vit-B-16Plus' [11/02/23 02:51:54] INFO Starting gunicorn 21.2.0 [11/02/23 02:51:54] INFO Listening at: http://0.0.0.0:3003 (9) [11/02/23 02:51:54] INFO Using worker: uvicorn.workers.UvicornWorker [11/02/23 02:51:54] INFO Booting worker with pid: 13 [11/02/23 02:53:55] CRITICAL WORKER TIMEOUT (pid:13) [11/02/23 02:53:57] ERROR Worker (pid:13) was sent SIGKILL! Perhaps out of memory? [11/02/23 02:53:57] INFO Booting worker with pid: 18 [11/02/23 02:55:49] INFO Created in-memory cache with unloading disabled. [11/02/23 02:55:49] INFO Initialized request thread pool with 4 threads. ``` docker logs immich_microservices： ```typescript <--- Last few GCs ---> [8:0x2d6b0310000] 208029 ms: Mark-Compact 4040.7 (4130.0) -> 4022.8 (4130.5) MB, 4719.06 / 0.00 ms (average mu = 0.133, current mu = 0.053) allocation failure; scavenge might not succeed [8:0x2d6b0310000] 212890 ms: Mark-Compact 4040.6 (4130.5) -> 4024.8 (4131.0) MB, 4703.15 / 0.00 ms (average mu = 0.083, current mu = 0.033) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0x109cd46 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_microservices] 12: 0x14f7b76 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_microservices] 13: 0x7fd57aa59ef6 [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [NestFactory] Starting Nest application... [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +30ms [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +1ms [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [InstanceLoader] ConfigModule dependencies initialized +4ms [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/02/2023, 3:00:09 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/02/2023, 3:00:11 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +2087ms [Nest] 7 - 11/02/2023, 3:00:11 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms [Nest] 7 - 11/02/2023, 3:00:11 AM LOG [InstanceLoader] InfraModule dependencies initialized +2ms [Nest] 7 - 11/02/2023, 3:00:11 AM LOG [InstanceLoader] DomainModule dependencies initialized +17ms [Nest] 7 - 11/02/2023, 3:00:11 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 7 - 11/02/2023, 3:00:23 AM LOG [MetadataService] Initialized local reverse geocoder with cities500 [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "4fc163b4-6fca-4726-8254-42c6722a96b4" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "9f1c480b-1b1e-40bc-844a-604412dcee02" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "68ae0bd1-249f-4172-a2cc-20e6f7c12e57" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "e4f4d8ad-eb26-4972-9d36-8be33f76604d" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "16fa4286-0e24-491e-bb79-97d25d1ff7b9" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "fff95cef-2df0-4920-b2e0-f8c4512e9065" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "df6f789e-422b-4b8e-b605-7a65e8eb07c2" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "d4a3bf64-54e4-4e13-a5a0-acef5a38e17d" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "0945cd10-5884-4fee-a46e-9156dbfa245e" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "4e645858-9bf0-47fc-b894-25c9045019bd" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "995a2c7e-f7b0-456d-bf3e-491e0a54455e" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "ce4a143a-a385-455e-87a7-de0a1f73fbc2" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "66a5b3b4-9da2-4b48-8411-c6957444e5e6" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "412b97df-50bb-4687-9f14-583c3807f2ad" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "1f5af953-223c-4046-8f59-e7f7a44e6f71" } [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:32 AM ERROR [JobService] Object: { "id": "fba524c4-3f47-43fc-8e9f-f66ba9f2a10e" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "93ea45b3-ed6a-4777-9caa-a45f46d14eba" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "5bde26a6-8098-49b8-828f-6742341aecb8" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "83ccd519-3d43-4aff-b800-f774de6989c6" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "555d88a4-9819-4f14-acb2-47a8428483df" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "fae0b256-1d13-4463-b5b3-a770e4ccc09d" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "41b630ba-b85e-4026-b59b-f3be82f525da" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "989fe15f-8d30-49f0-98a6-da7b0b535da3" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "2178c9fb-444d-40e1-8800-db14e92807f6" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "56d175c1-fe01-4f05-a0ec-e2ffdf29dccf" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "8bb8c1f3-5aaf-4101-ba67-2e5177be43ed" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "ed08f393-85b3-4525-b98a-b57ae1745c62" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "f4941c73-e6b9-419c-a5f0-ebadcaa97058" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "62460f21-f60f-4194-9f2b-fe1903677c29" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "c02f152f-8e1e-4149-93be-1696ccbe73c1" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "8d76603f-8546-4cab-97cd-f21f1468b64b" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "d73a2526-6678-4fe7-a21e-0d65384f2a5f" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "eb66ea24-eeec-4388-95b1-9337a7f407ca" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "eeea3f45-0de2-457a-a349-46843b62bc4a" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (clipEncoding/clip-encode): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "90280801-88c5-44d6-8ab3-f825c3c8d419" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "ed8b421e-8dd0-40ea-b641-a683dcc688c2" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "2e2fda63-bdbe-4495-b364-ee1615c1d9e9" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "bf9f4586-c921-482f-a60c-b3417b6cac82" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "e8cae0b1-73d6-4ee7-8be9-5840e09c09a9" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "1cbd0d89-7d2f-49ae-a2ae-345cb66e5c13" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "f2b0f50d-2605-48a8-8b72-729b707f5dd3" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "038fc9dc-80a7-4454-9df5-eb2ed871e78f" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "6efe68d6-72cd-47bc-8425-4fa4626961cd" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "f34c4404-fd80-40b0-b008-6f06801892a0" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "82e5a8eb-7b49-4491-9c71-eeaf7e42e6dc" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "1428ead5-caf9-4fb5-a464-251a0d102f94" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Object: { "id": "dd3865d9-212e-40ba-8ad5-7a1260299a1e" } [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: timeout exceeded when trying to connect [Nest] 7 - 11/02/2023, 3:00:33 AM ERROR [JobService] Error: timeout exceeded when trying to connect at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:205:27) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) ```
OVERLORD commented 2026-02-05 02:19:28 +03:00
Author
Owner
Copy Link

@dvbthien commented on GitHub (Nov 2, 2023):

I can't help. Developers are investigating the cause of this error. You can follow the discussion on Discord: Link. Your docker-compose.yml is different from mine:

immich_typesense:
  container_name: immich_typesense
  image: typesense/typesense:0.25.1

immich_redis:
    container_name: immich_redis
    image: redis:latest
@dvbthien commented on GitHub (Nov 2, 2023): I can't help. Developers are investigating the cause of this error. You can follow the discussion on Discord: [Link](https://discord.com/channels/979116623879368755/1169392004845809664). Your docker-compose.yml is different from mine: ``` immich_typesense: container_name: immich_typesense image: typesense/typesense:0.25.1 ``` ``` immich_redis: container_name: immich_redis image: redis:latest ```
OVERLORD commented 2026-02-05 02:19:31 +03:00
Author
Owner
Copy Link

@dvbthien commented on GitHub (Nov 2, 2023):

@nodis Removing all Typesense files in /volume2/docker/immich/tsdata and upgrading from Typesense 0.24.1 to 0.25.1
Change this line:
image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd to image: typesense/typesense:0.25.1. Then repull image and rebuilt typesense container. Good luck!

@dvbthien commented on GitHub (Nov 2, 2023): @nodis Removing all Typesense files in /volume2/docker/immich/tsdata and upgrading from Typesense 0.24.1 to 0.25.1 Change this line: `image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd` to `image: typesense/typesense:0.25.1`. Then repull image and rebuilt typesense container. Good luck!
OVERLORD commented 2026-02-05 02:19:33 +03:00
Author
Owner
Copy Link

@nodis commented on GitHub (Nov 2, 2023):

@nodis Removing all Typesense files in /volume2/docker/immich/tsdata and upgrading from Typesense 0.24.1 to 0.25.1 Change this line: image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd to image: typesense/typesense:0.25.1. Then repull image and rebuilt typesense container. Good luck!

@dvbthien Thank you very much. I'll give it a try

@nodis commented on GitHub (Nov 2, 2023): > @nodis Removing all Typesense files in /volume2/docker/immich/tsdata and upgrading from Typesense 0.24.1 to 0.25.1 Change this line: `image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd` to `image: typesense/typesense:0.25.1`. Then repull image and rebuilt typesense container. Good luck! @dvbthien Thank you very much. I'll give it a try
OVERLORD commented 2026-02-05 02:19:35 +03:00
Author
Owner
Copy Link

@hkaikai commented on GitHub (Nov 21, 2023):

@nodis Removing all Typesense files in /volume2/docker/immich/tsdata and upgrading from Typesense 0.24.1 to 0.25.1 Change this line: image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd to image: typesense/typesense:0.25.1. Then repull image and rebuilt typesense container. Good luck!

@dvbthien Thank you very much. I'll give it a try

@nodis Hello, have you tried it and have you solved it?

@hkaikai commented on GitHub (Nov 21, 2023): > > @nodis Removing all Typesense files in /volume2/docker/immich/tsdata and upgrading from Typesense 0.24.1 to 0.25.1 Change this line: `image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd` to `image: typesense/typesense:0.25.1`. Then repull image and rebuilt typesense container. Good luck! > > @dvbthien Thank you very much. I'll give it a try @nodis Hello, have you tried it and have you solved it?
OVERLORD commented 2026-02-05 02:19:38 +03:00
Author
Owner
Copy Link

@gustavoholst commented on GitHub (Nov 30, 2023):

Hi, I believe I am having the same issue, running 1.89.0 on nixos. There are some modifications that need to be made to convert the docker-compose file to a nix config file. I am happy to answer any questions about it if needed.

Key thing: microservices indicates TypeError: fetch failed when running any of the jobs. microservices, machine-learning, and typesense logs included below.

Environment
nixos
Version of Immich Server
1.89.0
Version of Immich Mobile App
n/a
Platform with the issue
✓ Server
Web
Mobile
Your docker-compose.yml content (replaced with .nix config file)

{

  virtualisation.oci-containers = let
    # The Immich version to use. You can pin this to a specific version like "v1.71.0"
    immichVersion = "v1.89.0";
    uploadPath = "/path/to/PhotoVideo/ImmichBackup";


    environment = {
      # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

      # The location where your uploaded files are stored
      UPLOAD_LOCATION = uploadPath;
  
      MACHINE_LEARNING_HOST = "10.88.4.3";

      TYPESENSE_API_KEY = "blah";
      TYPESENSE_ENABLED = "true";
      TYPESENSE_HOST = "10.88.4.4";
      TYPESENSE_DATA_DIR = "/data";
      #GLOG_minloglevel = "1";

      REDIS_HOSTNAME = "10.88.4.5";   
     
      DB_HOSTNAME = "10.88.4.6"; 
      DB_USERNAME = "postgres";
      POSTGRES_USER = "postgres";
      DB_DATABASE_NAME = "immich";
      POSTGRES_DB = "immich";
      DB_PASSWORD = "blah";
      POSTGRES_PASSWORD = "blah";
    };

  in {
    containers.immich_server = {
      image = "ghcr.io/immich-app/immich-server:${immichVersion}";
      autoStart = true;
      ports = [
        "2283:3001"
      ];
      volumes = [
        "${uploadPath}:/usr/src/app/upload"
        "/etc/localtime:/etc/localtime:ro"
        "/path/to/PhotoVideo:/mnt/media/personal:ro"
      ];
      environment = environment;
      dependsOn = [
        "immich_redis"
        "immich_postgres"
        "immich_typesense"
      ];
      cmd = [
        "start.sh"
        "immich"
      ];
      extraOptions = [
#        "--network=bridge"
        "--ip=10.88.4.1"
      ];
    };

    containers.immich_microservices = {
      image = "ghcr.io/immich-app/immich-server:${immichVersion}";
      autoStart = true;
      volumes = [
        "${uploadPath}:/usr/src/app/upload"
        "/etc/localtime:/etc/localtime:ro"
        "/path/to/PhotoVideo:/mnt/media/personal:ro"
      ];
      environment = environment;
      dependsOn = [
        "immich_redis"
        "immich_postgres"
        "immich_typesense"
      ];
      cmd = [
        "start.sh"
        "microservices"
      ];
      extraOptions = [
#        "--network=immich-bridge"
        "--ip=10.88.4.2"
      ];
    };

    containers.immich_machine-learning = {
      image = "ghcr.io/immich-app/immich-machine-learning:${immichVersion}";
      autoStart = true;
      volumes = [
        "/etc/oci-containers/immich-app/model-cache:/cache"
      ];
      environment = environment;
      extraOptions = [
#        "--network=immich-bridge"
        "--ip=10.88.4.3"
      ];
    };

    containers.immich_typesense = {
      image = "docker.io/typesense/typesense:0.25.1";
      autoStart = true;
      volumes = [
        "/etc/oci-containers/immich-app/tsdata:/data"
      ];
      environment = environment;
      extraOptions = [
#        "--network=immich-bridge"
        "--ip=10.88.4.4"
      ];
    };

    containers.immich_redis = {
      image = "docker.io/redis:6.2-alpine";
      autoStart = true;
      extraOptions = [
#        "--network=immich-bridge"
        "--ip=10.88.4.5"
      ];
    };

    containers.immich_postgres = {
      image = "docker.io/postgres:14-alpine";
      autoStart = true;
      volumes = [
        "/etc/oci-containers/immich-app/pgdata:/var/lib/postgresql/data"
      ];
      environment = environment;
      extraOptions = [
#        "--network=immich-bridge"
        "--ip=10.88.4.6"
      ];
    };
  };
}

Additional information
microservices log

[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [NestFactory] Starting Nest application...
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +19ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +3ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +145ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] InfraModule dependencies initialized +1ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] DomainModule dependencies initialized +9ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [MetadataRepository] Initializing metadata repository
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [MetadataService] Initialized local reverse geocoder
[Nest] 2  - 11/29/2023, 11:28:54 PM     LOG [SearchService] Running bootstrap
Request #1701300534822: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1701300534822: Sleeping for 4s and then retrying request...
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Creating schema: assets/assets-v10
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Creating schema: albums/albums-v2
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Creating schema: faces/faces-v1
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Alias mapping: []
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":true,"albums":true,"faces":true}
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [NestApplication] Nest application successfully started +18ms
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.89.0] [PRODUCTION]
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [SearchService] Indexing 0 albums
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Using new schema: (unset) => albums-v2
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [SearchService] Indexing 0 albums
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [SearchService] Indexing 0 faces
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Using new schema: (unset) => assets-v10
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Using new schema: albums-v2 => albums-v2
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Using new schema: (unset) => faces-v1
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [SearchService] Indexing 0 faces
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Using new schema: faces-v1 => faces-v1
[Nest] 2  - 11/29/2023, 11:28:58 PM     LOG [TypesenseRepository] Using new schema: assets-v10 => assets-v10
[Nest] 2  - 11/29/2023, 11:29:44 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 2  - 11/29/2023, 11:29:44 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:183:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 2  - 11/29/2023, 11:29:44 PM   ERROR [JobService] Object:
{
  "id": "2c53a4f6-2e4e-4489-8465-54a094fc7f8c"
}

[Nest] 2  - 11/29/2023, 11:29:44 PM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed
[Nest] 2  - 11/29/2023, 11:29:44 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:183:23)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 2  - 11/29/2023, 11:29:44 PM   ERROR [JobService] Object:
{
  "id": "ac26ddba-ff1b-4720-83a5-ba864380e7b4"
}

[Nest] 2  - 11/30/2023, 12:00:18 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 2  - 11/30/2023, 12:00:18 AM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 2  - 11/30/2023, 12:00:18 AM   ERROR [JobService] Object:
{
  "id": "2c53a4f6-2e4e-4489-8465-54a094fc7f8c"
}

[Nest] 2  - 11/30/2023, 12:00:18 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed
[Nest] 2  - 11/30/2023, 12:00:18 AM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21)
    at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22)
    at async /usr/src/app/dist/domain/job/job.service.js:108:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 2  - 11/30/2023, 12:00:18 AM   ERROR [JobService] Object:
{
  "id": "ac26ddba-ff1b-4720-83a5-ba864380e7b4"
}

machine-learning log

[11/29/23 23:18:02] INFO     Starting gunicorn 21.2.0
[11/29/23 23:18:02] INFO     Listening at: http://10.88.4.3:3003 (4)
[11/29/23 23:18:02] INFO     Using worker: uvicorn.workers.UvicornWorker
[11/29/23 23:18:02] INFO     Booting worker with pid: 16
[11/29/23 23:18:08] INFO     Created in-memory cache with unloading after 300s
                             of inactivity.
[11/29/23 23:18:08] INFO     Initialized request thread pool with 12 threads.
@gustavoholst commented on GitHub (Nov 30, 2023): Hi, I believe I am having the same issue, running 1.89.0 on nixos. There are some modifications that need to be made to convert the docker-compose file to a nix config file. I am happy to answer any questions about it if needed. Key thing: microservices indicates TypeError: fetch failed when running any of the jobs. microservices, machine-learning, and typesense logs included below. **Environment** nixos **Version of Immich Server** 1.89.0 **Version of Immich Mobile App** n/a **Platform with the issue** ✓ Server Web Mobile **Your docker-compose.yml content (replaced with .nix config file)** ``` { virtualisation.oci-containers = let # The Immich version to use. You can pin this to a specific version like "v1.71.0" immichVersion = "v1.89.0"; uploadPath = "/path/to/PhotoVideo/ImmichBackup"; environment = { # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION = uploadPath; MACHINE_LEARNING_HOST = "10.88.4.3"; TYPESENSE_API_KEY = "blah"; TYPESENSE_ENABLED = "true"; TYPESENSE_HOST = "10.88.4.4"; TYPESENSE_DATA_DIR = "/data"; #GLOG_minloglevel = "1"; REDIS_HOSTNAME = "10.88.4.5"; DB_HOSTNAME = "10.88.4.6"; DB_USERNAME = "postgres"; POSTGRES_USER = "postgres"; DB_DATABASE_NAME = "immich"; POSTGRES_DB = "immich"; DB_PASSWORD = "blah"; POSTGRES_PASSWORD = "blah"; }; in { containers.immich_server = { image = "ghcr.io/immich-app/immich-server:${immichVersion}"; autoStart = true; ports = [ "2283:3001" ]; volumes = [ "${uploadPath}:/usr/src/app/upload" "/etc/localtime:/etc/localtime:ro" "/path/to/PhotoVideo:/mnt/media/personal:ro" ]; environment = environment; dependsOn = [ "immich_redis" "immich_postgres" "immich_typesense" ]; cmd = [ "start.sh" "immich" ]; extraOptions = [ # "--network=bridge" "--ip=10.88.4.1" ]; }; containers.immich_microservices = { image = "ghcr.io/immich-app/immich-server:${immichVersion}"; autoStart = true; volumes = [ "${uploadPath}:/usr/src/app/upload" "/etc/localtime:/etc/localtime:ro" "/path/to/PhotoVideo:/mnt/media/personal:ro" ]; environment = environment; dependsOn = [ "immich_redis" "immich_postgres" "immich_typesense" ]; cmd = [ "start.sh" "microservices" ]; extraOptions = [ # "--network=immich-bridge" "--ip=10.88.4.2" ]; }; containers.immich_machine-learning = { image = "ghcr.io/immich-app/immich-machine-learning:${immichVersion}"; autoStart = true; volumes = [ "/etc/oci-containers/immich-app/model-cache:/cache" ]; environment = environment; extraOptions = [ # "--network=immich-bridge" "--ip=10.88.4.3" ]; }; containers.immich_typesense = { image = "docker.io/typesense/typesense:0.25.1"; autoStart = true; volumes = [ "/etc/oci-containers/immich-app/tsdata:/data" ]; environment = environment; extraOptions = [ # "--network=immich-bridge" "--ip=10.88.4.4" ]; }; containers.immich_redis = { image = "docker.io/redis:6.2-alpine"; autoStart = true; extraOptions = [ # "--network=immich-bridge" "--ip=10.88.4.5" ]; }; containers.immich_postgres = { image = "docker.io/postgres:14-alpine"; autoStart = true; volumes = [ "/etc/oci-containers/immich-app/pgdata:/var/lib/postgresql/data" ]; environment = environment; extraOptions = [ # "--network=immich-bridge" "--ip=10.88.4.6" ]; }; }; } ``` **Additional information** microservices log ``` [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [NestFactory] Starting Nest application... [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +19ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] ConfigModule dependencies initialized +3ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +145ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] InfraModule dependencies initialized +1ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] DomainModule dependencies initialized +9ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [MetadataRepository] Initializing metadata repository [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [MetadataService] Initialized local reverse geocoder [Nest] 2 - 11/29/2023, 11:28:54 PM LOG [SearchService] Running bootstrap Request #1701300534822: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging" Request #1701300534822: Sleeping for 4s and then retrying request... [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Creating schema: assets/assets-v10 [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Creating schema: albums/albums-v2 [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Creating schema: faces/faces-v1 [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Alias mapping: [] [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Collections needing migration: {"assets":true,"albums":true,"faces":true} [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [NestApplication] Nest application successfully started +18ms [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.89.0] [PRODUCTION] [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [SearchService] Indexing 0 albums [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Using new schema: (unset) => albums-v2 [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [SearchService] Indexing 0 albums [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [SearchService] Indexing 0 faces [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Using new schema: (unset) => assets-v10 [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Using new schema: albums-v2 => albums-v2 [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Using new schema: (unset) => faces-v1 [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [SearchService] Indexing 0 faces [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Using new schema: faces-v1 => faces-v1 [Nest] 2 - 11/29/2023, 11:28:58 PM LOG [TypesenseRepository] Using new schema: assets-v10 => assets-v10 [Nest] 2 - 11/29/2023, 11:29:44 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 2 - 11/29/2023, 11:29:44 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:183:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24) [Nest] 2 - 11/29/2023, 11:29:44 PM ERROR [JobService] Object: { "id": "2c53a4f6-2e4e-4489-8465-54a094fc7f8c" } [Nest] 2 - 11/29/2023, 11:29:44 PM ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): TypeError: fetch failed [Nest] 2 - 11/29/2023, 11:29:44 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:183:23) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24) [Nest] 2 - 11/29/2023, 11:29:44 PM ERROR [JobService] Object: { "id": "ac26ddba-ff1b-4720-83a5-ba864380e7b4" } [Nest] 2 - 11/30/2023, 12:00:18 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 2 - 11/30/2023, 12:00:18 AM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24) [Nest] 2 - 11/30/2023, 12:00:18 AM ERROR [JobService] Object: { "id": "2c53a4f6-2e4e-4489-8465-54a094fc7f8c" } [Nest] 2 - 11/30/2023, 12:00:18 AM ERROR [JobService] Unable to run job handler (objectTagging/classify-image): TypeError: fetch failed [Nest] 2 - 11/30/2023, 12:00:18 AM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11372:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:16:21) at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22) at async /usr/src/app/dist/domain/job/job.service.js:108:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24) [Nest] 2 - 11/30/2023, 12:00:18 AM ERROR [JobService] Object: { "id": "ac26ddba-ff1b-4720-83a5-ba864380e7b4" } ``` machine-learning log ``` [11/29/23 23:18:02] INFO Starting gunicorn 21.2.0 [11/29/23 23:18:02] INFO Listening at: http://10.88.4.3:3003 (4) [11/29/23 23:18:02] INFO Using worker: uvicorn.workers.UvicornWorker [11/29/23 23:18:02] INFO Booting worker with pid: 16 [11/29/23 23:18:08] INFO Created in-memory cache with unloading after 300s of inactivity. [11/29/23 23:18:08] INFO Initialized request thread pool with 12 threads. ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1545
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?