immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

OpenVINO ML pod crashes on jobs triggered by uploads but works fine on manually triggered jobs #6158

New Issue
Open
opened 2026-02-05 12:03:49 +03:00 by OVERLORD · 13 comments
OVERLORD commented 2026-02-05 12:03:49 +03:00
Owner
Copy Link

Originally created by @againstpetra on GitHub (May 29, 2025).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • Yes

The bug

when I upload an image to immich and the server automatically dispatches a job to do face detection, the openvino ML pod crashes and the server is unable to complete the job. but if I go to the admin panel and then manually trigger a face detention job it works flawlessly

I am using OpenVINO with my CPU's integrated graphics and I know that can lead to more issues than an external GPU. but this issue feels more strange: the functionality does work, just not when triggered automatically on upload by the server itself

The OS that Immich Server is running on

Arch / Kubernetes

Version of Immich Server

v1.134.0

Version of Immich Mobile App

v1.134.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

# this isn't a docker-compose.yml file because I don't use docker-compose. I use helm. these are the helm values.
env:
  DB_HOSTNAME: cnpg-rw.immich.svc.cluster.local
  DB_USERNAME: postgres
  DB_PASSWORD:
    valueFrom:
      secretKeyRef:
        name: cnpg-auth
        key: password
  REDIS_PASSWORD:
    valueFrom:
      secretKeyRef:
        name: redis-auth
        key: redis-password
image:
  tag: v1.134.0
immich:
  persistence:
    library:
      existingClaim: media
redis:
  enabled: true
  architecture: standalone
  auth:
    enabled: true
    existingSecret: redis-auth
    existingSecretPasswordKey: redis-password
server:
  enabled: true
  resources:
    limits:
      gpu.intel.com/i915: 1
    ingress:
      <redacted>
machine-learning:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-machine-learning
    tag: v1.134.0-openvino
  resources:
    limits:
      gpu.intel.com/i915: 1
  persistence:
    cache:
      enabled: true
      type: pvc
      existingClaim: cache

Your .env content

n/a, see above.

Reproduction steps

  1. upload an image
  2. the pod crashes / restarts and the job fails
  3. go to admin panel and trigger the job manually, it works

Relevant log output

the ML pods logs until it crashes (no intersting log):

[05/28/25 07:42:58] INFO     Starting gunicorn 23.0.0                           
[05/28/25 07:42:58] INFO     Listening at: http://[::]:3003 (8)                 
[05/28/25 07:42:58] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
[05/28/25 07:42:58] INFO     Booting worker with pid: 9                         
[05/28/25 07:43:00] INFO     Started server process [9]                         
[05/28/25 07:43:00] INFO     Waiting for application startup.                   
[05/28/25 07:43:00] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[05/28/25 07:43:00] INFO     Initialized request thread pool with 20 threads.   
[05/28/25 07:43:00] INFO     Application startup complete.                      
[05/28/25 10:04:01] INFO     Loading detection model 'buffalo_l' to memory      
[05/28/25 10:04:01] INFO     Setting execution providers to                     
                             ['OpenVINOExecutionProvider',                      
                             'CPUExecutionProvider'], in descending order of    
                             preference                                         
[05/28/25 10:04:01] INFO     Loading visual model 'ViT-B-32__openai' to memory  
[05/28/25 10:04:01] INFO     Setting execution providers to                     
                             ['OpenVINOExecutionProvider',                      
                             'CPUExecutionProvider'], in descending order of    
                             preference    

the logs of it restarting:

[05/28/25 10:04:30] INFO     Starting gunicorn 23.0.0                           
[05/28/25 10:04:30] INFO     Listening at: http://[::]:3003 (8)                 
[05/28/25 10:04:30] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
[05/28/25 10:04:30] INFO     Booting worker with pid: 9                         
[05/28/25 10:04:31] INFO     Started server process [9]                         
[05/28/25 10:04:31] INFO     Waiting for application startup.                   
[05/28/25 10:04:31] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[05/28/25 10:04:31] INFO     Initialized request thread pool with 20 threads.   
[05/28/25 10:04:31] INFO     Application startup complete. 

and then from the server:

[Nest] 7  - 05/28/2025, 10:04:24 AM   ERROR [Microservices:{"source":"upload","id":"7e6aeae2-7c34-4ca7-b783-f43491dbe0d6"}] Unable to run job handler (face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15)
    at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:107:26)
    at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:232:52)
    at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:166:28)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13)
    at async /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:526:32
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:751:24)
[Nest] 7  - 05/28/2025, 10:04:24 AM   ERROR [Microservices:{"source":"upload","id":"7e6aeae2-7c34-4ca7-b783-f43491dbe0d6"}] Unable to run job handler (smart-search): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15)
    at async MachineLearningRepository.encodeImage (/usr/src/app/dist/repositories/machine-learning.repository.js:116:26)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:91:27)
    at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:166:28)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13)
    at async /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:526:32
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:751:24)
[Nest] 7  - 05/28/2025, 10:04:24 AM   ERROR [Microservices:{"source":"upload","id":"87420a98-ed41-4afa-892e-6a16d96b43cc"}] Unable to run job handler (smart-search): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15)
    at async MachineLearningRepository.encodeImage (/usr/src/app/dist/repositories/machine-learning.repository.js:116:26)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:91:27)
    at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:166:28)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13)
    at async /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:526:32
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:751:24)
[Nest] 7  - 05/28/2025, 10:04:24 AM   ERROR [Microservices:{"source":"upload","id":"87420a98-ed41-4afa-892e-6a16d96b43cc"}] Unable to run job handler (face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs
    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15)
    at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:107:26)
    at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:232:52)
    at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:166:28)
    at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13)
    at async /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:526:32
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:751:24)

Additional information

No response

Originally created by @againstpetra on GitHub (May 29, 2025). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug when I upload an image to immich and the server automatically dispatches a job to do face detection, the openvino ML pod crashes and the server is unable to complete the job. but if I go to the admin panel and then manually trigger a face detention job it works flawlessly I am using OpenVINO with my CPU's integrated graphics and I know that can lead to more issues than an external GPU. but this issue feels more strange: the functionality does work, just not when triggered automatically on upload by the server itself ### The OS that Immich Server is running on Arch / Kubernetes ### Version of Immich Server v1.134.0 ### Version of Immich Mobile App v1.134.0 ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML # this isn't a docker-compose.yml file because I don't use docker-compose. I use helm. these are the helm values. env: DB_HOSTNAME: cnpg-rw.immich.svc.cluster.local DB_USERNAME: postgres DB_PASSWORD: valueFrom: secretKeyRef: name: cnpg-auth key: password REDIS_PASSWORD: valueFrom: secretKeyRef: name: redis-auth key: redis-password image: tag: v1.134.0 immich: persistence: library: existingClaim: media redis: enabled: true architecture: standalone auth: enabled: true existingSecret: redis-auth existingSecretPasswordKey: redis-password server: enabled: true resources: limits: gpu.intel.com/i915: 1 ingress: <redacted> machine-learning: enabled: true image: repository: ghcr.io/immich-app/immich-machine-learning tag: v1.134.0-openvino resources: limits: gpu.intel.com/i915: 1 persistence: cache: enabled: true type: pvc existingClaim: cache ``` ### Your .env content ```Shell n/a, see above. ``` ### Reproduction steps 1. upload an image 2. the pod crashes / restarts and the job fails 3. go to admin panel and trigger the job manually, it works ### Relevant log output ```shell the ML pods logs until it crashes (no intersting log): [05/28/25 07:42:58] INFO Starting gunicorn 23.0.0 [05/28/25 07:42:58] INFO Listening at: http://[::]:3003 (8) [05/28/25 07:42:58] INFO Using worker: immich_ml.config.CustomUvicornWorker [05/28/25 07:42:58] INFO Booting worker with pid: 9 [05/28/25 07:43:00] INFO Started server process [9] [05/28/25 07:43:00] INFO Waiting for application startup. [05/28/25 07:43:00] INFO Created in-memory cache with unloading after 300s of inactivity. [05/28/25 07:43:00] INFO Initialized request thread pool with 20 threads. [05/28/25 07:43:00] INFO Application startup complete. [05/28/25 10:04:01] INFO Loading detection model 'buffalo_l' to memory [05/28/25 10:04:01] INFO Setting execution providers to ['OpenVINOExecutionProvider', 'CPUExecutionProvider'], in descending order of preference [05/28/25 10:04:01] INFO Loading visual model 'ViT-B-32__openai' to memory [05/28/25 10:04:01] INFO Setting execution providers to ['OpenVINOExecutionProvider', 'CPUExecutionProvider'], in descending order of preference the logs of it restarting: [05/28/25 10:04:30] INFO Starting gunicorn 23.0.0 [05/28/25 10:04:30] INFO Listening at: http://[::]:3003 (8) [05/28/25 10:04:30] INFO Using worker: immich_ml.config.CustomUvicornWorker [05/28/25 10:04:30] INFO Booting worker with pid: 9 [05/28/25 10:04:31] INFO Started server process [9] [05/28/25 10:04:31] INFO Waiting for application startup. [05/28/25 10:04:31] INFO Created in-memory cache with unloading after 300s of inactivity. [05/28/25 10:04:31] INFO Initialized request thread pool with 20 threads. [05/28/25 10:04:31] INFO Application startup complete. and then from the server: [Nest] 7 - 05/28/2025, 10:04:24 AM ERROR [Microservices:{"source":"upload","id":"7e6aeae2-7c34-4ca7-b783-f43491dbe0d6"}] Unable to run job handler (face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15) at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:107:26) at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:232:52) at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:166:28) at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13) at async /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:526:32 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:751:24) [Nest] 7 - 05/28/2025, 10:04:24 AM ERROR [Microservices:{"source":"upload","id":"7e6aeae2-7c34-4ca7-b783-f43491dbe0d6"}] Unable to run job handler (smart-search): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15) at async MachineLearningRepository.encodeImage (/usr/src/app/dist/repositories/machine-learning.repository.js:116:26) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:91:27) at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:166:28) at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13) at async /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:526:32 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:751:24) [Nest] 7 - 05/28/2025, 10:04:24 AM ERROR [Microservices:{"source":"upload","id":"87420a98-ed41-4afa-892e-6a16d96b43cc"}] Unable to run job handler (smart-search): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15) at async MachineLearningRepository.encodeImage (/usr/src/app/dist/repositories/machine-learning.repository.js:116:26) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/services/smart-info.service.js:91:27) at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:166:28) at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13) at async /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:526:32 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:751:24) [Nest] 7 - 05/28/2025, 10:04:24 AM ERROR [Microservices:{"source":"upload","id":"87420a98-ed41-4afa-892e-6a16d96b43cc"}] Unable to run job handler (face-detection): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_l","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_l"}}}' failed for all URLs at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15) at async MachineLearningRepository.detectFaces (/usr/src/app/dist/repositories/machine-learning.repository.js:107:26) at async PersonService.handleDetectFaces (/usr/src/app/dist/services/person.service.js:232:52) at async JobService.onJobStart (/usr/src/app/dist/services/job.service.js:166:28) at async EventRepository.onEvent (/usr/src/app/dist/repositories/event.repository.js:126:13) at async /usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:526:32 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:751:24) ``` ### Additional information _No response_
OVERLORD added the 🗄️server label 2026-02-05 12:03:49 +03:00
OVERLORD commented 2026-02-05 12:03:51 +03:00
Author
Owner
Copy Link

@xarblu commented on GitHub (Jun 2, 2025):

I am using OpenVINO with my CPU's integrated graphics and I know that can lead to more issues than an external GPU

I hit the same issue with a dedicated GPU (Arc A380) so shouldn't be related to integrated graphics.

I'm running Immich via Podman Quadlets but I'm 99% sure that shouldn't be the issue (at least it never was an issue before)

@xarblu commented on GitHub (Jun 2, 2025): > I am using OpenVINO with my CPU's integrated graphics and I know that can lead to more issues than an external GPU I hit the same issue with a dedicated GPU (Arc A380) so shouldn't be related to integrated graphics. I'm running Immich via Podman Quadlets but I'm 99% sure that shouldn't be the issue (at least it never was an issue before)
OVERLORD commented 2026-02-05 12:03:53 +03:00
Author
Owner
Copy Link

@redaws00 commented on GitHub (Jun 7, 2025):

Same issue here. Running on Docker v4.42.0 on Windows with dedicated RTX 3090.

@redaws00 commented on GitHub (Jun 7, 2025): Same issue here. Running on Docker v4.42.0 on Windows with dedicated RTX 3090.
OVERLORD commented 2026-02-05 12:03:55 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Jun 7, 2025):

Are you sure this isn't a case of the container reaching resource limits / running out of memory?

Also @redaws00, it's unlikely that an issue with your GPU has the same root cause as the OP using OpenVINO.

@mertalev commented on GitHub (Jun 7, 2025): Are you sure this isn't a case of the container reaching resource limits / running out of memory? Also @redaws00, it's unlikely that an issue with your GPU has the same root cause as the OP using OpenVINO.
OVERLORD commented 2026-02-05 12:03:56 +03:00
Author
Owner
Copy Link

@redaws00 commented on GitHub (Jun 7, 2025):

@mertalev I'm new to this application...and Docker on Windows in general. Is there a good way to get logs to look at? My current work around is to have machine learning in a separate container that I disable and enable when needed. When functioning together, both the web and ML crash.

@redaws00 commented on GitHub (Jun 7, 2025): @mertalev I'm new to this application...and Docker on Windows in general. Is there a good way to get logs to look at? My current work around is to have machine learning in a separate container that I disable and enable when needed. When functioning together, both the web and ML crash.
OVERLORD commented 2026-02-05 12:03:58 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Jun 7, 2025):

You can post the machine learning logs with docker logs immich_machine_learning.

In general, Docker on Windows can be convenient, but it's also much less stable than Docker on Linux. Issues can often be fixed by upgrading to the latest Docker Desktop version and latest NVIDIA driver, or even just restarting the PC (a classic).

@mertalev commented on GitHub (Jun 7, 2025): You can post the machine learning logs with `docker logs immich_machine_learning`. In general, Docker on Windows can be convenient, but it's also much less stable than Docker on Linux. Issues can often be fixed by upgrading to the latest Docker Desktop version and latest NVIDIA driver, or even just restarting the PC (a classic).
OVERLORD commented 2026-02-05 12:03:59 +03:00
Author
Owner
Copy Link

@redaws00 commented on GitHub (Jun 7, 2025):

I tried that already. Unfortunately, my 3090 machine is Windows and i have to keep it that way (being a .Net Developer). Docker logs are no good because once the container locks up, all I get is the following:
request returned 500 Internal Server Error for API route and version http://%2F%2F.%2Fpipe%2FdockerDesktopLinuxEngine/v1.49/containers/7fa6423c94830c0969a92a92bebf2dbe2e3647261523a99c596350e979d20327/json, check if the server supports the requested API version

Are there any logs stored in the container?

@redaws00 commented on GitHub (Jun 7, 2025): I tried that already. Unfortunately, my 3090 machine is Windows and i have to keep it that way (being a .Net Developer). Docker logs are no good because once the container locks up, all I get is the following: request returned 500 Internal Server Error for API route and version http://%2F%2F.%2Fpipe%2FdockerDesktopLinuxEngine/v1.49/containers/7fa6423c94830c0969a92a92bebf2dbe2e3647261523a99c596350e979d20327/json, check if the server supports the requested API version Are there any logs stored in the container?
OVERLORD commented 2026-02-05 12:04:00 +03:00
Author
Owner
Copy Link

@againstpetra commented on GitHub (Jun 7, 2025):

Are you sure this isn't a case of the container reaching resource limits / running out of memory?

I just tried uploading something and had the memory usage side-by-side, it doesn't really spike and there's nothing in the kubectl events about it getting OOM'd or something - it just immediately starts failing probes

0s          Warning   Unhealthy          pod/immich-machine-learning-f6c9d559b-p55tx   Liveness probe failed: Get "http://10.0.0.100:3003/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s          Warning   Unhealthy          pod/immich-machine-learning-f6c9d559b-p55tx   Readiness probe failed: Get "http://10.0.0.100:3003/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

and eventually gets killed. I also don't get why it would reach resource limits given it's fine when triggered manually.

@againstpetra commented on GitHub (Jun 7, 2025): > Are you sure this isn't a case of the container reaching resource limits / running out of memory? I just tried uploading something and had the memory usage side-by-side, it doesn't really spike and there's nothing in the kubectl events about it getting OOM'd or something - it just immediately starts failing probes ``` 0s Warning Unhealthy pod/immich-machine-learning-f6c9d559b-p55tx Liveness probe failed: Get "http://10.0.0.100:3003/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 0s Warning Unhealthy pod/immich-machine-learning-f6c9d559b-p55tx Readiness probe failed: Get "http://10.0.0.100:3003/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ``` and eventually gets killed. I also don't get why it would reach resource limits given it's fine when triggered manually.
OVERLORD commented 2026-02-05 12:04:01 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Jun 7, 2025):

That's pretty weird. There should be no difference between what the job does on manual queue vs on upload, and no difference in what it sends to the machine learning service. I would have to guess it's a timing issue of some kind, but I'm not sure where that would be either.

@mertalev commented on GitHub (Jun 7, 2025): That's pretty weird. There should be no difference between what the job does on manual queue vs on upload, and no difference in what it sends to the machine learning service. I would have to guess it's a timing issue of some kind, but I'm not sure where that would be either.
OVERLORD commented 2026-02-05 12:04:02 +03:00
Author
Owner
Copy Link

@croneter commented on GitHub (Oct 8, 2025):

Are you sure this isn't a case of the container reaching resource limits / running out of memory?

I just tried uploading something and had the memory usage side-by-side, it doesn't really spike and there's nothing in the kubectl events about it getting OOM'd or something - it just immediately starts failing probes

0s          Warning   Unhealthy          pod/immich-machine-learning-f6c9d559b-p55tx   Liveness probe failed: Get "http://10.0.0.100:3003/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
0s          Warning   Unhealthy          pod/immich-machine-learning-f6c9d559b-p55tx   Readiness probe failed: Get "http://10.0.0.100:3003/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

and eventually gets killed. I also don't get why it would reach resource limits given it's fine when triggered manually.

I'm seeing exactly the same: the /ping endpoint stops functioning as soon as I trigger e.g. a smart search. Kubernetes then kills the pod after a while.

@croneter commented on GitHub (Oct 8, 2025): > > Are you sure this isn't a case of the container reaching resource limits / running out of memory? > > I just tried uploading something and had the memory usage side-by-side, it doesn't really spike and there's nothing in the kubectl events about it getting OOM'd or something - it just immediately starts failing probes > > ``` > 0s Warning Unhealthy pod/immich-machine-learning-f6c9d559b-p55tx Liveness probe failed: Get "http://10.0.0.100:3003/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) > 0s Warning Unhealthy pod/immich-machine-learning-f6c9d559b-p55tx Readiness probe failed: Get "http://10.0.0.100:3003/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers) > ``` > > and eventually gets killed. I also don't get why it would reach resource limits given it's fine when triggered manually. I'm seeing exactly the same: the `/ping` endpoint stops functioning as soon as I trigger e.g. a smart search. Kubernetes then kills the pod after a while.
OVERLORD commented 2026-02-05 12:04:03 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Oct 8, 2025):

The likely reason is that the model has to be compiled to OpenVINO format the first time it's loaded, which (unfortunately) blocks the ML server from responding to any requests until it's done.

@mertalev commented on GitHub (Oct 8, 2025): The likely reason is that the model has to be compiled to OpenVINO format the first time it's loaded, which (unfortunately) blocks the ML server from responding to any requests until it's done.
OVERLORD commented 2026-02-05 12:04:04 +03:00
Author
Owner
Copy Link

@redaws00 commented on GitHub (Oct 8, 2025):

About how long should it take to compile? I realize that times differ system to system.

@redaws00 commented on GitHub (Oct 8, 2025): About how long should it take to compile? I realize that times differ system to system.
OVERLORD commented 2026-02-05 12:04:04 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Oct 8, 2025):

Typically a few minutes, at most maybe 15 in a very constrained environment with a large model.

@mertalev commented on GitHub (Oct 8, 2025): Typically a few minutes, at most maybe 15 in a very constrained environment with a large model.
OVERLORD commented 2026-02-05 12:04:05 +03:00
Author
Owner
Copy Link

@redaws00 commented on GitHub (Oct 8, 2025):

I don't think that's the problem I'm having. My current solution is to separate the ML from the main app and only run the ML once a week while manually triggering the jobs. It still occasionally locks up all of Docker (on Windows), but I can get the jobs completed.

@redaws00 commented on GitHub (Oct 8, 2025): I don't think that's the problem I'm having. My current solution is to separate the ML from the main app and only run the ML once a week while manually triggering the jobs. It still occasionally locks up all of Docker (on Windows), but I can get the jobs completed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label 🗄️server
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#6158
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?