immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-16 05:34:30 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Uploads fail with ML errors while machine-learning pod downloads models on startup #8282

New Issue
Closed
opened 2026-02-05 13:38:21 +03:00 by OVERLORD · 2 comments
OVERLORD commented 2026-02-05 13:38:21 +03:00
Owner
Copy Link

Originally created by @timholm on GitHub (Jan 20, 2026).

Description

In Kubernetes deployments, when the machine-learning pod restarts and downloads models on startup, any photo/video uploads during this time fail with ML-related errors. The uploads appear to succeed initially but ML processing (face detection, smart search, OCR) fails.

Environment

  • Kubernetes (k3s)
  • Immich v2.4.1
  • Machine-learning using default emptyDir cache (models re-download on every pod restart)

Error Logs

Server logs during upload while ML is downloading models:

[Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
[Microservices] Unable to run job handler (SmartSearch): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
[Microservices] Unable to run job handler (AssetDetectFaces): Error: Machine learning request '{"facial-recognition"...}' failed for all URLs
[Microservices] Unable to run job handler (Ocr): Error: Machine learning request '{"ocr"...}' failed for all URLs

ML pod logs showing model downloads taking several minutes:

Downloading visual model 'ViT-B-32__openai' to /cache/clip/ViT-B-32__openai/visual/model.onnx. This may take a while.
Fetching 11 files: 100%|██████████| 11/11 [06:17<00:00, 34.33s/it]

Root Cause

The ML service reports as healthy/ready before models are fully downloaded and loaded. When uploads trigger ML processing, the service can't respond because it's busy downloading.

Impact

  • Users uploading during ML pod startup see failures
  • In Kubernetes, pod restarts are common (updates, node issues, scaling)
  • With emptyDir cache (default), every restart triggers ~6 minute model download window

Suggestions

  1. Readiness probe: ML pod should only report ready after models are loaded, not just when gunicorn starts
  2. Graceful degradation: Server could queue ML jobs when ML service unavailable instead of failing
  3. Helm chart default: Consider persistent cache by default for Kubernetes deployments
  4. Pre-download option: Init container or startup hook to download models before marking ready

Workaround

Using persistent storage for /cache prevents re-downloads, but iSCSI storage can timeout on slower hardware (Raspberry Pi) during first format. NFS works better for this use case.

Originally created by @timholm on GitHub (Jan 20, 2026). ## Description In Kubernetes deployments, when the machine-learning pod restarts and downloads models on startup, any photo/video uploads during this time fail with ML-related errors. The uploads appear to succeed initially but ML processing (face detection, smart search, OCR) fails. ## Environment - Kubernetes (k3s) - Immich v2.4.1 - Machine-learning using default `emptyDir` cache (models re-download on every pod restart) ## Error Logs Server logs during upload while ML is downloading models: ``` [Microservices:MachineLearningRepository] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed [Microservices] Unable to run job handler (SmartSearch): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs [Microservices] Unable to run job handler (AssetDetectFaces): Error: Machine learning request '{"facial-recognition"...}' failed for all URLs [Microservices] Unable to run job handler (Ocr): Error: Machine learning request '{"ocr"...}' failed for all URLs ``` ML pod logs showing model downloads taking several minutes: ``` Downloading visual model 'ViT-B-32__openai' to /cache/clip/ViT-B-32__openai/visual/model.onnx. This may take a while. Fetching 11 files: 100%|██████████| 11/11 [06:17<00:00, 34.33s/it] ``` ## Root Cause The ML service reports as healthy/ready before models are fully downloaded and loaded. When uploads trigger ML processing, the service can't respond because it's busy downloading. ## Impact - Users uploading during ML pod startup see failures - In Kubernetes, pod restarts are common (updates, node issues, scaling) - With `emptyDir` cache (default), every restart triggers ~6 minute model download window ## Suggestions 1. **Readiness probe**: ML pod should only report ready after models are loaded, not just when gunicorn starts 2. **Graceful degradation**: Server could queue ML jobs when ML service unavailable instead of failing 3. **Helm chart default**: Consider persistent cache by default for Kubernetes deployments 4. **Pre-download option**: Init container or startup hook to download models before marking ready ## Workaround Using persistent storage for `/cache` prevents re-downloads, but iSCSI storage can timeout on slower hardware (Raspberry Pi) during first format. NFS works better for this use case.
OVERLORD commented 2026-02-05 13:38:23 +03:00
Author
Owner
Copy Link

@timholm commented on GitHub (Jan 20, 2026):

Clarification: Kubernetes-specific issue

This issue is specific to Kubernetes deployments and does not affect Docker users.

Why Docker doesn't have this problem:

  • Docker containers persist across restarts
  • The /cache volume (even as a local volume) survives container restarts
  • Models are downloaded once and cached indefinitely

Why Kubernetes has this problem:

  • Pods are ephemeral - they get recreated frequently
  • ArgoCD (GitOps) recreates pods on every sync/update
  • Default emptyDir cache is lost when pod is deleted
  • Each new pod must re-download all ML models (~700MB-1GB)
  • During the 5-6 minute download window, uploads fail

Deployment details:

  • Using ArgoCD for GitOps deployment
  • Helm chart: immich-app/immich-charts
  • Auto-sync enabled = pods recreated on any config change
  • Image updater enabled = pods recreated on new immich releases

The core issue is that the ML readiness probe passes before models are loaded, so Kubernetes considers the pod "ready" while it's still downloading. ArgoCD then routes traffic to it, causing upload failures.

@timholm commented on GitHub (Jan 20, 2026): ## Clarification: Kubernetes-specific issue This issue is **specific to Kubernetes deployments** and does not affect Docker users. ### Why Docker doesn't have this problem: - Docker containers persist across restarts - The `/cache` volume (even as a local volume) survives container restarts - Models are downloaded once and cached indefinitely ### Why Kubernetes has this problem: - Pods are ephemeral - they get recreated frequently - **ArgoCD** (GitOps) recreates pods on every sync/update - Default `emptyDir` cache is lost when pod is deleted - Each new pod must re-download all ML models (~700MB-1GB) - During the 5-6 minute download window, uploads fail ### Deployment details: - Using **ArgoCD** for GitOps deployment - Helm chart: `immich-app/immich-charts` - Auto-sync enabled = pods recreated on any config change - Image updater enabled = pods recreated on new immich releases The core issue is that the ML readiness probe passes before models are loaded, so Kubernetes considers the pod "ready" while it's still downloading. ArgoCD then routes traffic to it, causing upload failures.
OVERLORD commented 2026-02-05 13:38:24 +03:00
Author
Owner
Copy Link

@github-actions[bot] commented on GitHub (Jan 20, 2026):

This issue has automatically been closed as it is likely a duplicate. We get a lot of duplicate threads each day, which is why we ask you in the template to confirm that you searched for duplicates before opening one. If you're sure this is not a duplicate, please leave a comment and we will reopen the thread if necessary.

@github-actions[bot] commented on GitHub (Jan 20, 2026): This issue has automatically been closed as it is likely a duplicate. We get a lot of duplicate threads each day, which is why we ask you in the template to confirm that you searched for duplicates before opening one. If you're sure this is not a duplicate, please leave a comment and we will reopen the thread if necessary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#8282
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?