immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-16 05:34:30 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

[PR #3207] fix(ml): race condition when loading models #9890

New Issue
Closed
opened 2026-02-05 14:12:29 +03:00 by OVERLORD · 0 comments
OVERLORD commented 2026-02-05 14:12:29 +03:00
Owner
Copy Link

Original Pull Request: https://github.com/immich-app/immich/pull/3207

State: closed
Merged: Yes

Description

This change reverts to loading models synchronously, preventing multiple calls from loading the same model several times.

An earlier change made model loading happen in a background thread so other requests could continue to be handled. However, this had the effect of allowing multiple calls to load the same model repeatedly with concurrent requests. While the default behavior is to load all models at startup, this race condition allowed memory usage to spike dramatically after the models were unloaded.

Additionally defaults to models never being unloaded. The current implementation for unloading isn't robust enough and can result in greater memory usage than simply keeping models in memory.

Fixes #3142

How Has This Been Tested?

Set the job concurrency to a high number (such as 8) for either image tagging or facial recognition (the CLIP model has no logs) and start the job. The logs should only show one instance of the corresponding log below. I tested this with all ML jobs running simultaneously and saw it showed correct behavior (on main, this swelled memory usage to over 10gb).

Image classification log:
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.

Facial recognition log:

Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/facial-recognition/buffalo_l/models/buffalo_l/1k3d68.onnx landmark_3d_68
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/facial-recognition/buffalo_l/models/buffalo_l/2d106det.onnx landmark_2d_106
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /cache/facial-recognition/buffalo_l/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
model ignore: /cache/facial-recognition/buffalo_l/models/buffalo_l/genderage.onnx genderage
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /cache/facial-recognition/buffalo_l/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)
**Original Pull Request:** https://github.com/immich-app/immich/pull/3207 **State:** closed **Merged:** Yes --- ## Description This change reverts to loading models synchronously, preventing multiple calls from loading the same model several times. An earlier change made model loading happen in a background thread so other requests could continue to be handled. However, this had the effect of allowing multiple calls to load the same model repeatedly with concurrent requests. While the default behavior is to load all models at startup, this race condition allowed memory usage to spike dramatically after the models were unloaded. Additionally defaults to models never being unloaded. The current implementation for unloading isn't robust enough and can result in greater memory usage than simply keeping models in memory. Fixes #3142 ## How Has This Been Tested? Set the job concurrency to a high number (such as 8) for either image tagging or facial recognition (the CLIP model has no logs) and start the job. The logs should only show one instance of the corresponding log below. I tested this with all ML jobs running simultaneously and saw it showed correct behavior (on `main`, this swelled memory usage to over 10gb). Image classification log: `Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.` Facial recognition log: ``` Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} model ignore: /cache/facial-recognition/buffalo_l/models/buffalo_l/1k3d68.onnx landmark_3d_68 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} model ignore: /cache/facial-recognition/buffalo_l/models/buffalo_l/2d106det.onnx landmark_2d_106 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /cache/facial-recognition/buffalo_l/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0 Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} model ignore: /cache/facial-recognition/buffalo_l/models/buffalo_l/genderage.onnx genderage Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} find model: /cache/facial-recognition/buffalo_l/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5 set det-size: (640, 640) ```
OVERLORD added the pull-request label 2026-02-05 14:12:29 +03:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#9890
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?