immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Job "EXTRACT METADATA" looks like doesn't scale up #2597

New Issue
Closed
opened 2026-02-05 06:15:46 +03:00 by OVERLORD · 5 comments
OVERLORD commented 2026-02-05 06:15:46 +03:00
Owner
Copy Link

Originally created by @marcin-github on GitHub (Mar 15, 2024).

The bug

I've got library with ~500k of images. I forced to rescan library, now I have queue with 190 000 elements waiting for metadata extraction. I changed concurrent jobs to higher value, load on server didn't changed. I set "Extract Metadata Concurrency" to 150 - load is still low. No I/O wait, cpu aren't burning. Looks like changing this value doesn't change number of threads.

The OS that Immich Server is running on

docker container

Version of Immich Server

1.98.2

Version of Immich Mobile App

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

---

Your .env content

---

Reproduction steps

---

Additional information

No response

Originally created by @marcin-github on GitHub (Mar 15, 2024). ### The bug I've got library with ~500k of images. I forced to rescan library, now I have queue with 190 000 elements waiting for metadata extraction. I changed concurrent jobs to higher value, load on server didn't changed. I set "Extract Metadata Concurrency" to 150 - load is still low. No I/O wait, cpu aren't burning. Looks like changing this value doesn't change number of threads. ### The OS that Immich Server is running on docker container ### Version of Immich Server 1.98.2 ### Version of Immich Mobile App --- ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML --- ``` ### Your .env content ```Shell --- ``` ### Reproduction steps ```bash --- ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 06:15:50 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Mar 15, 2024):

This was because of a slow query, see #7909.

@mertalev commented on GitHub (Mar 15, 2024): This was because of a slow query, see #7909.
OVERLORD commented 2026-02-05 06:15:58 +03:00
Author
Owner
Copy Link

@marcin-github commented on GitHub (Mar 15, 2024):

I'm not sure about it... Because:
a) I added "the three indexes" ;)
b) there were no high CPU usage from postgres
c) there were no sql queries longer than 5-7ms
d) I tried the same for lesser much library and it still behaves in the same way

@marcin-github commented on GitHub (Mar 15, 2024): I'm not sure about it... Because: a) I added "the three indexes" ;) b) there were no high CPU usage from postgres c) there were no sql queries longer than 5-7ms d) I tried the same for lesser much library and it still behaves in the same way
OVERLORD commented 2026-02-05 06:16:01 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Mar 15, 2024):

Ah, then you're probably hitting against exiftool-vendored's built-in throttling. I'm not sure if this can be changed on-the-fly when the concurrency is changed.

@mertalev commented on GitHub (Mar 15, 2024): Ah, then you're probably hitting against exiftool-vendored's built-in [throttling](https://www.npmjs.com/package/exiftool-vendored#parallelism). I'm not sure if this can be changed on-the-fly when the concurrency is changed.
OVERLORD commented 2026-02-05 06:16:02 +03:00
Author
Owner
Copy Link

@marcin-github commented on GitHub (Mar 18, 2024):

It could be as you said, thank you.

@marcin-github commented on GitHub (Mar 18, 2024): It could be as you said, thank you.
OVERLORD commented 2026-02-05 06:16:09 +03:00
Author
Owner
Copy Link

@GRbit commented on GitHub (Jul 15, 2024):

@mertalev do you think this issue could be reopened to do some magic around this exiftool-vendored lib?

It feels pretty awkward with 8 cores having only 4 exiftool processes running giving me ~10% cpu load. From parallelization factor 8 the average speed doesn't change. I'll take days until my 200k photos will be processed 😢

@GRbit commented on GitHub (Jul 15, 2024): @mertalev do you think this issue could be reopened to do some magic around this exiftool-vendored lib? It feels pretty awkward with 8 cores having only 4 exiftool processes running giving me ~10% cpu load. From parallelization factor 8 the average speed doesn't change. I'll take days until my 200k photos will be processed :cry:
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2597
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?