immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 08:41:28 +03:00
Code Issues 27 Packages Projects Releases 10 Wiki Activity

CLI Upload is getting slower and slower, depends on Album size #2062

New Issue
Closed
opened 2026-02-05 04:55:14 +03:00 by OVERLORD · 13 comments
OVERLORD commented 2026-02-05 04:55:14 +03:00
Owner
Copy Link

Originally created by @commanderalpha on GitHub (Jan 26, 2024).

Originally assigned to: @jrasm91 on GitHub.

The bug

Hello,
I'm uploading my camera Folder to Immich with the CLI Tool. This works fine after the bug with the too many files are fixed.
But now I see some other issue. I use the album function to create albums out of the folder names. But I have one folder with around 30k of potos und movies (around 200GB).
I see that the speed in files/sec is getting slower and slower. I aborted the uplaod at around 50%. At this time the speed was slow one pictures every three seconds.
So I moved all the uploaded file from the folder, so that the queue is not that big. But this doesnt helped.

But I figured out that depends on the items in the album in Immich.
When i start the Upload to a new album like "camera2" the speed is back at normal condition. Of course getting slower. Maybe i will use a third album.

Best regards

The OS that Immich Server is running on

Docker in Unraid

Version of Immich Server

1.93.3

Version of Immich Mobile App

doesnt matter

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

I dont know how to get the right information. 

I'm running three Containers for Immich (Redis, Postgree, Immich)

Your .env content

I dont know how to get the right information.

Reproduction steps

see above

Additional information

Jobs on the server are paused during upload to get max upload performance

Originally created by @commanderalpha on GitHub (Jan 26, 2024). Originally assigned to: @jrasm91 on GitHub. ### The bug Hello, I'm uploading my camera Folder to Immich with the CLI Tool. This works fine after the bug with the too many files are fixed. But now I see some other issue. I use the album function to create albums out of the folder names. But I have one folder with around 30k of potos und movies (around 200GB). I see that the speed in files/sec is getting slower and slower. I aborted the uplaod at around 50%. At this time the speed was slow one pictures every three seconds. So I moved all the uploaded file from the folder, so that the queue is not that big. But this doesnt helped. But I figured out that depends on the items in the album in Immich. When i start the Upload to a new album like "camera2" the speed is back at normal condition. Of course getting slower. Maybe i will use a third album. Best regards ### The OS that Immich Server is running on Docker in Unraid ### Version of Immich Server 1.93.3 ### Version of Immich Mobile App doesnt matter ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML I dont know how to get the right information. I'm running three Containers for Immich (Redis, Postgree, Immich) ``` ### Your .env content ```Shell I dont know how to get the right information. ``` ### Reproduction steps ```bash see above ``` ### Additional information Jobs on the server are paused during upload to get max upload performance
OVERLORD added the cli label 2026-02-05 04:55:14 +03:00
OVERLORD commented 2026-02-05 04:55:25 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Jan 26, 2024):

As you are doing bulk upload, your server CPU and disk IO usage will be maxed out, so this is expected

@alextran1502 commented on GitHub (Jan 26, 2024): As you are doing bulk upload, your server CPU and disk IO usage will be maxed out, so this is expected
OVERLORD commented 2026-02-05 04:55:27 +03:00
Author
Owner
Copy Link

@commanderalpha commented on GitHub (Jan 27, 2024):

Hi Alex,

no i don't think its a performance issue, because:

  1. I paused the server Jobs at the immich server
  2. When I observe the CPU Load and Disk IO it doesn't look that the server is under heavy load.
  3. If you already have an album with 20k items, the upload is slow from the first picture.
  4. If you upload the same files again in the same album, they didn't transver due to duplicate check, but speed is super slow.
  5. Even if you dont stop the Jobs on the server and the load is high, the upload is fast at the begining.
  6. The speed decreases linear form item to item. As i wrote, after some hours you reach just one picture every 3-4 seconds, immich server is faster at this point and got all the jobs from the first picutre done.

Maybe someone else can check this?
Thanks.

But yes of course, such kind of upload you just one or two times.

BR

@commanderalpha commented on GitHub (Jan 27, 2024): Hi Alex, no i don't think its a performance issue, because: 1. I paused the server Jobs at the immich server 2. When I observe the CPU Load and Disk IO it doesn't look that the server is under heavy load. 3. If you already have an album with 20k items, the upload is slow from the first picture. 4. If you upload the same files again in the same album, they didn't transver due to duplicate check, but speed is super slow. 5. Even if you dont stop the Jobs on the server and the load is high, the upload is fast at the begining. 6. The speed decreases linear form item to item. As i wrote, after some hours you reach just one picture every 3-4 seconds, immich server is faster at this point and got all the jobs from the first picutre done. Maybe someone else can check this? Thanks. But yes of course, such kind of upload you just one or two times. BR
OVERLORD commented 2026-02-05 04:55:35 +03:00
Author
Owner
Copy Link

@capsyweb commented on GitHub (Feb 21, 2024):

Hello, I used to get immich working with the official method, and used to get cli working as well, now I am using the 3 containers like you but I can't seem to find a way to make cli work, how did you get yours to work?

@capsyweb commented on GitHub (Feb 21, 2024): Hello, I used to get immich working with the official method, and used to get cli working as well, now I am using the 3 containers like you but I can't seem to find a way to make cli work, how did you get yours to work?
OVERLORD commented 2026-02-05 04:55:38 +03:00
Author
Owner
Copy Link

@pelaxa commented on GitHub (Mar 2, 2024):

I am noticing the same thing for Movies. they upload very slowly. A 470M file took many minutes to a point where I thought something had gone wrong. The server has almost no load at all.

@pelaxa commented on GitHub (Mar 2, 2024): I am noticing the same thing for Movies. they upload very slowly. A 470M file took many minutes to a point where I thought something had gone wrong. The server has almost no load at all.
OVERLORD commented 2026-02-05 04:55:47 +03:00
Author
Owner
Copy Link

@Doomstein007 commented on GitHub (Mar 27, 2024):

Uploading a 3,5 GB Video, take very long only uploading with 3,3 MB over WLAN. Server not under load

@Doomstein007 commented on GitHub (Mar 27, 2024): Uploading a 3,5 GB Video, take very long only uploading with 3,3 MB over WLAN. Server not under load
OVERLORD commented 2026-02-05 04:55:49 +03:00
Author
Owner
Copy Link

@mertalev commented on GitHub (Apr 1, 2024):

This issue is specific to the slowdown described when setting an album in the CLI as the album grows in size.

@commanderalpha Can you share if you still see this behavior on the latest version of the CLI? It's been heavily refactored recently.

@mertalev commented on GitHub (Apr 1, 2024): This issue is specific to the slowdown described when setting an album in the CLI as the album grows in size. @commanderalpha Can you share if you still see this behavior on the latest version of the CLI? It's been heavily refactored recently.
OVERLORD commented 2026-02-05 04:55:50 +03:00
Author
Owner
Copy Link

@commanderalpha commented on GitHub (Apr 21, 2024):

Hi,
I tried it. Uploading a Folder with 37k images into an album with around 33k images. So uploading around 4k images.
crawling assets take some time, but that seems ok. Uploading is working as expected.
So for me, it seems to be fixed.
THX.

BR

@commanderalpha commented on GitHub (Apr 21, 2024): Hi, I tried it. Uploading a Folder with 37k images into an album with around 33k images. So uploading around 4k images. crawling assets take some time, but that seems ok. Uploading is working as expected. So for me, it seems to be fixed. THX. BR
OVERLORD commented 2026-02-05 04:55:52 +03:00
Author
Owner
Copy Link

@nsubiron commented on GitHub (Jul 25, 2024):

I'm seeing this issue on the web interface, uploading is fast but then the item is stuck in adding to album for several seconds blocking the queue. The bigger the album the slower it gets. The server is running on a pretty old PC, the issue is already pretty bad even before getting to 1k images on the same album.

@nsubiron commented on GitHub (Jul 25, 2024): I'm seeing this issue on the web interface, uploading is fast but then the item is stuck in adding to album for several seconds blocking the queue. The bigger the album the slower it gets. The server is running on a pretty old PC, the issue is already pretty bad even before getting to 1k images on the same album.
OVERLORD commented 2026-02-05 04:55:54 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 9, 2024):

This has been fixed in the CLI.

@jrasm91 commented on GitHub (Sep 9, 2024): This has been fixed in the CLI.
OVERLORD commented 2026-02-05 04:55:59 +03:00
Author
Owner
Copy Link

@benjaminberlin commented on GitHub (Sep 20, 2024):

I want to upload 140,000 images with immich go. Currently it uploads 350 images in 35 minutes. The estimated total time is therefore 242 hours, or 10 days with 100% CPU load

@benjaminberlin commented on GitHub (Sep 20, 2024): I want to upload 140,000 images with immich go. Currently it uploads 350 images in 35 minutes. The estimated total time is therefore 242 hours, or 10 days with 100% CPU load
OVERLORD commented 2026-02-05 04:56:00 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 20, 2024):

This is unrelated to this bug, but feel free to open a new discussion here: https://github.com/immich-app/immich/discussions/categories/q-a

@jrasm91 commented on GitHub (Sep 20, 2024): This is unrelated to this bug, but feel free to open a new discussion here: https://github.com/immich-app/immich/discussions/categories/q-a
OVERLORD commented 2026-02-05 04:56:06 +03:00
Author
Owner
Copy Link

@sbhal commented on GitHub (Oct 31, 2025):

My upload has been going on for over 1 week.

root@26544b71b816:/usr/src/app# immich upload --ignore "/@eaDir/" --recursive "$MY_IMPORT_PATH"
Crawling for assets...
Hashing files           | ████████████████████████████████████████ | 100% | ETA: 0s | 796896/796896 assets
Checking for duplicates | ████████████████████████████████████████ | 100% | ETA: 0s | 796896/796896 assets
Found 467009 new files and 329887 duplicates
Uploading assets | ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ | 38% | ETA: NFs | 495.7 GB/680.2 GB
Uploading assets | █████████████████░░░░░░░░░░░░░░░░░░░░░░░ | 41% | ETA: 09h30m | 540.1 GB/680.2 GB

sbhal@DS920PLUS:~$ top -b -n 1 | head -n20
top - 12:50:47 up 92 days, 11:36,  1 user,  load average: 17.50, 17.41, 17.52 [IO: 14.74, 14.44, 14.22 CPU: 2.76, 2.97, 3.28]
Tasks: 1182 total,   2 running, 1176 sleeping,   0 stopped,   4 zombie
%Cpu(s): 31.2 us, 10.4 sy,  0.0 ni, 14.6 id, 42.7 wa,  0.0 hi,  1.0 si,  0.0 st
GiB Mem :   19.387 total,    0.385 free,   14.716 used,    4.286 buff/cache
GiB Swap:   13.633 total,    1.686 free,   11.947 used.    3.603 avail Mem
@sbhal commented on GitHub (Oct 31, 2025): My upload has been going on for over 1 week. ``` root@26544b71b816:/usr/src/app# immich upload --ignore "/@eaDir/" --recursive "$MY_IMPORT_PATH" Crawling for assets... Hashing files | ████████████████████████████████████████ | 100% | ETA: 0s | 796896/796896 assets Checking for duplicates | ████████████████████████████████████████ | 100% | ETA: 0s | 796896/796896 assets Found 467009 new files and 329887 duplicates Uploading assets | ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ | 38% | ETA: NFs | 495.7 GB/680.2 GB Uploading assets | █████████████████░░░░░░░░░░░░░░░░░░░░░░░ | 41% | ETA: 09h30m | 540.1 GB/680.2 GB sbhal@DS920PLUS:~$ top -b -n 1 | head -n20 top - 12:50:47 up 92 days, 11:36, 1 user, load average: 17.50, 17.41, 17.52 [IO: 14.74, 14.44, 14.22 CPU: 2.76, 2.97, 3.28] Tasks: 1182 total, 2 running, 1176 sleeping, 0 stopped, 4 zombie %Cpu(s): 31.2 us, 10.4 sy, 0.0 ni, 14.6 id, 42.7 wa, 0.0 hi, 1.0 si, 0.0 st GiB Mem : 19.387 total, 0.385 free, 14.716 used, 4.286 buff/cache GiB Swap: 13.633 total, 1.686 free, 11.947 used. 3.603 avail Mem ```
OVERLORD commented 2026-02-05 04:56:12 +03:00
Author
Owner
Copy Link

@Kyle-192 commented on GitHub (Nov 28, 2025):

Confirming This Issue with Extensive Testing (v2.3.1)

Complete transparency: I was using claude to help me through the issues and create this comment. I'm no expert, just a guy trying to learn and use proxmox and immich.

I'm experiencing the exact same progressive slowdown issue described here. After extensive troubleshooting, I can confirm this is a CLI-specific bug and not infrastructure-related.

My Setup

  • Immich Version: v2.3.1
  • Immich CLI: Latest (installed via npm install -g @immich/cli)
  • Server: Ubuntu VM (Proxmox), Docker Compose setup
  • Client: Ubuntu Desktop VM
  • Storage: NFS4 mount to Asustor NAS (7.3TB capacity, 6.9TB free)
  • Network: Gigabit, 1ms latency between VMs
  • Source: USB 3.0 drive (149 MB/s read speed)

Observed Behavior

Initial Performance (First 1-2 hours):

  • Upload speed: 20-22 MB/s ✅
  • Progress: Steady and reliable
  • Everything works perfectly

After 3-6 Hours of Sustained Upload:

  • Upload speed gradually degrades to ~700 KB/s (30x slower!)
  • Eventually upload completely freezes
  • Terminal shows no progress updates
  • Process still running (visible in ps aux) but no network activity
  • No error messages displayed - just hangs silently

What I Tested

I conducted extensive diagnostics to rule out infrastructure issues:

✅ Hardware Verified Healthy

  • USB Drive: 149 MB/s read speed (tested with hdparm)
  • NAS Write: Capable of normal speeds when not uploading via CLI
  • Network: 1ms ping, no packet loss, gigabit connection
  • CPU: Both client (1-3%) and server (7-9%) very low usage
  • RAM: Plenty available (6GB free on 8GB Immich server)

✅ Redis Verified Healthy

$ docker exec immich_redis redis-cli INFO memory
used_memory_human:8.71M  # Very low usage

$ docker exec immich_redis redis-cli INFO clients
connected_clients:73
blocked_clients:17

$ docker exec immich_redis redis-cli KEYS "bull:*"
(empty - no queued jobs)

✅ Immich Server Responsive

  • Docker containers all healthy
  • No errors in logs
  • Web interface fully functional

✅ Network Interface

$ ethtool enp6s18 | grep Speed
Speed: Unknown!

Note: Interface shows "Unknown" speed but actual throughput is fine (verified with web upload)

The Smoking Gun: Web Upload Works Perfectly

Using the Immich web interface to upload the same files:

  • Consistent 18+ MB/s upload speed
  • No slowdown over time
  • No freezing
  • Completes successfully

This definitively proves the issue is CLI-specific, not server/infrastructure related.

Tested Configurations (All Failed Similarly)

  1. immich upload --recursive --skip-hash (default)
  2. immich upload --recursive --concurrency 4 (default concurrency)
  3. immich upload --recursive --concurrency 2 (reduced)
  4. immich upload --recursive --concurrency 1 (minimal)
  5. immich upload --recursive (without skip-hash, checks hashes locally)

All configurations exhibited the same pattern:

  • Start fast (20+ MB/s)
  • Gradually slow down over hours
  • Eventually freeze completely

Suspected Root Cause

Based on extensive testing, this appears to be a connection pool/memory management issue in the CLI:

  1. Connection pool exhaustion - connections not being properly closed/recycled during sustained operations
  2. Memory leak - gradual performance degradation over hours suggests accumulating memory issues
  3. Timeout accumulation - failed requests might not be handled properly, causing cascade failures

The fact that web upload maintains consistent 18+ MB/s while CLI degrades suggests the CLI's HTTP client (Axios) or connection handling has issues with long-running upload sessions.

Current Workaround

Upload in small batches with fresh CLI process each time:

# Upload one month at a time
immich upload --recursive --album "Photos" "/path/to/2024/01"
immich upload --recursive --album "Photos" "/path/to/2024/02"
# etc.

Starting a fresh process for each batch prevents the slowdown, but is tedious for large libraries.

Impact

This makes the CLI nearly unusable for migrating large photo libraries (a primary use case for new Immich users). The alternative (web interface) works but:

  • Requires manual file selection through browser
  • Harder to organize by folder structure
  • Not scriptable/automatable

Additional Context

During testing, I observed that Redis connections would increase to ~73 during active uploads but remain healthy. The CLI process would continue running with 17% CPU usage but establish zero network connections to the Immich server when frozen - suggesting the issue is in the CLI's network layer, not Redis or the server itself.

This is a significant usability issue for initial library imports. The CLI should be the recommended method for bulk uploads, but currently the web interface is more reliable despite being less convenient.

Happy to provide additional logs, testing, or debugging information if it would help resolve this issue.

@Kyle-192 commented on GitHub (Nov 28, 2025): ## Confirming This Issue with Extensive Testing (v2.3.1) **Complete transparency: I was using claude to help me through the issues and create this comment. I'm no expert, just a guy trying to learn and use proxmox and immich.** I'm experiencing the exact same progressive slowdown issue described here. After extensive troubleshooting, I can confirm this is a **CLI-specific bug** and not infrastructure-related. ### My Setup - **Immich Version:** v2.3.1 - **Immich CLI:** Latest (installed via `npm install -g @immich/cli`) - **Server:** Ubuntu VM (Proxmox), Docker Compose setup - **Client:** Ubuntu Desktop VM - **Storage:** NFS4 mount to Asustor NAS (7.3TB capacity, 6.9TB free) - **Network:** Gigabit, 1ms latency between VMs - **Source:** USB 3.0 drive (149 MB/s read speed) ### Observed Behavior **Initial Performance (First 1-2 hours):** - Upload speed: **20-22 MB/s** ✅ - Progress: Steady and reliable - Everything works perfectly **After 3-6 Hours of Sustained Upload:** - Upload speed gradually degrades to **~700 KB/s** (30x slower!) - Eventually upload **completely freezes** - Terminal shows no progress updates - Process still running (visible in `ps aux`) but **no network activity** - **No error messages displayed** - just hangs silently ### What I Tested I conducted extensive diagnostics to rule out infrastructure issues: #### ✅ Hardware Verified Healthy - **USB Drive:** 149 MB/s read speed (tested with `hdparm`) - **NAS Write:** Capable of normal speeds when not uploading via CLI - **Network:** 1ms ping, no packet loss, gigabit connection - **CPU:** Both client (1-3%) and server (7-9%) very low usage - **RAM:** Plenty available (6GB free on 8GB Immich server) #### ✅ Redis Verified Healthy ```bash $ docker exec immich_redis redis-cli INFO memory used_memory_human:8.71M # Very low usage $ docker exec immich_redis redis-cli INFO clients connected_clients:73 blocked_clients:17 $ docker exec immich_redis redis-cli KEYS "bull:*" (empty - no queued jobs) ``` #### ✅ Immich Server Responsive - Docker containers all healthy - No errors in logs - Web interface fully functional #### ✅ Network Interface ```bash $ ethtool enp6s18 | grep Speed Speed: Unknown! ``` Note: Interface shows "Unknown" speed but actual throughput is fine (verified with web upload) ### The Smoking Gun: Web Upload Works Perfectly **Using the Immich web interface to upload the same files:** - Consistent **18+ MB/s** upload speed - No slowdown over time - No freezing - Completes successfully This definitively proves the issue is **CLI-specific**, not server/infrastructure related. ### Tested Configurations (All Failed Similarly) 1. `immich upload --recursive --skip-hash` (default) 2. `immich upload --recursive --concurrency 4` (default concurrency) 3. `immich upload --recursive --concurrency 2` (reduced) 4. `immich upload --recursive --concurrency 1` (minimal) 5. `immich upload --recursive` (without skip-hash, checks hashes locally) **All configurations exhibited the same pattern:** - Start fast (20+ MB/s) - Gradually slow down over hours - Eventually freeze completely ### Suspected Root Cause Based on extensive testing, this appears to be a **connection pool/memory management issue** in the CLI: 1. **Connection pool exhaustion** - connections not being properly closed/recycled during sustained operations 2. **Memory leak** - gradual performance degradation over hours suggests accumulating memory issues 3. **Timeout accumulation** - failed requests might not be handled properly, causing cascade failures The fact that web upload maintains consistent 18+ MB/s while CLI degrades suggests the CLI's HTTP client (Axios) or connection handling has issues with long-running upload sessions. ### Current Workaround **Upload in small batches with fresh CLI process each time:** ```bash # Upload one month at a time immich upload --recursive --album "Photos" "/path/to/2024/01" immich upload --recursive --album "Photos" "/path/to/2024/02" # etc. ``` Starting a fresh process for each batch prevents the slowdown, but is tedious for large libraries. ### Impact This makes the CLI nearly **unusable for migrating large photo libraries** (a primary use case for new Immich users). The alternative (web interface) works but: - Requires manual file selection through browser - Harder to organize by folder structure - Not scriptable/automatable ### Additional Context During testing, I observed that Redis connections would increase to ~73 during active uploads but remain healthy. The CLI process would continue running with 17% CPU usage but establish zero network connections to the Immich server when frozen - suggesting the issue is in the CLI's network layer, not Redis or the server itself. --- **This is a significant usability issue for initial library imports.** The CLI should be the recommended method for bulk uploads, but currently the web interface is more reliable despite being less convenient. Happy to provide additional logs, testing, or debugging information if it would help resolve this issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label cli
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2062
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?