immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Emptying Trash with a lot of offline images doesn't work #5459

New Issue
Closed
opened 2026-02-05 11:28:41 +03:00 by OVERLORD · 3 comments
OVERLORD commented 2026-02-05 11:28:41 +03:00
Owner
Copy Link

Originally created by @daniel-freiermuth on GitHub (Feb 24, 2025).

The bug

Background

Immich instance with ~80K pictures. I happened to add immich's library folder as an external library (by adding a wrong docker mount). This caused a lot of new images (~1000K) to be added to immich 🙈 When I realized my mistake the next morning, immich was already happily processing those. After fixing the mount problem, immich correctly recognized the newly added images as offline and put those into the trash. Now I'd like to empy the trash and that's where the problem starts.

The bug

In the Web UI, when clicking on the button "Empy Trash", the immediately says "Permanently deleted 0 assets." No logs are happening (docker-compose logs -f).

Selecting a few images (~200) and then deleting those works.

Selecting more (~3K) and then trying to delete those also brings the bubble with the correct count, but the images still seem to be there. (If I understood things correctly, immich will enqueue the images for deletion, so it makes sense that they are not gone right away?)

When trying to delete all or a lot of images, the immich_server container finally exits with the logs [1].

Another logged error that might or might be related is [2]. I couldn't figure out a causal connection to me using the Web UI.

The OS that Immich Server is running on

Docker on Fedora 40

Version of Immich Server

v1.126.1

Version of Immich Mobile App

n.a.

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload:z
      - /etc/localtime:/etc/localtime:ro,z
      - /var/lib/photos/input:/eos_photos/input:z
    env_file:
      - .env
    ports:
      - 2283:2283
      - 8081:8081
    depends_on:
      - redis
      - database

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    #image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    #image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data:z
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=v1.126.1
IMMICH_TELEMETRY_INCLUDE=all

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=[redacted]

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

  1. Add a lot of images via external library.
  2. Remove the files again.
  3. Try to delete the now-offline images.

Relevant log output

[1]:
immich_server              | 
immich_server              | <--- Last few GCs --->
immich_server              | 
immich_server              | [18:0x32f40024000]   328067 ms: Mark-Compact 4069.5 (4143.8) -> 4068.8 (4144.6) MB, pooled: 0 MB, 4977.27 / 0.00 ms  (average mu = 0.217, current mu = 0.016) allocation failure; GC in old space requested
immich_server              | 
immich_server              | 
immich_server              | <--- JS stacktrace --->
immich_server              | 
immich_server              | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
immich_server              | ----- Native stack trace -----
immich_server              | 
immich_server              |  1: 0xe3811e node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  2: 0x125fb70 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  3: 0x125fe47 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  4: 0x148d885  [immich-api]
immich_server              |  5: 0x14a70f9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich-api]
immich_server              |  6: 0x147b7c8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]
immich_server              |  7: 0x147c6f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]
immich_server              |  8: 0x14553ce v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich-api]
immich_server              |  9: 0x18b6b50 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [immich-api]
immich_server              | 10: 0x7f5ffedac476 
immich_server              | api worker exited with code null


and a slightly different one:

immich_server              | 
immich_server              | <--- Last few GCs --->
immich_server              | 
immich_server              | [17:0x43cd2024000]   451806 ms: Scavenge (interleaved) 4045.8 (4124.7) -> 4043.9 (4128.4) MB, pooled: 0 MB, 34.61 / 0.00 ms  (average mu = 0.674, current mu = 0.479) allocation failure; 
immich_server              | [17:0x43cd2024000]   451983 ms: Scavenge (interleaved) 4049.4 (4128.4) -> 4047.0 (4146.7) MB, pooled: 0 MB, 133.47 / 0.00 ms  (average mu = 0.674, current mu = 0.479) allocation failure; 
immich_server              | 
immich_server              | 
immich_server              | <--- JS stacktrace --->
immich_server              | 
immich_server              | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
immich_server              | ----- Native stack trace -----
immich_server              | 
immich_server              |  1: 0xe3811e node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  2: 0x125fb70 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  3: 0x125fe47 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api]
immich_server              |  4: 0x148d885  [immich-api]
immich_server              |  5: 0x14a70f9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich-api]
immich_server              |  6: 0x147b7c8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]
immich_server              |  7: 0x147c6f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api]
immich_server              |  8: 0x14553ce v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich-api]
immich_server              |  9: 0x18b69cc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich-api]
immich_server              | 10: 0x7f750ceec476 
immich_server              | api worker exited with code null


——————————————————————————————————————————————————————————————————————
[2]:
immich_server              | [Nest] 18  - 02/24/2025, 7:24:43 PM   ERROR [Api:RangeError: Invalid string length
immich_server              |     at JSON.stringify (<anonymous>)
immich_server              |     at stringify (/usr/src/app/node_modules/express/lib/response.js:1008:12)
immich_server              |     at ServerResponse.json (/usr/src/app/node_modules/express/lib/response.js:234:14)
immich_server              |     at ExpressAdapter.reply (/usr/src/app/node_modules/@nestjs/platform-express/adapters/express-adapter.js:62:62)
immich_server              |     at RouterResponseController.apply (/usr/src/app/node_modules/@nestjs/core/router/router-response-controller.js:15:36)
immich_server              |     at /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:176:48
immich_server              |     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
immich_server              |     at async /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:47:13
immich_server              |     at async /usr/src/app/node_modules/@nestjs/core/router/router-proxy.js:9:17~86qoqsji] Unknown error: RangeError: Invalid string length

Additional information

First of all, Thank you so much into looking this!! And of course for creating immich.

The issue points towards RAM. Can I increase the RAM availabe to immich somehow (my hardware certainly has enough)? But I guess, actually immich shouldn't take so much.

Please let me know any more logs I should catch or experiments I should perform.

Originally created by @daniel-freiermuth on GitHub (Feb 24, 2025). ### The bug # Background Immich instance with ~80K pictures. I happened to add immich's library folder as an external library (by adding a wrong docker mount). This caused a lot of new images (~1000K) to be added to immich 🙈 When I realized my mistake the next morning, immich was already happily processing those. After fixing the mount problem, immich correctly recognized the newly added images as offline and put those into the trash. Now I'd like to empy the trash and that's where the problem starts. # The bug In the Web UI, when clicking on the button "Empy Trash", the immediately says "Permanently deleted 0 assets." No logs are happening (docker-compose logs -f). Selecting a few images (~200) and then deleting those works. Selecting more (~3K) and then trying to delete those also brings the bubble with the correct count, but the images still seem to be there. (If I understood things correctly, immich will enqueue the images for deletion, so it makes sense that they are not gone right away?) When trying to delete all or a lot of images, the immich_server container finally exits with the logs [1]. Another logged error that might or might be related is [2]. I couldn't figure out a causal connection to me using the Web UI. ### The OS that Immich Server is running on Docker on Fedora 40 ### Version of Immich Server v1.126.1 ### Version of Immich Mobile App n.a. ### Platform with the issue - [x] Server - [x] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/usr/src/app/upload:z - /etc/localtime:/etc/localtime:ro,z - /var/lib/photos/input:/eos_photos/input:z env_file: - .env ports: - 2283:2283 - 8081:8081 depends_on: - redis - database immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache env_file: - .env redis: container_name: immich_redis image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e #image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672 database: container_name: immich_postgres image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 #image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data:z command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"] volumes: model-cache: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=./library # The location where your database files are stored DB_DATA_LOCATION=./postgres # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=v1.126.1 IMMICH_TELEMETRY_INCLUDE=all # Connection secret for postgres. You should change it to a random password DB_PASSWORD=[redacted] # The values below this line do not need to be changed ################################################################################### DB_USERNAME=postgres DB_DATABASE_NAME=immich ``` ### Reproduction steps 1. Add a lot of images via external library. 2. Remove the files again. 3. Try to delete the now-offline images. ### Relevant log output ```shell [1]: immich_server | immich_server | <--- Last few GCs ---> immich_server | immich_server | [18:0x32f40024000] 328067 ms: Mark-Compact 4069.5 (4143.8) -> 4068.8 (4144.6) MB, pooled: 0 MB, 4977.27 / 0.00 ms (average mu = 0.217, current mu = 0.016) allocation failure; GC in old space requested immich_server | immich_server | immich_server | <--- JS stacktrace ---> immich_server | immich_server | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory immich_server | ----- Native stack trace ----- immich_server | immich_server | 1: 0xe3811e node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich-api] immich_server | 2: 0x125fb70 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api] immich_server | 3: 0x125fe47 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api] immich_server | 4: 0x148d885 [immich-api] immich_server | 5: 0x14a70f9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich-api] immich_server | 6: 0x147b7c8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api] immich_server | 7: 0x147c6f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api] immich_server | 8: 0x14553ce v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich-api] immich_server | 9: 0x18b6b50 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [immich-api] immich_server | 10: 0x7f5ffedac476 immich_server | api worker exited with code null and a slightly different one: immich_server | immich_server | <--- Last few GCs ---> immich_server | immich_server | [17:0x43cd2024000] 451806 ms: Scavenge (interleaved) 4045.8 (4124.7) -> 4043.9 (4128.4) MB, pooled: 0 MB, 34.61 / 0.00 ms (average mu = 0.674, current mu = 0.479) allocation failure; immich_server | [17:0x43cd2024000] 451983 ms: Scavenge (interleaved) 4049.4 (4128.4) -> 4047.0 (4146.7) MB, pooled: 0 MB, 133.47 / 0.00 ms (average mu = 0.674, current mu = 0.479) allocation failure; immich_server | immich_server | immich_server | <--- JS stacktrace ---> immich_server | immich_server | FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory immich_server | ----- Native stack trace ----- immich_server | immich_server | 1: 0xe3811e node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich-api] immich_server | 2: 0x125fb70 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api] immich_server | 3: 0x125fe47 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich-api] immich_server | 4: 0x148d885 [immich-api] immich_server | 5: 0x14a70f9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich-api] immich_server | 6: 0x147b7c8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api] immich_server | 7: 0x147c6f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich-api] immich_server | 8: 0x14553ce v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich-api] immich_server | 9: 0x18b69cc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich-api] immich_server | 10: 0x7f750ceec476 immich_server | api worker exited with code null —————————————————————————————————————————————————————————————————————— [2]: immich_server | [Nest] 18 - 02/24/2025, 7:24:43 PM ERROR [Api:RangeError: Invalid string length immich_server | at JSON.stringify (<anonymous>) immich_server | at stringify (/usr/src/app/node_modules/express/lib/response.js:1008:12) immich_server | at ServerResponse.json (/usr/src/app/node_modules/express/lib/response.js:234:14) immich_server | at ExpressAdapter.reply (/usr/src/app/node_modules/@nestjs/platform-express/adapters/express-adapter.js:62:62) immich_server | at RouterResponseController.apply (/usr/src/app/node_modules/@nestjs/core/router/router-response-controller.js:15:36) immich_server | at /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:176:48 immich_server | at process.processTicksAndRejections (node:internal/process/task_queues:105:5) immich_server | at async /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:47:13 immich_server | at async /usr/src/app/node_modules/@nestjs/core/router/router-proxy.js:9:17~86qoqsji] Unknown error: RangeError: Invalid string length ``` ### Additional information First of all, Thank you so much into looking this!! And of course for creating immich. The issue points towards RAM. Can I increase the RAM availabe to immich somehow (my hardware certainly has enough)? But I guess, actually immich shouldn't take so much. Please let me know any more logs I should catch or experiments I should perform.
OVERLORD added the 📱mobile label 2026-02-05 11:28:41 +03:00
OVERLORD commented 2026-02-05 11:28:44 +03:00
Author
Owner
Copy Link

@JayBigGuy10 commented on GitHub (Apr 24, 2025):

I am still experiencing this too on V1.131.3

I was using a different method (DSFile) to backup my images to my server from android, which were then added to immich as part of a big external library (6k out of about 50k).

I decided to bite the bullet and remove them from the external library and re-upload them using the immich android app back up.

Was confused when I couldn't find the remove offline files job and found these related to the change

https://github.com/immich-app/immich/discussions/7529
https://github.com/immich-app/immich/pull/12220

When I try and empty my trash i experience the same behaviour as above

@JayBigGuy10 commented on GitHub (Apr 24, 2025): I am still experiencing this too on V1.131.3 I was using a different method (DSFile) to backup my images to my server from android, which were then added to immich as part of a big external library (6k out of about 50k). I decided to bite the bullet and remove them from the external library and re-upload them using the immich android app back up. Was confused when I couldn't find the remove offline files job and found these related to the change https://github.com/immich-app/immich/discussions/7529 https://github.com/immich-app/immich/pull/12220 When I try and empty my trash i experience the same behaviour as above
OVERLORD commented 2026-02-05 11:28:45 +03:00
Author
Owner
Copy Link

@daniel-freiermuth commented on GitHub (Apr 27, 2025):

Update from my side:

I also tried using scripts [1, 2] that use the API for deleting the offline files, without luck. It turns out that they use the report endpoint [3] for retrieving the list of offline files. Unfortunately, just requesting this endpoint already triggers the crash.
[1] https://github.com/Thoroslives/immich_remove_offline_files/blob/main/immich_remove_offline_files.py , seemingly updated variant of
[2] https://github.com/clumsyCoder00/Immich-Tools/blob/main/(deprecated)%20immich-del-offline.py
[3] https://immich.app/docs/api/get-audit-files/

In contrast, the web interface is able to show the offline files. Turns out, it uses different endpoints [4, 5] which do not trigger the crash.
[4] https://immich.app/docs/api/get-time-buckets
[5] https://immich.app/docs/api/get-time-bucket

I amended the script from above using this endpoint. The script then requests all files to be deleted separately via the asset deletion endpoint [6]. Also this caused the crash when trying to request deletion of many assets in quick succession. While sending deletion requests, the Prometheus metrics show a queue building up that at some point (for me after ±2000 assets) explodes dramatically, quickly followed by the crash if the deletion requests are not stopped. Even if done so, it takes quite a while until processing is completed. If only smaller bursts of deletion requests were sent, processing was overproportionally faster. So only sending bursts of 2k deletion requests and waiting for the processing to be completed seemed to work. Even better, by rate-limiting the deletion requests, I could achieve a steady deletion process without queue escalation. By closely monitoring the metrics, I figured that the max rate-limit for my server was around two pictures per second.
[6] https://immich.app/docs/api/delete-assets

As this would have taken several days to complete, I just rolled back a backup which solved my problem.

@daniel-freiermuth commented on GitHub (Apr 27, 2025): Update from my side: I also tried using scripts [1, 2] that use the API for deleting the offline files, without luck. It turns out that they use the report endpoint [3] for retrieving the list of offline files. Unfortunately, just requesting this endpoint already triggers the crash. [1] https://github.com/Thoroslives/immich_remove_offline_files/blob/main/immich_remove_offline_files.py , seemingly updated variant of [2] https://github.com/clumsyCoder00/Immich-Tools/blob/main/(deprecated)%20immich-del-offline.py [3] https://immich.app/docs/api/get-audit-files/ In contrast, the web interface is able to show the offline files. Turns out, it uses different endpoints [4, 5] which do not trigger the crash. [4] https://immich.app/docs/api/get-time-buckets [5] https://immich.app/docs/api/get-time-bucket I amended the script from above using this endpoint. The script then requests all files to be deleted separately via the asset deletion endpoint [6]. Also this caused the crash when trying to request deletion of many assets in quick succession. While sending deletion requests, the Prometheus metrics show a queue building up that at some point (for me after ±2000 assets) explodes dramatically, quickly followed by the crash if the deletion requests are not stopped. Even if done so, it takes quite a while until processing is completed. If only smaller bursts of deletion requests were sent, processing was overproportionally faster. So only sending bursts of 2k deletion requests and waiting for the processing to be completed seemed to work. Even better, by rate-limiting the deletion requests, I could achieve a steady deletion process without queue escalation. By closely monitoring the metrics, I figured that the max rate-limit for my server was around two pictures per second. [6] https://immich.app/docs/api/delete-assets As this would have taken several days to complete, I just rolled back a backup which solved my problem.
OVERLORD commented 2026-02-05 11:28:46 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Sep 18, 2025):

There were some recent weird issues around trash that were the result of background jobs being paused. On the latest version, if you go to the admin jobs panel there will be a button in the top right if there are any paused jobs. Is that showing for any of you, and if so does hitting it to unpause solve this issue?

@bo0tzz commented on GitHub (Sep 18, 2025): There were some recent weird issues around trash that were the result of background jobs being paused. On the latest version, if you go to the admin jobs panel there will be a button in the top right if there are any paused jobs. Is that showing for any of you, and if so does hitting it to unpause solve this issue?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label 📱mobile
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#5459
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?