immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Inefficient Thumbnail generation for external libraries #3569

New Issue
Closed
opened 2026-02-05 09:04:43 +03:00 by OVERLORD · 0 comments
OVERLORD commented 2026-02-05 09:04:43 +03:00
Owner
Copy Link

Originally created by @NickIAm on GitHub (Jun 19, 2024).

The bug

When generating thumbnails for external libraries, each files is read for the first task of thumbnail generation, then read again for the second task. I had a large external library on a rclone mount that was re-imported and caused over 10x the data egress on the dataset, and it didn't even finish generating the thumbnails. It was about 16,000 files and 110GB data that used 900 GB of egress while generating thumbnails. My work around was to upload the files using immich-go instead of importing them as external libraries.

The OS that Immich Server is running on

Debian 12

Version of Immich Server

v1.106.4

Version of Immich Mobile App

v1.106.3

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

# My Version
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${IMPORTED_LIBRARIES}:/mnt/libraries:ro
      - ${ENCODED_VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
      - ${THUMB_LOCATION}:/usr/src/app/upload/thumbs
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
      - immich-machine-learning
    restart: always


  immich-machine-learning:
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always


  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always


  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]

  backup:
    container_name: immich_db_dumper
    image: prodrigestivill/postgres-backup-local:14
    env_file:
      - stack.env
    environment:
      POSTGRES_HOST: database
      POSTGRES_CLUSTER: 'TRUE'
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      SCHEDULE: "@daily"
      POSTGRES_EXTRA_OPTS: '--clean --if-exists'
      BACKUP_DIR: /db_dumps
    volumes:
      - /home/nick/immich/dba_dumps:/db_dumps
    depends_on:
      - database

volumes:
  pgdata:
  model-cache:

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=immich
DB_PASSWORD=<redacted>
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/mnt/immich/upload/
TYPESENSE_API_KEY=<redacted>
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-server:3002
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immichml:3003
IMPORTED_LIBRARIES=/mnt/external_libraries
ENCODED_VIDEO_LOCATION=/home/nick/immich/cache/encoded_video
THUMB_LOCATION=/home/nick/immich/cache/thumbs

Reproduction steps

1. Mount a large image collection using rclone mount
2. Add a volume mount to the docker-compose and import it as an external library
3. Watch your cloud provider dashboard as the thumbnails are generated. The egress be much bigger than the original size of the images.
...

Relevant log output

No response

Additional information

I have the rclone mount vfs cache set to a max size of 30GB as I run immich on a VPS with limited storage. I stopped it before it was finished after I noticed the huge egress. I have the files stored in storj.io using the rclone native remote to access it.

Originally created by @NickIAm on GitHub (Jun 19, 2024). ### The bug When generating thumbnails for external libraries, each files is read for the first task of thumbnail generation, then read again for the second task. I had a large external library on a rclone mount that was re-imported and caused over 10x the data egress on the dataset, and it didn't even finish generating the thumbnails. It was about 16,000 files and 110GB data that used 900 GB of egress while generating thumbnails. My work around was to upload the files using immich-go instead of importing them as external libraries. ### The OS that Immich Server is running on Debian 12 ### Version of Immich Server v1.106.4 ### Version of Immich Mobile App v1.106.3 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" # My Version # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - ${IMPORTED_LIBRARIES}:/mnt/libraries:ro - ${ENCODED_VIDEO_LOCATION}:/usr/src/app/upload/encoded-video - ${THUMB_LOCATION}:/usr/src/app/upload/thumbs - /etc/localtime:/etc/localtime:ro env_file: - stack.env ports: - 2283:3001 depends_on: - redis - database - immich-machine-learning restart: always immich-machine-learning: # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache env_file: - stack.env restart: always redis: container_name: immich_redis image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672 restart: always database: container_name: immich_postgres image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 env_file: - stack.env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"] backup: container_name: immich_db_dumper image: prodrigestivill/postgres-backup-local:14 env_file: - stack.env environment: POSTGRES_HOST: database POSTGRES_CLUSTER: 'TRUE' POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_USER: ${DB_USERNAME} POSTGRES_PASSWORD: ${DB_PASSWORD} SCHEDULE: "@daily" POSTGRES_EXTRA_OPTS: '--clean --if-exists' BACKUP_DIR: /db_dumps volumes: - /home/nick/immich/dba_dumps:/db_dumps depends_on: - database volumes: pgdata: model-cache: ``` ### Your .env content ```Shell DB_HOSTNAME=immich_postgres DB_USERNAME=immich DB_PASSWORD=<redacted> DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis UPLOAD_LOCATION=/mnt/immich/upload/ TYPESENSE_API_KEY=<redacted> PUBLIC_LOGIN_PAGE_MESSAGE= IMMICH_WEB_URL=http://immich-server:3002 IMMICH_SERVER_URL=http://immich-server:3001 IMMICH_MACHINE_LEARNING_URL=http://immichml:3003 IMPORTED_LIBRARIES=/mnt/external_libraries ENCODED_VIDEO_LOCATION=/home/nick/immich/cache/encoded_video THUMB_LOCATION=/home/nick/immich/cache/thumbs ``` ### Reproduction steps ```bash 1. Mount a large image collection using rclone mount 2. Add a volume mount to the docker-compose and import it as an external library 3. Watch your cloud provider dashboard as the thumbnails are generated. The egress be much bigger than the original size of the images. ... ``` ### Relevant log output _No response_ ### Additional information I have the rclone mount vfs cache set to a max size of 30GB as I run immich on a VPS with limited storage. I stopped it before it was finished after I noticed the huge egress. I have the files stored in storj.io using the rclone native remote to access it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#3569
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?