immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Export of large album incorrect #4728

New Issue
Closed
opened 2026-02-05 10:49:25 +03:00 by OVERLORD · 9 comments
OVERLORD commented 2026-02-05 10:49:25 +03:00
Owner
Copy Link

Originally created by @plord12 on GitHub (Nov 10, 2024).

The bug

I have an album of 2952 images, when I export the album I get :

  • Scanned+1-20241110_081518.zip - 2890 files
  • Scanned+2-20241110_081608.zip - 2890 files
  • Scanned+3-20241110_081652.zip - 49 files

So 5829 files

The first two zips seem to be duplicates ( although a few files exported in a different order in the zip ).

If I unzip them all and ignore duplicate filenames I end up with 2892 images

The OS that Immich Server is running on

Debian

Version of Immich Server

c1.120.1

Version of Immich Mobile App

n/a

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml
      service: rkmpp # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:2283
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-armnn
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: armnn # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

IMMICH_LOG_LEVEL=verbose

Reproduction steps

  1. Create a large album
  2. Export
  3. Check for missing files / duplicates

Relevant log output

No response

Additional information

No response

Originally created by @plord12 on GitHub (Nov 10, 2024). ### The bug I have an album of 2952 images, when I export the album I get : * Scanned+1-20241110_081518.zip - 2890 files * Scanned+2-20241110_081608.zip - 2890 files * Scanned+3-20241110_081652.zip - 49 files So 5829 files The first two zips seem to be duplicates ( although a few files exported in a different order in the zip ). If I unzip them all and ignore duplicate filenames I end up with 2892 images ### The OS that Immich Server is running on Debian ### Version of Immich Server c1.120.1 ### Version of Immich Mobile App n/a ### Platform with the issue - [ ] Server - [X] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} extends: file: hwaccel.transcoding.yml service: rkmpp # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:2283 depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-armnn extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration file: hwaccel.ml.yml service: armnn # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache env_file: - .env restart: always redis: container_name: immich_redis image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b healthcheck: test: redis-cli ping || exit 1 restart: always database: container_name: immich_postgres image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' volumes: - ${DB_DATA_LOCATION}:/var/lib/postgresql/data healthcheck: test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 interval: 5m start_interval: 30s start_period: 5m command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"] restart: always volumes: model-cache: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=./library # The location where your database files are stored DB_DATA_LOCATION=./postgres # To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List # TZ=Etc/UTC # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secret for postgres. You should change it to a random password DB_PASSWORD=postgres # The values below this line do not need to be changed ################################################################################### DB_USERNAME=postgres DB_DATABASE_NAME=immich IMMICH_LOG_LEVEL=verbose ``` ### Reproduction steps 1. Create a large album 2. Export 3. Check for missing files / duplicates ### Relevant log output _No response_ ### Additional information _No response_
OVERLORD commented 2026-02-05 10:49:28 +03:00
Author
Owner
Copy Link

@CrushedAsian255 commented on GitHub (Nov 11, 2024):

Does this happen with all albums?

@CrushedAsian255 commented on GitHub (Nov 11, 2024): Does this happen with all albums?
OVERLORD commented 2026-02-05 10:49:29 +03:00
Author
Owner
Copy Link

@plord12 commented on GitHub (Nov 11, 2024):

Does this happen with all albums?

I can't tell ... I've only one album thats big and requires more than one zip file when exporting.

Other albums, that use only one zip, are working as expected.

@plord12 commented on GitHub (Nov 11, 2024): > Does this happen with all albums? I can't tell ... I've only one album thats big and requires more than one zip file when exporting. Other albums, that use only one zip, are working as expected.
OVERLORD commented 2026-02-05 10:49:30 +03:00
Author
Owner
Copy Link

@hobias commented on GitHub (Mar 22, 2025):

I can confirm this bug. I have an album with 1440 images and the export is also garbage.
There are three zip files created where the first two seem duplicates and possibly corrupt (ERRORS:
Unexpected end of archive) . From the requested 1440 images I could extract around 850 from the three archives.

@hobias commented on GitHub (Mar 22, 2025): I can confirm this bug. I have an album with 1440 images and the export is also garbage. There are three zip files created where the first two seem duplicates and possibly corrupt (ERRORS: Unexpected end of archive) . From the requested 1440 images I could extract around 850 from the three archives.
OVERLORD commented 2026-02-05 10:49:31 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Aug 27, 2025):

I just tested and could not reproduce this. Let me know if it is still happening.

@jrasm91 commented on GitHub (Aug 27, 2025): I just tested and could not reproduce this. Let me know if it is still happening.
OVERLORD commented 2026-02-05 10:49:32 +03:00
Author
Owner
Copy Link

@plord12 commented on GitHub (Aug 28, 2025):

Well, I've just repeated the exercise.

I see -

  • Scanned+1-20250827_225104.zip - 2917 files
  • Scanned+2-20250827_225315.zip - 123 files

total 3040. The immich album reports 3053 items.

If I unzip the first (on my mac) into an empty directory I see -

$ unzip ../Scanned+1-20250827_225104.zip 
Archive:  ../Scanned+1-20250827_225104.zip
 extracting: PICT0008.JPG            
 extracting: PICT0011.JPG            
 extracting: PICT0016.JPG  
....
 extracting: 109-0923_IMG.JPG        
 extracting: pict0094.jpg            
replace pict0008.jpg? [y]es, [n]o, [A]ll, [N]one, [r]ename:

This works on linux .. so I presume the "cause" is the mac case insensitive filesystem.

Even so, on linux when I extract the second file I see -

$ unzip /tmp/Scanned+2-20250827_225315.zip 
Archive:  /tmp/Scanned+2-20250827_225315.zip
replace PICT0068.JPG? [y]es, [n]o, [A]ll, [N]one, [r]ename:

The 2 zips contain duplicates - total unique files is 2917.

So 3 issues -

  1. The number of files in the zip doesn't match the number of files in the album
  2. Extracting on mac results in duplicate files
  3. Second archive consists of duplicates of the first file
@plord12 commented on GitHub (Aug 28, 2025): Well, I've just repeated the exercise. I see - - Scanned+1-20250827_225104.zip - 2917 files - Scanned+2-20250827_225315.zip - 123 files total 3040. The immich album reports 3053 items. If I unzip the first (on my mac) into an empty directory I see - ``` $ unzip ../Scanned+1-20250827_225104.zip Archive: ../Scanned+1-20250827_225104.zip extracting: PICT0008.JPG extracting: PICT0011.JPG extracting: PICT0016.JPG .... extracting: 109-0923_IMG.JPG extracting: pict0094.jpg replace pict0008.jpg? [y]es, [n]o, [A]ll, [N]one, [r]ename: ``` This works on linux .. so I presume the "cause" is the mac case insensitive filesystem. Even so, on linux when I extract the second file I see - ``` $ unzip /tmp/Scanned+2-20250827_225315.zip Archive: /tmp/Scanned+2-20250827_225315.zip replace PICT0068.JPG? [y]es, [n]o, [A]ll, [N]one, [r]ename: ``` The 2 zips contain duplicates - total unique files is 2917. So 3 issues - 1. The number of files in the zip doesn't match the number of files in the album 2. Extracting on mac results in duplicate files 3. Second archive consists of duplicates of the first file
OVERLORD commented 2026-02-05 10:49:34 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Aug 28, 2025):

Is this just an issue on mac because of case insensitivity with file names? Can you reproduce it on https://demo.immich.app/?

@jrasm91 commented on GitHub (Aug 28, 2025): Is this just an issue on mac because of case insensitivity with file names? Can you reproduce it on https://demo.immich.app/?
OVERLORD commented 2026-02-05 10:49:34 +03:00
Author
Owner
Copy Link

@plord12 commented on GitHub (Aug 28, 2025):

When I unzip on linux I loose files, so only one part of this is mac specific.

I can't export my album and import on the demo server since export doesn't work. Maybe I can think of a way to create a few 1,000 test images to re-create this.

@plord12 commented on GitHub (Aug 28, 2025): When I unzip on linux I loose files, so only one part of this is mac specific. I can't export my album and import on the demo server since export doesn't work. Maybe I can think of a way to create a few 1,000 test images to re-create this.
OVERLORD commented 2026-02-05 10:49:35 +03:00
Author
Owner
Copy Link

@plord12 commented on GitHub (Aug 28, 2025):

I think the issue is that my album contains images with duplicate names -

Image

When I look at the download zips I see, for example -

$ unzip -l Scanned+1-20250827_225104.zip | grep -i pict0008.jpg
  1715605  10-27-2024 08:18   PICT0008.JPG
   672710  09-18-2016 20:22   pict0008.jpg
$ unzip -l Scanned+2-20250827_225315.zip | grep -i pict0008.jpg 
  1487473  01-08-2012 21:27   PICT0008.JPG

If I try and replicate my album, I can upload different files with the same filename but they get exported as different files (+1 added).

Perhaps the upload worked differently when I created my album ?

@plord12 commented on GitHub (Aug 28, 2025): I think the issue is that my album contains images with duplicate names - <img width="760" height="269" alt="Image" src="https://github.com/user-attachments/assets/7671bf7b-1686-423a-836d-7a85d00682cb" /> When I look at the download zips I see, for example - ``` $ unzip -l Scanned+1-20250827_225104.zip | grep -i pict0008.jpg 1715605 10-27-2024 08:18 PICT0008.JPG 672710 09-18-2016 20:22 pict0008.jpg $ unzip -l Scanned+2-20250827_225315.zip | grep -i pict0008.jpg 1487473 01-08-2012 21:27 PICT0008.JPG ``` If I try and replicate my album, I can upload different files with the same filename but they get exported as different files (+1 added). Perhaps the upload worked differently when I created my album ?
OVERLORD commented 2026-02-05 10:49:36 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Aug 28, 2025):

The next thing to try is probably two files with the same name but different casing and then download them from an album.

@jrasm91 commented on GitHub (Aug 28, 2025): The next thing to try is probably two files with the same name but different casing and then download them from an album.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#4728
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?