immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

De-Duplicate utility does not detect identical images #4374

New Issue
Closed
opened 2026-02-05 10:16:02 +03:00 by OVERLORD · 2 comments
OVERLORD commented 2026-02-05 10:16:02 +03:00
Owner
Copy Link

Originally created by @AFumi39 on GitHub (Sep 20, 2024).

The bug

I noticed some issues with the duplicate detector, but in particular there's a bug that prevent some images to be recognized as duplicates. I tried running duplicate detector again and again, but these images never been detected as duplicates even if they are identical. See the attachments as example
immich-20240920_185859.zip
immich-20240920_185841.zip

The OS that Immich Server is running on

Docker

Version of Immich Server

V1.115.0

Version of Immich Mobile App

1.115.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload:rw
      - /etc/localtime:/etc/localtime:ro
      - /volume1/docker/immich-app/ext-library:/mnt/media/ext-library:rw
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false
    labels:
      - 'wud.tag.include=v^([0-9]\d*)\.([0-9]\d*)\.([0-9]\d*)$$'
      - 'wud.watch.digest=true'
      - 'wud.link.template=https://github.com/immich-app/immich/releases/tag/$${raw}'
      - 'wud.display.name=Immich server'
      - 'wud.display.icon=mdi:image-multiple-outline'

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache:rw
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false
    labels:
      - 'wud.watch=false'

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
    labels:
      - 'wud.watch=false'

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data:rw
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      #start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always
    labels:
      - 'wud.watch=false'

  backup:
    container_name: immich_db_dumper
    image: prodrigestivill/postgres-backup-local:14
    restart: always
    env_file:
      - stack.env
    environment:
      POSTGRES_HOST: database
      POSTGRES_CLUSTER: 'TRUE'
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      SCHEDULE: "@daily"
      POSTGRES_EXTRA_OPTS: '--clean --if-exists'
      BACKUP_DIR: /db_dumps
    volumes:
      - /volume1/backup/immich_db_backup:/db_dumps:rw
    depends_on:
      - database
    labels:
      - 'wud.watch=false'

volumes:
  model-cache:

Your .env content

UPLOAD_LOCATION=/volume1/docker/immich-app/library
DB_DATA_LOCATION=/volume1/docker/immich-app/postgres
TZ=Europe/Rome
IMMICH_VERSION=v1.115.0
DB_PASSWORD=***
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

...

Relevant log output

No response

Additional information

No response

Originally created by @AFumi39 on GitHub (Sep 20, 2024). ### The bug I noticed some issues with the duplicate detector, but in particular there's a bug that prevent some images to be recognized as duplicates. I tried running duplicate detector again and again, but these images never been detected as duplicates even if they are identical. See the attachments as example [immich-20240920_185859.zip](https://github.com/user-attachments/files/17078394/immich-20240920_185859.zip) [immich-20240920_185841.zip](https://github.com/user-attachments/files/17078395/immich-20240920_185841.zip) ### The OS that Immich Server is running on Docker ### Version of Immich Server V1.115.0 ### Version of Immich Mobile App 1.115.0 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION} # extends: # file: hwaccel.transcoding.yml # service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/usr/src/app/upload:rw - /etc/localtime:/etc/localtime:ro - /volume1/docker/immich-app/ext-library:/mnt/media/ext-library:rw env_file: - stack.env ports: - 2283:3001 depends_on: - redis - database restart: always healthcheck: disable: false labels: - 'wud.tag.include=v^([0-9]\d*)\.([0-9]\d*)\.([0-9]\d*)$$' - 'wud.watch.digest=true' - 'wud.link.template=https://github.com/immich-app/immich/releases/tag/$${raw}' - 'wud.display.name=Immich server' - 'wud.display.icon=mdi:image-multiple-outline' immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache:rw env_file: - stack.env restart: always healthcheck: disable: false labels: - 'wud.watch=false' redis: container_name: immich_redis image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e healthcheck: test: redis-cli ping || exit 1 restart: always labels: - 'wud.watch=false' database: container_name: immich_postgres image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data:rw healthcheck: test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 interval: 5m #start_interval: 30s start_period: 5m command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"] restart: always labels: - 'wud.watch=false' backup: container_name: immich_db_dumper image: prodrigestivill/postgres-backup-local:14 restart: always env_file: - stack.env environment: POSTGRES_HOST: database POSTGRES_CLUSTER: 'TRUE' POSTGRES_USER: ${DB_USERNAME} POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_DB: ${DB_DATABASE_NAME} SCHEDULE: "@daily" POSTGRES_EXTRA_OPTS: '--clean --if-exists' BACKUP_DIR: /db_dumps volumes: - /volume1/backup/immich_db_backup:/db_dumps:rw depends_on: - database labels: - 'wud.watch=false' volumes: model-cache: ``` ### Your .env content ```Shell UPLOAD_LOCATION=/volume1/docker/immich-app/library DB_DATA_LOCATION=/volume1/docker/immich-app/postgres TZ=Europe/Rome IMMICH_VERSION=v1.115.0 DB_PASSWORD=*** DB_USERNAME=postgres DB_DATABASE_NAME=immich ``` ### Reproduction steps 1. 2. 3. ... ### Relevant log output _No response_ ### Additional information _No response_
OVERLORD commented 2026-02-05 10:16:09 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Sep 20, 2024):

Are there any errors in the log when you run the job?

@bo0tzz commented on GitHub (Sep 20, 2024): Are there any errors in the log when you run the job?
OVERLORD commented 2026-02-05 10:16:12 +03:00
Author
Owner
Copy Link

@AFumi39 commented on GitHub (Sep 20, 2024):

Are there any errors in the log when you run the job?

Only these warnings

[Nest] 7  - 09/20/2024, 7:42:58 PM    WARN [Microservices:DuplicateService] Asset 3ce1360b-603f-47dc-876c-3adcffdd4b59 is missing preview image
[Nest] 7  - 09/20/2024, 8:00:15 PM    WARN [Microservices:DuplicateService] Asset 53334656-c9ce-4551-ba33-cf7322dd309b is missing preview image
[Nest] 7  - 09/20/2024, 8:02:00 PM    WARN [Microservices:DuplicateService] Asset de7117b7-dde2-47bd-8469-988608be4d9a is missing preview image
[Nest] 7  - 09/20/2024, 8:02:19 PM    WARN [Microservices:DuplicateService] Asset c78fdaf5-9cdd-4ccc-995d-344e1c92a53b is missing preview image
[Nest] 7  - 09/20/2024, 8:02:38 PM    WARN [Microservices:DuplicateService] Asset ac5e8c19-47d7-46a4-99ed-6292e92ca773 is missing preview image
[Nest] 7  - 09/20/2024, 8:03:13 PM    WARN [Microservices:DuplicateService] Asset 8e7def39-1131-4075-8a9f-3a0a45942c77 is missing preview image
[Nest] 7  - 09/20/2024, 8:03:29 PM    WARN [Microservices:DuplicateService] Asset 8f2b63b9-6c29-40d2-be58-393f914ff67e is missing preview image
[Nest] 7  - 09/20/2024, 8:03:30 PM    WARN [Microservices:DuplicateService] Asset d70a27a3-ed5a-480c-9453-9062a9f7052e is missing preview image
[Nest] 7  - 09/20/2024, 8:03:32 PM    WARN [Microservices:DuplicateService] Asset d70f2c6d-9939-436c-ae88-13f674169d02 is missing preview image
[Nest] 7  - 09/20/2024, 8:03:32 PM    WARN [Microservices:DuplicateService] Asset bb53b8da-88ae-4ac4-a8c2-5a3c30acddad is missing preview image
[Nest] 7  - 09/20/2024, 8:04:05 PM    WARN [Microservices:DuplicateService] Asset 9ae53060-75ae-4ece-b7aa-2a077ba81adb is missing preview image
@AFumi39 commented on GitHub (Sep 20, 2024): > Are there any errors in the log when you run the job? Only these warnings ``` [Nest] 7 - 09/20/2024, 7:42:58 PM WARN [Microservices:DuplicateService] Asset 3ce1360b-603f-47dc-876c-3adcffdd4b59 is missing preview image [Nest] 7 - 09/20/2024, 8:00:15 PM WARN [Microservices:DuplicateService] Asset 53334656-c9ce-4551-ba33-cf7322dd309b is missing preview image [Nest] 7 - 09/20/2024, 8:02:00 PM WARN [Microservices:DuplicateService] Asset de7117b7-dde2-47bd-8469-988608be4d9a is missing preview image [Nest] 7 - 09/20/2024, 8:02:19 PM WARN [Microservices:DuplicateService] Asset c78fdaf5-9cdd-4ccc-995d-344e1c92a53b is missing preview image [Nest] 7 - 09/20/2024, 8:02:38 PM WARN [Microservices:DuplicateService] Asset ac5e8c19-47d7-46a4-99ed-6292e92ca773 is missing preview image [Nest] 7 - 09/20/2024, 8:03:13 PM WARN [Microservices:DuplicateService] Asset 8e7def39-1131-4075-8a9f-3a0a45942c77 is missing preview image [Nest] 7 - 09/20/2024, 8:03:29 PM WARN [Microservices:DuplicateService] Asset 8f2b63b9-6c29-40d2-be58-393f914ff67e is missing preview image [Nest] 7 - 09/20/2024, 8:03:30 PM WARN [Microservices:DuplicateService] Asset d70a27a3-ed5a-480c-9453-9062a9f7052e is missing preview image [Nest] 7 - 09/20/2024, 8:03:32 PM WARN [Microservices:DuplicateService] Asset d70f2c6d-9939-436c-ae88-13f674169d02 is missing preview image [Nest] 7 - 09/20/2024, 8:03:32 PM WARN [Microservices:DuplicateService] Asset bb53b8da-88ae-4ac4-a8c2-5a3c30acddad is missing preview image [Nest] 7 - 09/20/2024, 8:04:05 PM WARN [Microservices:DuplicateService] Asset 9ae53060-75ae-4ece-b7aa-2a077ba81adb is missing preview image ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#4374
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?