immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-25 03:12:21 +03:00
Code Issues 783 Packages Projects Releases 12 Wiki Activity

Microservices error - Possible Duplicates causing Duplicate Key Value? #3383

New Issue
Closed
opened 2026-02-05 08:14:14 +03:00 by OVERLORD · 2 comments
OVERLORD commented 2026-02-05 08:14:14 +03:00
Owner
Copy Link

Originally created by @undaunt on GitHub (May 31, 2024).

The bug

I previously uploaded my iCloud library via bulk API and have been keeping it updated within Immich via the mobile app. I expand on this use case here (https://github.com/immich-app/immich/discussions/9899) but I'm seeing a lot of errors right now while my external library copy of my content is being scanned into Immich.

I'm wondering if this is because some of the exact same images already exist in the upload library, but since it's a separate library Immich doesn't like the potential hash collisions, but its not auto excluded the same as if I had two of the same photo and the mobile app was scanning for upload? Just throwing out possible ideas, I'm not sure.

The OS that Immich Server is running on

Ubuntu 24.04 LTS

Version of Immich Server

v1.105.1

Version of Immich Mobile App

N/A

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

services:
  immich-server:
    container_name: immich-server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    restart: unless-stopped
    networks:
      - ${PROXY_NETWORK}
      - ${APP_NETWORK}
    depends_on:
      - immich-redis
      - immich-db
    command: [ "start.sh", "immich" ]
    volumes:
      - ${PHOTOSDIR}/immich:/usr/src/app/upload
      - ${PHOTOSDIR}/icloud:/usr/src/app/external
      - /etc/localtime:/etc/localtime:ro
    environment:
      DB_HOSTNAME: ${POSTGRES_IMMICH_HOST}
      DB_DATABASE_NAME: ${POSTGRES_IMMICH_DB}
      DB_USERNAME: ${POSTGRES_IMMICH_USER}
      DB_PASSWORD: ${POSTGRES_IMMICH_PASS}
      REDIS_HOSTNAME: ${IMMICH_REDIS_HOSTNAME}

  immich-microservices:
    container_name: immich-microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    restart: unless-stopped
    networks:
      - ${APP_NETWORK}
    depends_on:
      - immich-redis
      - immich-db
    cpus: 3
    devices:
      - /dev/dri:/dev/dri
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${PHOTOSDIR}/immich:/usr/src/app/upload
      - ${PHOTOSDIR}/icloud:/usr/src/app/external
      - /etc/localtime:/etc/localtime:ro
    environment:
      DB_HOSTNAME: ${POSTGRES_IMMICH_HOST}
      DB_DATABASE_NAME: ${POSTGRES_IMMICH_DB}
      DB_USERNAME: ${POSTGRES_IMMICH_USER}
      DB_PASSWORD: ${POSTGRES_IMMICH_PASS}
      REDIS_HOSTNAME: ${IMMICH_REDIS_HOSTNAME}

  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    container_name: immich-machine-learning
    volumes:
      - ${APPDATA}/photos/immich/models:/cache
    restart: unless-stopped
    networks:
      - ${APP_NETWORK}

  immich-redis:
    image: redis:6.2-alpine@sha256:b6124ab2e45cc332e16398022a411d7e37181f21ff7874835e0180f56a09e82a
    container_name: immich-redis
    restart: unless-stopped
    networks:
      - ${APP_NETWORK}
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

  immich-db:
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    container_name: immich-db
    restart: unless-stopped
    networks:
      - ${APP_NETWORK}
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_IMMICH_PASS}
      POSTGRES_USER: ${POSTGRES_IMMICH_USER}
      POSTGRES_DB: ${POSTGRES_IMMICH_DB}
    volumes:
      - ${APPDATA}/photos/immich/db:/var/lib/postgresql/data

Your .env content

See above

Reproduction steps

1. Import iCloud content via API or mobile app to upload library
2. Add external library with copy of same content
3. Scan external library

Relevant log output

05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] ERROR:  duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] DETAIL:  Key ("livePhotoVideoId")=(14cee637-f8e9-4406-ba16-40674d08849f) already exists.
05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] STATEMENT:  UPDATE "assets" SET "id" = $1, "livePhotoVideoId" = $2, "updatedAt" = CURRENT_TIMESTAMP WHERE "id" IN ($3)
05/30/2024 02:31:31 PM 2024-05-30 21:31:31.957 UTC [446] ERROR:  duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
05/30/2024 2:31:31 PM 2024-05-30 21:31:31.957 UTC [446] DETAIL:  Key ("livePhotoVideoId")=(18f20df4-9819-499c-9d26-e0fde703fc5d) already exists.
05/30/2024 02:31:31 PM 2024-05-30 21:31:31.957 UTC [446] STATEMENT:  UPDATE "assets" SET "id" = $1, "livePhotoVideoId" = $2, "updatedAt" = CURRENT_TIMESTAMP WHERE "id" IN ($3)

LOG [ImmichMicroservices] [MediaService] Successfully generated JPEG image preview for asset 879e2dbd-a63e-4c3f-97d4-b5fd017b2ad2
2024-05-30T21:23:46.601505859Z [Nest] 7  - 05/30/2024, 9:23:46 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated JPEG image preview for asset e073dc5b-bd13-4e05-aa88-8bb3a1185294
2024-05-30T21:23:46.820207260Z [Nest] 7  - 05/30/2024, 9:23:46 PM   ERROR [ImmichMicroservices] [JobService] Unable to run job handler (metadataExtraction/link-live-photos): QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
2024-05-30T21:23:46.820328158Z [Nest] 7  - 05/30/2024, 9:23:46 PM   ERROR [ImmichMicroservices] [JobService] QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
2024-05-30T21:23:46.820342819Z     at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
2024-05-30T21:23:46.820347411Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-05-30T21:23:46.820352164Z     at async UpdateQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/UpdateQueryBuilder.js:83:33)
2024-05-30T21:23:46.820356296Z     at async AssetRepository.update (/usr/src/app/dist/repositories/asset.repository.js:183:9)
2024-05-30T21:23:46.820360399Z     at async MetadataService.handleLivePhotoLinking (/usr/src/app/dist/services/metadata.service.js:155:9)
2024-05-30T21:23:46.820364637Z     at async /usr/src/app/dist/services/job.service.js:145:36
2024-05-30T21:23:46.820368572Z     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
2024-05-30T21:23:46.820372526Z     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
2024-05-30T21:23:46.820376565Z [Nest] 7  - 05/30/2024, 9:23:46 PM   ERROR [ImmichMicroservices] [JobService] Object:
2024-05-30T21:23:46.820382675Z {
2024-05-30T21:23:46.820386810Z   "id": "686ee129-e364-4616-b14d-0e09fad7b987",
2024-05-30T21:23:46.820391084Z   "source": "upload"
2024-05-30T21:23:46.820395487Z }

Additional information

No response

Originally created by @undaunt on GitHub (May 31, 2024). ### The bug I previously uploaded my iCloud library via bulk API and have been keeping it updated within Immich via the mobile app. I expand on this use case here (https://github.com/immich-app/immich/discussions/9899) but I'm seeing a lot of errors right now while my external library copy of my content is being scanned into Immich. I'm wondering if this is because some of the exact same images already exist in the upload library, but since it's a separate library Immich doesn't like the potential hash collisions, but its not auto excluded the same as if I had two of the same photo and the mobile app was scanning for upload? Just throwing out possible ideas, I'm not sure. ### The OS that Immich Server is running on Ubuntu 24.04 LTS ### Version of Immich Server v1.105.1 ### Version of Immich Mobile App N/A ### Platform with the issue - [X] Server - [X] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML services: immich-server: container_name: immich-server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} restart: unless-stopped networks: - ${PROXY_NETWORK} - ${APP_NETWORK} depends_on: - immich-redis - immich-db command: [ "start.sh", "immich" ] volumes: - ${PHOTOSDIR}/immich:/usr/src/app/upload - ${PHOTOSDIR}/icloud:/usr/src/app/external - /etc/localtime:/etc/localtime:ro environment: DB_HOSTNAME: ${POSTGRES_IMMICH_HOST} DB_DATABASE_NAME: ${POSTGRES_IMMICH_DB} DB_USERNAME: ${POSTGRES_IMMICH_USER} DB_PASSWORD: ${POSTGRES_IMMICH_PASS} REDIS_HOSTNAME: ${IMMICH_REDIS_HOSTNAME} immich-microservices: container_name: immich-microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} restart: unless-stopped networks: - ${APP_NETWORK} depends_on: - immich-redis - immich-db cpus: 3 devices: - /dev/dri:/dev/dri command: [ "start.sh", "microservices" ] volumes: - ${PHOTOSDIR}/immich:/usr/src/app/upload - ${PHOTOSDIR}/icloud:/usr/src/app/external - /etc/localtime:/etc/localtime:ro environment: DB_HOSTNAME: ${POSTGRES_IMMICH_HOST} DB_DATABASE_NAME: ${POSTGRES_IMMICH_DB} DB_USERNAME: ${POSTGRES_IMMICH_USER} DB_PASSWORD: ${POSTGRES_IMMICH_PASS} REDIS_HOSTNAME: ${IMMICH_REDIS_HOSTNAME} immich-machine-learning: image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} container_name: immich-machine-learning volumes: - ${APPDATA}/photos/immich/models:/cache restart: unless-stopped networks: - ${APP_NETWORK} immich-redis: image: redis:6.2-alpine@sha256:b6124ab2e45cc332e16398022a411d7e37181f21ff7874835e0180f56a09e82a container_name: immich-redis restart: unless-stopped networks: - ${APP_NETWORK} healthcheck: test: ["CMD-SHELL", "redis-cli ping | grep PONG"] start_period: 20s interval: 30s retries: 5 timeout: 3s volumes: - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro immich-db: image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 container_name: immich-db restart: unless-stopped networks: - ${APP_NETWORK} environment: POSTGRES_PASSWORD: ${POSTGRES_IMMICH_PASS} POSTGRES_USER: ${POSTGRES_IMMICH_USER} POSTGRES_DB: ${POSTGRES_IMMICH_DB} volumes: - ${APPDATA}/photos/immich/db:/var/lib/postgresql/data ``` ### Your .env content ```Shell See above ``` ### Reproduction steps ```bash 1. Import iCloud content via API or mobile app to upload library 2. Add external library with copy of same content 3. Scan external library ``` ### Relevant log output ``` 05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] ERROR: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" 05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] DETAIL: Key ("livePhotoVideoId")=(14cee637-f8e9-4406-ba16-40674d08849f) already exists. 05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] STATEMENT: UPDATE "assets" SET "id" = $1, "livePhotoVideoId" = $2, "updatedAt" = CURRENT_TIMESTAMP WHERE "id" IN ($3) 05/30/2024 02:31:31 PM 2024-05-30 21:31:31.957 UTC [446] ERROR: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" 05/30/2024 2:31:31 PM 2024-05-30 21:31:31.957 UTC [446] DETAIL: Key ("livePhotoVideoId")=(18f20df4-9819-499c-9d26-e0fde703fc5d) already exists. 05/30/2024 02:31:31 PM 2024-05-30 21:31:31.957 UTC [446] STATEMENT: UPDATE "assets" SET "id" = $1, "livePhotoVideoId" = $2, "updatedAt" = CURRENT_TIMESTAMP WHERE "id" IN ($3) ``` ```shell LOG [ImmichMicroservices] [MediaService] Successfully generated JPEG image preview for asset 879e2dbd-a63e-4c3f-97d4-b5fd017b2ad2 2024-05-30T21:23:46.601505859Z [Nest] 7 - 05/30/2024, 9:23:46 PM LOG [ImmichMicroservices] [MediaService] Successfully generated JPEG image preview for asset e073dc5b-bd13-4e05-aa88-8bb3a1185294 2024-05-30T21:23:46.820207260Z [Nest] 7 - 05/30/2024, 9:23:46 PM ERROR [ImmichMicroservices] [JobService] Unable to run job handler (metadataExtraction/link-live-photos): QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" 2024-05-30T21:23:46.820328158Z [Nest] 7 - 05/30/2024, 9:23:46 PM ERROR [ImmichMicroservices] [JobService] QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" 2024-05-30T21:23:46.820342819Z at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19) 2024-05-30T21:23:46.820347411Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2024-05-30T21:23:46.820352164Z at async UpdateQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/UpdateQueryBuilder.js:83:33) 2024-05-30T21:23:46.820356296Z at async AssetRepository.update (/usr/src/app/dist/repositories/asset.repository.js:183:9) 2024-05-30T21:23:46.820360399Z at async MetadataService.handleLivePhotoLinking (/usr/src/app/dist/services/metadata.service.js:155:9) 2024-05-30T21:23:46.820364637Z at async /usr/src/app/dist/services/job.service.js:145:36 2024-05-30T21:23:46.820368572Z at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28) 2024-05-30T21:23:46.820372526Z at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) 2024-05-30T21:23:46.820376565Z [Nest] 7 - 05/30/2024, 9:23:46 PM ERROR [ImmichMicroservices] [JobService] Object: 2024-05-30T21:23:46.820382675Z { 2024-05-30T21:23:46.820386810Z "id": "686ee129-e364-4616-b14d-0e09fad7b987", 2024-05-30T21:23:46.820391084Z "source": "upload" 2024-05-30T21:23:46.820395487Z } ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 08:14:22 +03:00
Author
Owner
Copy Link

@davidvpe commented on GitHub (Jun 12, 2024):

I am also experiencing this issue

[Nest] 17  - 06/12/2024, 7:52:41 AM   ERROR [Api:AssetMediaService~ajxhyi4p] Error uploading file QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async SubjectExecutor.executeInsertOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:260:42)
    at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:92:9)
    at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21)
    at async AssetMediaService.create (/usr/src/app/dist/services/asset-media.service.js:292:23)
    at async AssetMediaService.uploadAsset (/usr/src/app/dist/services/asset-media.service.js:131:27)
    at async AssetMediaController.uploadAsset (/usr/src/app/dist/controllers/asset-media.controller.js:39:29)

A bit of a back story. Sometimes the app logs out by itself forcing it to resync all assets again and again, but since the last update this error started appearing, It's not very concerning as I know that the assets are already on the server, but if they are there then this's not exactly an error, just something that needs to be skipped next time.

@davidvpe commented on GitHub (Jun 12, 2024): I am also experiencing this issue ``` [Nest] 17 - 06/12/2024, 7:52:41 AM ERROR [Api:AssetMediaService~ajxhyi4p] Error uploading file QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef" at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33) at async SubjectExecutor.executeInsertOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:260:42) at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:92:9) at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21) at async AssetMediaService.create (/usr/src/app/dist/services/asset-media.service.js:292:23) at async AssetMediaService.uploadAsset (/usr/src/app/dist/services/asset-media.service.js:131:27) at async AssetMediaController.uploadAsset (/usr/src/app/dist/controllers/asset-media.controller.js:39:29) ``` A bit of a back story. Sometimes the app logs out by itself forcing it to resync all assets again and again, but since the last update this error started appearing, It's not very concerning as I know that the assets are already on the server, but if they are there then this's not exactly an error, just something that needs to be skipped next time.
OVERLORD commented 2026-02-05 08:14:25 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Jun 28, 2024):

Fixed in #10637

@alextran1502 commented on GitHub (Jun 28, 2024): Fixed in #10637
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#3383
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?