[BUG] Negative remainder count on phone, photos stuck in uploads directory, broken image icons on web version #1128

Closed
opened 2026-02-05 00:33:37 +03:00 by OVERLORD · 7 comments
Owner

Originally created by @exptom on GitHub (Jul 20, 2023).

The bug

Hi,
I tested immich last week and uploaded some photos (low numbers) to it and all seemed good (really liked it!).
Yesterday I did a brand new proper setup of it (docker-compose) to start migrating fully from Google Photos. However I have hit a problem doing an initial sync from my phone (Pixel 4a).

  • Firstly, the "remainder" to backup on the phone is stuck at -1448 and it shows the wrong total.
    image

  • Secondly on the web version there are lots of missing image icons. These appear to correspond to images that are stuck in the uploads folder on the server.
    image

  • Looking at the logs in the "microservices" container I can see lots of error messages about duplicate id's.

e.g.

Nest] 6  - 07/19/2023, 8:03:12 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 6  - 07/19/2023, 8:03:12 PM   ERROR [StorageTemplateService] Error: Destination file already exists: upload/library/tom/2022/07/31/IMG/{filename}}+9.jpg
    at FilesystemProvider.moveFile (/usr/src/app/dist/infra/repositories/filesystem.provider.js:34:19)
    at async StorageTemplateService.moveAsset (/usr/src/app/dist/domain/storage-template/storage-template.service.js:134:17)
    at async StorageTemplateService.handleMigrationSingle (/usr/src/app/dist/domain/storage-template/storage-template.service.js:80:9)
    at async /usr/src/app/dist/domain/job/job.service.js:104:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:342:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:525:24)
[Nest] 6  - 07/19/2023, 8:03:12 PM   ERROR [StorageTemplateService] Object:
{
  "id": "5ecdabbd-5b68-4f5a-b2b0-fc4dd9b94f08",
  "source": "upload/upload/43bb7e54-32a7-4256-aac9-953e1a5135c7/5902c0d2-58a2-48b6-b34e-98c89725f00d.jpg",
  "destination": "upload/library/tom/2022/07/31/IMG/{filename}}+9.jpg"
}

[Nest] 6  - 07/19/2023, 8:03:12 PM    WARN [StorageTemplateService] Unable to save new originalPath to database, undoing move for path upload/upload/43bb7e54-32a7-4256-aac9-953e1a5135c7/72224e46-14e6-4845-8bf6-e791ba8a2d2b.jpg - filename PXL_20220731_151308681 - id 325723ac-0c6b-4cc3-8de7-c0a6ffde3003
[Nest] 6  - 07/19/2023, 8:03:12 PM    WARN [StorageTemplateService] QueryFailedError: duplicate key value violates unique constraint "UQ_4ed4f8052685ff5b1e7ca1058ba"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async UpdateQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/UpdateQueryBuilder.js:83:33)
    at async updateSubject (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:380:38)
    at async Promise.all (index 0)
    at async SubjectExecutor.executeUpdateOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:422:9)
    at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:99:9)
    at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21)
    at async AssetRepository.save (/usr/src/app/dist/infra/repositories/asset.repository.js:108:24)
    at async StorageTemplateService.moveAsset (/usr/src/app/dist/domain/storage-template/storage-template.service.js:142:21)

There may be other relevant log entries. Please let me know which containers you need to see logs from.

Thanks for any help!

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.67.2

Version of Immich Mobile App

v1.67.2

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: '3.8'

networks:
  lan-10:
    external: true
  lan-172:
    external: true
  immich:

services:
  # www
  immich-web:
    container_name: "${PROJECT}-www"
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION}
    hostname: "${PROJECT}.${DOMAIN}"
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
      - .env
    networks:
      - lan-10
      - lan-172
      - immich
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.${PROJECT}.rule=host(`${PROJECT}.${DOMAIN}`)"
      - "traefik.http.routers.${PROJECT}.entrypoints=https"
      - "traefik.http.routers.${PROJECT}.tls=true"
      - "traefik.http.routers.${PROJECT}.middlewares=ipallow-lan-subnets"
      - "traefik.http.services.${PROJECT}.loadbalancer.server.port=${PROXY_PORT}"
    restart: unless-stopped

  # api
  immich-server:
    container_name: "${PROJECT}-api"
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION}
    entrypoint: ["/bin/sh", "./start-server.sh"]
    networks:
      - lan-10
      - lan-172
      - immich
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - immich-redis
      - immich-postgres
      - immich-typesense
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.${PROJECT}-api.rule=host(`${PROJECT}.${DOMAIN}`) && Pathprefix(`/api`)"
      - "traefik.http.routers.${PROJECT}-api.entrypoints=https"
      - "traefik.http.routers.${PROJECT}-api.tls=true"
      - "traefik.http.middlewares.immich-api-strip.stripprefix.prefixes=/api"
      - "traefik.http.routers.${PROJECT}-api.middlewares=ipallow-lan-subnets, immich-api-strip"
      - "traefik.http.services.${PROJECT}-api.loadbalancer.server.port=${PROXY_PORT_API}"
    restart: unless-stopped

  immich-microservices:
    container_name: "${PROJECT}-microservices"
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION}
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    networks:
      - immich
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - immich-redis
      - immich-postgres
      - immich-typesense
    restart: unless-stopped

  immich-machine-learning:
    container_name: "${PROJECT}-machinelearning"
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION}
    networks:
      - immich
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - immich-postgres
    restart: unless-stopped

  immich-redis:
    container_name: "${PROJECT}-redis"
    image: redis:6
    networks:
      - immich
    env_file:
      - .env
    restart: unless-stopped

  immich-postgres:
    container_name: "${PROJECT}-postgres"
    image: postgres:14
    networks:
      - immich
    env_file:
      - .env
    volumes:
      - /marvinfast/appdata/immich/postgres:/var/lib/postgresql/data
    restart: unless-stopped

  immich-typesense:
    container_name: "${PROJECT}-typesense"
    image: typesense/typesense:0.24.1
    networks:
      - immich
    env_file:
      - .env
    logging:
      driver: none
    volumes:
      - /marvinfast/appdata/typesense/postgres:/data
    restart: unless-stopped

Your .env content

# common
PROJECT="photos"
DOMAIN=
TZ=Europe/London
PROXY_PORT=3000
PROXY_PORT_API=3001
SUBNET_172="172.16.10.0/24"
SUBNET_10="10.10.10.0/24"
HOST_10="10.10.10.10"
HOST_172="172.16.10.10"

# immich
IMMICH_VERSION="v1.67.2"
UPLOAD_LOCATION="/marvin/photos/immich"
NODE_ENV=production
PUBLIC_LOGIN_PAGE_MESSAGE=""
DB_HOSTNAME=immich-postgres
DB_USERNAME=immich
DB_DATABASE_NAME=immich
PUBLIC_IMMICH_SERVER_URL="https://${PROJECT}.${DOMAIN}"
IMMICH_WEB_URL="http://immich-web:3000"
IMMICH_SERVER_URL="http://immich-server:3001"
IMMICH_MACHINE_LEARNING_URL="http://immich-machine-learning:3003"

# redis
REDIS_HOSTNAME=immich-redis

# postgres
POSTGRES_USER=${DB_USERNAME}
POSTGRES_DB=${DB_DATABASE_NAME}
PG_DATA=/var/lib/postgresql/data

# typesense
TYPESENSE_HOST=immich-typesense
TYPESENSE_DATA_DIR=/data

## secrets
DB_PASSWORD=
POSTGRES_PASSWORD=${DB_PASSWORD}
TYPESENSE_API_KEY=

Reproduction steps

1. Docker compose up
2. Create admin account
3. Log in on phone and sync "camera" album

Additional information

No response

Originally created by @exptom on GitHub (Jul 20, 2023). ### The bug Hi, I tested immich last week and uploaded some photos (low numbers) to it and all seemed good (really liked it!). Yesterday I did a brand new proper setup of it (docker-compose) to start migrating fully from Google Photos. However I have hit a problem doing an initial sync from my phone (Pixel 4a). - Firstly, the "remainder" to backup on the phone is stuck at **-1448** and it shows the wrong total. ![image](https://github.com/immich-app/immich/assets/4302032/861b3816-c029-4025-a5a7-772a6188ae5d) - Secondly on the web version there are lots of missing image icons. These appear to correspond to images that are stuck in the uploads folder on the server. ![image](https://github.com/immich-app/immich/assets/4302032/585c207a-7f54-4fe4-b988-802acd061cc3) - Looking at the logs in the "microservices" container I can see lots of error messages about duplicate id's. e.g. ``` Nest] 6 - 07/19/2023, 8:03:12 PM ERROR [StorageTemplateService] Problem applying storage template [Nest] 6 - 07/19/2023, 8:03:12 PM ERROR [StorageTemplateService] Error: Destination file already exists: upload/library/tom/2022/07/31/IMG/{filename}}+9.jpg at FilesystemProvider.moveFile (/usr/src/app/dist/infra/repositories/filesystem.provider.js:34:19) at async StorageTemplateService.moveAsset (/usr/src/app/dist/domain/storage-template/storage-template.service.js:134:17) at async StorageTemplateService.handleMigrationSingle (/usr/src/app/dist/domain/storage-template/storage-template.service.js:80:9) at async /usr/src/app/dist/domain/job/job.service.js:104:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:342:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:525:24) [Nest] 6 - 07/19/2023, 8:03:12 PM ERROR [StorageTemplateService] Object: { "id": "5ecdabbd-5b68-4f5a-b2b0-fc4dd9b94f08", "source": "upload/upload/43bb7e54-32a7-4256-aac9-953e1a5135c7/5902c0d2-58a2-48b6-b34e-98c89725f00d.jpg", "destination": "upload/library/tom/2022/07/31/IMG/{filename}}+9.jpg" } [Nest] 6 - 07/19/2023, 8:03:12 PM WARN [StorageTemplateService] Unable to save new originalPath to database, undoing move for path upload/upload/43bb7e54-32a7-4256-aac9-953e1a5135c7/72224e46-14e6-4845-8bf6-e791ba8a2d2b.jpg - filename PXL_20220731_151308681 - id 325723ac-0c6b-4cc3-8de7-c0a6ffde3003 [Nest] 6 - 07/19/2023, 8:03:12 PM WARN [StorageTemplateService] QueryFailedError: duplicate key value violates unique constraint "UQ_4ed4f8052685ff5b1e7ca1058ba" at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async UpdateQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/UpdateQueryBuilder.js:83:33) at async updateSubject (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:380:38) at async Promise.all (index 0) at async SubjectExecutor.executeUpdateOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:422:9) at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:99:9) at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21) at async AssetRepository.save (/usr/src/app/dist/infra/repositories/asset.repository.js:108:24) at async StorageTemplateService.moveAsset (/usr/src/app/dist/domain/storage-template/storage-template.service.js:142:21) ``` There may be other relevant log entries. Please let me know which containers you need to see logs from. Thanks for any help! ### The OS that Immich Server is running on Ubuntu 22.04 ### Version of Immich Server v1.67.2 ### Version of Immich Mobile App v1.67.2 ### Platform with the issue - [X] Server - [X] Web - [X] Mobile ### Your docker-compose.yml content ```YAML version: '3.8' networks: lan-10: external: true lan-172: external: true immich: services: # www immich-web: container_name: "${PROJECT}-www" image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION} hostname: "${PROJECT}.${DOMAIN}" entrypoint: ["/bin/sh", "./entrypoint.sh"] env_file: - .env networks: - lan-10 - lan-172 - immich labels: - "traefik.enable=true" - "traefik.http.routers.${PROJECT}.rule=host(`${PROJECT}.${DOMAIN}`)" - "traefik.http.routers.${PROJECT}.entrypoints=https" - "traefik.http.routers.${PROJECT}.tls=true" - "traefik.http.routers.${PROJECT}.middlewares=ipallow-lan-subnets" - "traefik.http.services.${PROJECT}.loadbalancer.server.port=${PROXY_PORT}" restart: unless-stopped # api immich-server: container_name: "${PROJECT}-api" image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION} entrypoint: ["/bin/sh", "./start-server.sh"] networks: - lan-10 - lan-172 - immich volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - .env depends_on: - immich-redis - immich-postgres - immich-typesense labels: - "traefik.enable=true" - "traefik.http.routers.${PROJECT}-api.rule=host(`${PROJECT}.${DOMAIN}`) && Pathprefix(`/api`)" - "traefik.http.routers.${PROJECT}-api.entrypoints=https" - "traefik.http.routers.${PROJECT}-api.tls=true" - "traefik.http.middlewares.immich-api-strip.stripprefix.prefixes=/api" - "traefik.http.routers.${PROJECT}-api.middlewares=ipallow-lan-subnets, immich-api-strip" - "traefik.http.services.${PROJECT}-api.loadbalancer.server.port=${PROXY_PORT_API}" restart: unless-stopped immich-microservices: container_name: "${PROJECT}-microservices" image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION} entrypoint: ["/bin/sh", "./start-microservices.sh"] networks: - immich volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - .env depends_on: - immich-redis - immich-postgres - immich-typesense restart: unless-stopped immich-machine-learning: container_name: "${PROJECT}-machinelearning" image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION} networks: - immich volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - .env depends_on: - immich-postgres restart: unless-stopped immich-redis: container_name: "${PROJECT}-redis" image: redis:6 networks: - immich env_file: - .env restart: unless-stopped immich-postgres: container_name: "${PROJECT}-postgres" image: postgres:14 networks: - immich env_file: - .env volumes: - /marvinfast/appdata/immich/postgres:/var/lib/postgresql/data restart: unless-stopped immich-typesense: container_name: "${PROJECT}-typesense" image: typesense/typesense:0.24.1 networks: - immich env_file: - .env logging: driver: none volumes: - /marvinfast/appdata/typesense/postgres:/data restart: unless-stopped ``` ### Your .env content ```Shell # common PROJECT="photos" DOMAIN= TZ=Europe/London PROXY_PORT=3000 PROXY_PORT_API=3001 SUBNET_172="172.16.10.0/24" SUBNET_10="10.10.10.0/24" HOST_10="10.10.10.10" HOST_172="172.16.10.10" # immich IMMICH_VERSION="v1.67.2" UPLOAD_LOCATION="/marvin/photos/immich" NODE_ENV=production PUBLIC_LOGIN_PAGE_MESSAGE="" DB_HOSTNAME=immich-postgres DB_USERNAME=immich DB_DATABASE_NAME=immich PUBLIC_IMMICH_SERVER_URL="https://${PROJECT}.${DOMAIN}" IMMICH_WEB_URL="http://immich-web:3000" IMMICH_SERVER_URL="http://immich-server:3001" IMMICH_MACHINE_LEARNING_URL="http://immich-machine-learning:3003" # redis REDIS_HOSTNAME=immich-redis # postgres POSTGRES_USER=${DB_USERNAME} POSTGRES_DB=${DB_DATABASE_NAME} PG_DATA=/var/lib/postgresql/data # typesense TYPESENSE_HOST=immich-typesense TYPESENSE_DATA_DIR=/data ## secrets DB_PASSWORD= POSTGRES_PASSWORD=${DB_PASSWORD} TYPESENSE_API_KEY= ``` ### Reproduction steps ```bash 1. Docker compose up 2. Create admin account 3. Log in on phone and sync "camera" album ``` ### Additional information _No response_
Author
Owner

@alextran1502 commented on GitHub (Jul 21, 2023):

Hello, thank you for reporting.

  1. The negative number is a known cosmetic issue, you can fix it by clearing the app cache.
  2. Can you try running the missing thumbnail job to see if it fixes the issue? We are still trying to pin down some of these issue to fix them
@alextran1502 commented on GitHub (Jul 21, 2023): Hello, thank you for reporting. 1. The negative number is a known cosmetic issue, you can fix it by clearing the app cache. 2. Can you try running the missing thumbnail job to see if it fixes the issue? We are still trying to pin down some of these issue to fix them
Author
Owner

@exptom commented on GitHub (Jul 21, 2023):

I had already tried both those things (from seeing other similar-ish tickets and they didn't work. 😞 )

  1. Clearing cache didnt work. I think that number is the number of photos stuck in the uploads folder on the server
  2. Trying to generate thumbnails didnt work. I think these missing images are the image files stuck in the uploads folder.

The app thinks they have all be uploaded and have the correct cloud icon. It seems like they havent been copied from uploads into the actual library folder (lots of database unique key errors as per that log above)

@exptom commented on GitHub (Jul 21, 2023): I had already tried both those things (from seeing other similar-ish tickets and they didn't work. :disappointed: ) 1. Clearing cache didnt work. I think that number is the number of photos stuck in the uploads folder on the server 2. Trying to generate thumbnails didnt work. I think these missing images are the image files stuck in the uploads folder. The app thinks they have all be uploaded and have the correct cloud icon. It seems like they havent been copied from uploads into the actual library folder (lots of database unique key errors as per that log above)
Author
Owner

@alextran1502 commented on GitHub (Jul 21, 2023):

@exptom Are you using the default storage template? I am seeing this

  "destination": "upload/library/tom/2022/07/31/IMG/{filename}}+9.jpg"

Which have }} at the end

@alextran1502 commented on GitHub (Jul 21, 2023): @exptom Are you using the default storage template? I am seeing this ``` "destination": "upload/library/tom/2022/07/31/IMG/{filename}}+9.jpg" ``` Which have `}}` at the end
Author
Owner

@exptom commented on GitHub (Jul 21, 2023):

@alextran1502 How stupid of me! I'm sorry I missed a { from the start of the filename variable so it's trying to create lots of photos with {filename}}+9 in 🙄

I've fixed the template and rerun the storage migration but the uploads folder isnt clearing. Should I delete everything and start from scratch again?

@exptom commented on GitHub (Jul 21, 2023): @alextran1502 How stupid of me! I'm sorry I missed a `{` from the start of the filename variable so it's trying to create lots of photos with `{filename}}+9` in :roll_eyes: I've fixed the template and rerun the storage migration but the uploads folder isnt clearing. Should I delete everything and start from scratch again?
Author
Owner

@alextran1502 commented on GitHub (Jul 21, 2023):

@exptom I would advise doing so to save lots of hassle 😸 This sparks the questions, we have the template preview in the web UI to help indicate issues like this, did it show anything that would help users to avoid these situations in the future?

@alextran1502 commented on GitHub (Jul 21, 2023): @exptom I would advise doing so to save lots of hassle 😸 This sparks the questions, we have the template preview in the web UI to help indicate issues like this, did it show anything that would help users to avoid these situations in the future?
Author
Owner

@exptom commented on GitHub (Jul 21, 2023):

@alextran1502 A clean install and resync on my phone has worked straight away! Now to try import my entire 20gb google photos library that I downloaded with takeout 😅

Sorry for wasting your time.

I'm not sure what else could have been done in the UI for the storage template. Perhaps a warning message if it detects a single { or } in the template. Something like "It looks like you have an incomplete variable name - double check you want that before saving!". I'm not sure why I didn't notice it, I had been using the preview option to play around with a few template path formats.

@exptom commented on GitHub (Jul 21, 2023): @alextran1502 A clean install and resync on my phone has worked straight away! Now to try import my entire 20gb google photos library that I downloaded with takeout :sweat_smile: Sorry for wasting your time. I'm not sure what else could have been done in the UI for the storage template. Perhaps a warning message if it detects a single `{` or `}` in the template. Something like "It looks like you have an incomplete variable name - double check you want that before saving!". I'm not sure why I didn't notice it, I had been using the preview option to play around with a few template path formats.
Author
Owner

@alextran1502 commented on GitHub (Jul 21, 2023):

I remember I put in an error on the template if it can't be parsed. Hmm, will have to circle back to this.

For Google Takeout please make sure you rewrite the metadata back to the file EXIF since they stripped them and put in a corresponding json files. You can find a thread pinned in Discussion section

@alextran1502 commented on GitHub (Jul 21, 2023): I remember I put in an error on the template if it can't be parsed. Hmm, will have to circle back to this. For Google Takeout please make sure you rewrite the metadata back to the file EXIF since they stripped them and put in a corresponding json files. You can find a thread pinned in Discussion section
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1128