immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 11 Packages Projects Releases 10 Wiki Activity

[BUG] "STORAGE TEMPLATE MIGRATION" job makes duplicates in library #873

New Issue
Closed
opened 2026-02-04 23:12:59 +03:00 by OVERLORD · 22 comments
OVERLORD commented 2026-02-04 23:12:59 +03:00
Owner
Copy Link

Originally created by @thierrybla on GitHub (May 22, 2023).

The bug

After using the "STORAGE TEMPLATE MIGRATION" job it makes quite a lot of duplicates.

The OS that Immich Server is running on

Docker

Version of Immich Server

v1.56.2

Version of Immich Mobile App

v1.56.2

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "2.1"
services:
  immich:
    image: ghcr.io/imagegenius/immich:latest
    container_name: immich
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Etc/UTC
      - DB_HOSTNAME=192.168.1.x
      - DB_USERNAME=postgres
      - DB_PASSWORD=postgres
      - DB_DATABASE_NAME=immich
      - REDIS_HOSTNAME=192.168.1.x
      - DB_PORT=5432
      - REDIS_PORT=6379
      - REDIS_PASSWORD=
    volumes:
      - path_to_appdata:/config
      - path_to_photos:/photos
      - path_to_machine-learning:/config/machine-learning #optional
    ports:
      - 8080:8080
    restart: unless-stopped

Your .env content

N/A

Reproduction steps

1. Upload photos with random names
2. Click "STORAGE TEMPLATE MIGRATION" job
3. Look in your library and see duplicates

Additional information

No response

Originally created by @thierrybla on GitHub (May 22, 2023). ### The bug After using the "STORAGE TEMPLATE MIGRATION" job it makes quite a lot of duplicates. ### The OS that Immich Server is running on Docker ### Version of Immich Server v1.56.2 ### Version of Immich Mobile App v1.56.2 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "2.1" services: immich: image: ghcr.io/imagegenius/immich:latest container_name: immich environment: - PUID=1000 - PGID=1000 - TZ=Etc/UTC - DB_HOSTNAME=192.168.1.x - DB_USERNAME=postgres - DB_PASSWORD=postgres - DB_DATABASE_NAME=immich - REDIS_HOSTNAME=192.168.1.x - DB_PORT=5432 - REDIS_PORT=6379 - REDIS_PASSWORD= volumes: - path_to_appdata:/config - path_to_photos:/photos - path_to_machine-learning:/config/machine-learning #optional ports: - 8080:8080 restart: unless-stopped ``` ### Your .env content ```Shell N/A ``` ### Reproduction steps ```bash 1. Upload photos with random names 2. Click "STORAGE TEMPLATE MIGRATION" job 3. Look in your library and see duplicates ``` ### Additional information _No response_
OVERLORD commented 2026-02-04 23:13:08 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 22, 2023):

2023-05-18-IMG-20230518-WA0000
Photo before the job

and after:
2023-05-18-2023-05-18 06-52-01 - 20230518_065201_8C515D2A

They are both in my library at this moment.

@thierrybla commented on GitHub (May 22, 2023): ![2023-05-18-IMG-20230518-WA0000](https://github.com/immich-app/immich/assets/28044044/e7641476-d04d-4069-af30-91e539c1e4b8) Photo before the job and after: ![2023-05-18-2023-05-18 06-52-01 - 20230518_065201_8C515D2A](https://github.com/immich-app/immich/assets/28044044/1a5e20c3-d598-4c28-bbe0-dbc25b294617) They are both in my library at this moment.
OVERLORD commented 2026-02-04 23:13:11 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (May 22, 2023):

Hello, did you change the path when you ran the migration job or did you keep the old path? let me know and I can help taking a look

@alextran1502 commented on GitHub (May 22, 2023): Hello, did you change the path when you ran the migration job or did you keep the old path? let me know and I can help taking a look
OVERLORD commented 2026-02-04 23:13:15 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 22, 2023):

Hi! Thanks for the quick answer! I only changed the file name.

@thierrybla commented on GitHub (May 22, 2023): Hi! Thanks for the quick answer! I only changed the file name.
OVERLORD commented 2026-02-04 23:13:17 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (May 22, 2023):

@thierrybla Can you elaborate on changing the file name? Can you show the exact value changed in the storage migration box?

@alextran1502 commented on GitHub (May 22, 2023): @thierrybla Can you elaborate on changing the file name? Can you show the exact value changed in the storage migration box?
OVERLORD commented 2026-02-04 23:13:20 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (May 22, 2023):

Please note that any manual file system level operation will not applied to Immich because it causes the mismatch with the Immich's database

@alextran1502 commented on GitHub (May 22, 2023): Please note that any manual file system level operation will not applied to Immich because it causes the mismatch with the Immich's database
OVERLORD commented 2026-02-04 23:13:23 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 22, 2023):

image
This is basiclly what I did and then ran the job.

@thierrybla commented on GitHub (May 22, 2023): ![image](https://github.com/immich-app/immich/assets/28044044/c5e7d067-fccd-426d-b40b-f581e8a5dfd1) This is basiclly what I did and then ran the job.
OVERLORD commented 2026-02-04 23:13:27 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 22, 2023):

Please note that any manual file system level operation will not applied to Immich because it causes the mismatch with the Immich's database

I didn't change anything manually.

@thierrybla commented on GitHub (May 22, 2023): > Please note that any manual file system level operation will not applied to Immich because it causes the mismatch with the Immich's database I didn't change anything manually.
OVERLORD commented 2026-02-04 23:13:31 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 24, 2023):

@alextran1502 do you have time to debug this?

@thierrybla commented on GitHub (May 24, 2023): @alextran1502 do you have time to debug this?
OVERLORD commented 2026-02-04 23:13:36 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (May 24, 2023):

I did try and run the migration with the same template you specified but couldn't reproduce the issue you are having 🤔 . Can you try on the new release version?

@alextran1502 commented on GitHub (May 24, 2023): I did try and run the migration with the same template you specified but couldn't reproduce the issue you are having 🤔 . Can you try on the new release version?
OVERLORD commented 2026-02-04 23:13:39 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 24, 2023):

I did try and run the migration with the same template you specified but couldn't reproduce the issue you are having 🤔 . Can you try on the new release version?

Should I start from 0? Do you have any way to remove the created duplicates?

@thierrybla commented on GitHub (May 24, 2023): > I did try and run the migration with the same template you specified but couldn't reproduce the issue you are having 🤔 . Can you try on the new release version? Should I start from 0? Do you have any way to remove the created duplicates?
OVERLORD commented 2026-02-04 23:13:49 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (May 24, 2023):

Besides manually deleting them on the web, we don't have any other way. I am not sure about this bug, seems strange

@alextran1502 commented on GitHub (May 24, 2023): Besides manually deleting them on the web, we don't have any other way. I am not sure about this bug, seems strange
OVERLORD commented 2026-02-04 23:13:53 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 24, 2023):

Besides manually deleting them on the web, we don't have any other way. I am not sure about this bug, seems strange

What checks does immich do for duplicates? Seems having a different name is already enough to upload the same file twice?

Thank you for the swift replies :).

@thierrybla commented on GitHub (May 24, 2023): > Besides manually deleting them on the web, we don't have any other way. I am not sure about this bug, seems strange What checks does immich do for duplicates? Seems having a different name is already enough to upload the same file twice? Thank you for the swift replies :).
OVERLORD commented 2026-02-04 23:13:56 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (May 24, 2023):

A duplicate in Immichs means byte for byte identical files. If that's not the case with your files, then it's working as designed. We hope to improve this in the future.

@jrasm91 commented on GitHub (May 24, 2023): A duplicate in Immichs means byte for byte identical files. If that's not the case with your files, then it's working as designed. We hope to improve this in the future.
OVERLORD commented 2026-02-04 23:13:59 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 24, 2023):

A duplicate in Immichs means byte for byte identical files. If that's not the case with your files, then it's working as designed. We hope to improve this in the future.

Is this only upon upload or is there a background job for this as well?

@thierrybla commented on GitHub (May 24, 2023): > A duplicate in Immichs means byte for byte identical files. If that's not the case with your files, then it's working as designed. We hope to improve this in the future. Is this only upon upload or is there a background job for this as well?
OVERLORD commented 2026-02-04 23:14:02 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (May 24, 2023):

We calculate the file hash on upload and store it in the database. There is a unique key constraint on the column that makes it impossible for two files with the same hash to exist (for a single user). So uploading the same file again would not create a new record.

@jrasm91 commented on GitHub (May 24, 2023): We calculate the file hash on upload and store it in the database. There is a unique key constraint on the column that makes it impossible for two files with the same hash to exist (for a single user). So uploading the same file again would not create a new record.
OVERLORD commented 2026-02-04 23:14:12 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 25, 2023):

We calculate the file hash on upload and store it in the database. There is a unique key constraint on the column that makes it impossible for two files with the same hash to exist (for a single user). So uploading the same file again would not create a new record.

Hmm I see, still weird how Immich somehow made a duplicate of almost all my library files after running the job..

@alextran1502 I have a zip attached with the original photo and the photo that immich duplicated, maybe put the original photo called "2023-05-18-IMG-20230518-WA0000.jpg" into your own Immich and run the job yourself. It might be interesting to see if it also dupes it for you.

I will start over from scratch for myself.. this time I will right away change the template to something I want and not change it after the uploads are done.
2 photos.zip

@thierrybla commented on GitHub (May 25, 2023): > We calculate the file hash on upload and store it in the database. There is a unique key constraint on the column that makes it impossible for two files with the same hash to exist (for a single user). So uploading the same file again would not create a new record. Hmm I see, still weird how Immich somehow made a duplicate of almost all my library files after running the job.. @alextran1502 I have a zip attached with the original photo and the photo that immich duplicated, maybe put the original photo called "2023-05-18-IMG-20230518-WA0000.jpg" into your own Immich and run the job yourself. It might be interesting to see if it also dupes it for you. I will start over from scratch for myself.. this time I will right away change the template to something I want and not change it after the uploads are done. [2 photos.zip](https://github.com/immich-app/immich/files/11559318/2.photos.zip)
OVERLORD commented 2026-02-04 23:14:19 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (May 25, 2023):

Upon looking at the assets in the file browser, looks like they are two different version
image

The date modified and the size are different

@alextran1502 commented on GitHub (May 25, 2023): Upon looking at the assets in the file browser, looks like they are two different version <img width="677" alt="image" src="https://github.com/immich-app/immich/assets/27055614/19210854-07b0-4a9a-b256-38fa1faa5f92"> The date modified and the size are different
OVERLORD commented 2026-02-04 23:14:25 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 25, 2023):

Upon looking at the assets in the file browser, looks like they are two different version image

The date modified and the size are different

I can assure you that I didn't manually edit these files myself or used a different program to do anything to it after upload. Could immich change the things itself?

@thierrybla commented on GitHub (May 25, 2023): > Upon looking at the assets in the file browser, looks like they are two different version <img alt="image" width="677" src="https://user-images.githubusercontent.com/27055614/240745873-19210854-07b0-4a9a-b256-38fa1faa5f92.png"> > > The date modified and the size are different I can assure you that I didn't manually edit these files myself or used a different program to do anything to it after upload. Could immich change the things itself?
OVERLORD commented 2026-02-04 23:14:29 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 25, 2023):

To add, I uploaded the image with the CLI tool on the 20th and used the rename job on the 21th.

@thierrybla commented on GitHub (May 25, 2023): To add, I uploaded the image with the CLI tool on the 20th and used the rename job on the 21th.
OVERLORD commented 2026-02-04 23:14:34 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (May 25, 2023):

Immich doesn't touch the original file beside moving them to the template you specify. Can you start fresh and upload a few files and then try to move them again? Can you help record the change in the template?

@alextran1502 commented on GitHub (May 25, 2023): Immich doesn't touch the original file beside moving them to the template you specify. Can you start fresh and upload a few files and then try to move them again? Can you help record the change in the template?
OVERLORD commented 2026-02-04 23:14:39 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 25, 2023):

I will upload some images when I start fresh and try to reproduce what I did. I will report back here once I have done that.

@thierrybla commented on GitHub (May 25, 2023): I will upload some images when I start fresh and try to reproduce what I did. I will report back here once I have done that.
OVERLORD commented 2026-02-04 23:14:43 +03:00
Author
Owner
Copy Link

@thierrybla commented on GitHub (May 25, 2023):

For now I don't experience it, I will upload all my photos and see if something happens.. For now I will close this issue and reopen when I experience it again.

@thierrybla commented on GitHub (May 25, 2023): For now I don't experience it, I will upload all my photos and see if something happens.. For now I will close this issue and reopen when I experience it again.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#873
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?