immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 12 Packages Projects Releases 10 Wiki Activity

[BUG] Downloading image to mobile device that has been uploaded with CLI results in duplicates. #737

New Issue
Closed
opened 2026-02-04 22:11:41 +03:00 by OVERLORD · 31 comments
OVERLORD commented 2026-02-04 22:11:41 +03:00
Owner
Copy Link

Originally created by @florihupf on GitHub (Mar 5, 2023).

Originally assigned to: @fyfrey on GitHub.

The bug

Hi there,

maybe not the most common workflow, but never the less would be great to fix.

Repro:

a) Fresh install of Immich
b) Upload tons of pictures to the server using CLI tool (works great).
c) Install mobile app (here iOS)
d) Browse photo selection and download one to mobile device (say you want to use it as background image, etc).
e) Observe that mobile app will instantly detect this image and re-uploads it as a duplicate to the server.

Upon closer inspection what I can see if that the filename endings differ:

Orig.: IMG_6502.jpg
Dupe: IMG_6502.jpg.JPG

On the plus: Deleting the duplicate will only delete the duplicate and not both. 👍

Would be a great fix, if easy.

The OS that Immich Server is running on

Truenas Scale

Version of Immich Server

v1.51.0

Version of Immich Mobile App

v1.51.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

Stock.

Your .env content

Stock.

Reproduction steps

See above.

Additional information

No response

Originally created by @florihupf on GitHub (Mar 5, 2023). Originally assigned to: @fyfrey on GitHub. ### The bug Hi there, maybe not the most common workflow, but never the less would be great to fix. Repro: a) Fresh install of Immich b) Upload tons of pictures to the server using CLI tool (works great). c) Install mobile app (here iOS) d) Browse photo selection and download one to mobile device (say you want to use it as background image, etc). e) Observe that mobile app will instantly detect this image and re-uploads it as a duplicate to the server. Upon closer inspection what I can see if that the filename endings differ: Orig.: IMG_6502.jpg Dupe: IMG_6502.jpg.JPG On the plus: Deleting the duplicate will only delete the duplicate and not both. 👍 Would be a great fix, if easy. ### The OS that Immich Server is running on Truenas Scale ### Version of Immich Server v1.51.0 ### Version of Immich Mobile App v1.51.0 ### Platform with the issue - [X] Server - [ ] Web - [X] Mobile ### Your docker-compose.yml content ```YAML Stock. ``` ### Your .env content ```Shell Stock. ``` ### Reproduction steps ```bash See above. ``` ### Additional information _No response_
OVERLORD added the 🗄️server📱mobile labels 2026-02-04 22:11:41 +03:00
OVERLORD commented 2026-02-04 22:11:49 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Mar 5, 2023):

mobile app will instantly detect this image and re-uploads it as a duplicate to the server.

Does the image actually show up duplicated in the (web) UI after this?

@bo0tzz commented on GitHub (Mar 5, 2023): > mobile app will instantly detect this image and re-uploads it as a duplicate to the server. Does the image actually show up duplicated in the (web) UI after this?
OVERLORD commented 2026-02-04 22:11:52 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Mar 5, 2023):

Also, let's say for simplicity, the iOS photo roll is empty when doing c) from the repro steps.

@florihupf commented on GitHub (Mar 5, 2023): Also, let's say for simplicity, the iOS photo roll is empty when doing c) from the repro steps.
OVERLORD commented 2026-02-04 22:11:59 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Mar 5, 2023):

Yes, it also shows duplicated on the web.

@florihupf commented on GitHub (Mar 5, 2023): Yes, it also shows duplicated on the web.
OVERLORD commented 2026-02-04 22:12:01 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Mar 5, 2023):

That's very strange, since we have hash-based deduplication so it should be detected that it's the same file. Is there a chance that it's being picked up by iCloud photos or such and being compressed or otherwise altered?

@bo0tzz commented on GitHub (Mar 5, 2023): That's very strange, since we have hash-based deduplication so it should be detected that it's the same file. Is there a chance that it's being picked up by iCloud photos or such and being compressed or otherwise altered?
OVERLORD commented 2026-02-04 22:12:06 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Mar 5, 2023):

Don't think so, it happens right away. Also I have iClouds photos disabled.

@florihupf commented on GitHub (Mar 5, 2023): Don't think so, it happens right away. Also I have iClouds photos disabled.
OVERLORD commented 2026-02-04 22:12:11 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Mar 5, 2023):

Is there maybe a difference between how the hashes are computed for CLI uploads and mobile assets?

@florihupf commented on GitHub (Mar 5, 2023): Is there maybe a difference between how the hashes are computed for CLI uploads and mobile assets?
OVERLORD commented 2026-02-04 22:12:13 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 5, 2023):

Hashes are computed on the server regardless of what client uploads them.

@jrasm91 commented on GitHub (Mar 5, 2023): Hashes are computed on the server regardless of what client uploads them.
OVERLORD commented 2026-02-04 22:12:15 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Mar 5, 2023):

Does the file name go into the hash? Maybe downloading an image modifies it.

@florihupf commented on GitHub (Mar 5, 2023): Does the file name go into the hash? Maybe downloading an image modifies it.
OVERLORD commented 2026-02-04 22:12:17 +03:00
Author
Owner
Copy Link

@klejejs commented on GitHub (Mar 6, 2023):

I can confirm, I have a similar issue.

Steps to reproduce:

  • Download images to PC and upload them to the web with drag-n-drop
  • Download the same images from the same source to iOS and add them to the library
  • iOS app uploads the images again and there are duplicates in the Immich library

I did notice that GPhotos also has the same issue, so I wonder if the hash or something else about the images actually does change. 🤔
I also don't use iCloud if that can impact anything.

@klejejs commented on GitHub (Mar 6, 2023): I can confirm, I have a similar issue. Steps to reproduce: * Download images to PC and upload them to the web with drag-n-drop * Download the same images from the same source to iOS and add them to the library * iOS app uploads the images again and there are duplicates in the Immich library * I did notice that GPhotos also has the same issue, so I wonder if the hash or something else about the images actually does change. 🤔 I also don't use iCloud if that can impact anything.
OVERLORD commented 2026-02-04 22:12:22 +03:00
Author
Owner
Copy Link

@Dilanwizzy commented on GitHub (Mar 21, 2023):

This issue is related to IOS and how it handles pictures. By default IOS will try and optimise the images on your device to save storage and uploading the original to iCloud. So, when you download the image from immich onto your device. IOS will try and optmise it, and because the file has changed. Immich will pick this up as a different photo. I have tested this, and the files are usually different sizes no matter how many times I download and it gets reuploaded.

I found this while searching

"
someone on Reddit said the problem is from the iOS end of things. On the iphone, under Settings, Photos, you need to turn off "optimise iphone storage" and switch instead to "download and keep originals". The first option means that if your phone is low on space, it replaces full-resolution photos with smaller versions and keeps the full sized version in iCloud. I just did this and the problem appears to be solved.
"
Source

I have tried this. On my side already uploaded images seems to keep being reuploaded. Newer images are working properly and you don't get duplicates

As this is an IOS issue, I believe it will be appropriate to have some documentation around this so people understand. Nothing immich can do besides having to extract the metadata and comparing everything
@bo0tzz

@Dilanwizzy commented on GitHub (Mar 21, 2023): This issue is related to IOS and how it handles pictures. By default IOS will try and optimise the images on your device to save storage and uploading the original to iCloud. So, when you download the image from immich onto your device. IOS will try and optmise it, and because the file has changed. Immich will pick this up as a different photo. I have tested this, and the files are usually different sizes no matter how many times I download and it gets reuploaded. I found this while searching " someone on Reddit said the problem is from the iOS end of things. On the iphone, under Settings, Photos, you need to turn off "optimise iphone storage" and switch instead to "download and keep originals". The first option means that if your phone is low on space, it replaces full-resolution photos with smaller versions and keeps the full sized version in iCloud. I just did this and the problem appears to be solved. " [Source](https://support.google.com/photos/thread/8854911/stuck-with-thousands-of-duplicate-photos-after-using-backup-and-sync-from-computer?hl=en) I have tried this. On my side already uploaded images seems to keep being reuploaded. Newer images are working properly and you don't get duplicates As this is an IOS issue, I believe it will be appropriate to have some documentation around this so people understand. Nothing immich can do besides having to extract the metadata and comparing everything @bo0tzz
OVERLORD commented 2026-02-04 22:12:25 +03:00
Author
Owner
Copy Link

@klejejs commented on GitHub (Mar 21, 2023):

@Dilanwizzy isn't that the case only if you have iCloud photos enabled? I do not upload photos to iCloud so I don't have that option available, but I still have the the issue.

@klejejs commented on GitHub (Mar 21, 2023): @Dilanwizzy isn't that the case only if you have iCloud photos enabled? I do not upload photos to iCloud so I don't have that option available, but I still have the the issue.
OVERLORD commented 2026-02-04 22:12:27 +03:00
Author
Owner
Copy Link

@Dilanwizzy commented on GitHub (Mar 21, 2023):

@klejejs The uploaded pictures have different sizes. I have just done what you explained and each time the size is different. The size difference is usually by 1 or 2 kb. Here is a file continuously downloaded.

3,292,209
3,291,980
3,291,298
3,292,438

File size each time is different. I have also seen it be the same and the image does not get uploaded. I don't see this with a file taken on my iPhone and downloaded. This happens only when the file is downloaded from a different source and then you save the image.

The issue is 95% looking like its all related to IOS, and how they handle images. I am not seeing this on Android

@Dilanwizzy commented on GitHub (Mar 21, 2023): @klejejs The uploaded pictures have different sizes. I have just done what you explained and each time the size is different. The size difference is usually by 1 or 2 kb. Here is a file continuously downloaded. 3,292,209 3,291,980 3,291,298 3,292,438 File size each time is different. I have also seen it be the same and the image does not get uploaded. I don't see this with a file taken on my iPhone and downloaded. This happens only when the file is downloaded from a different source and then you save the image. The issue is 95% looking like its all related to IOS, and how they handle images. I am not seeing this on Android
OVERLORD commented 2026-02-04 22:12:29 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 21, 2023):

Instead of looking at sizes it would be best to do a sha1 hash.

You should be able to verify the hash at each step and it should be the same at each step. Immich should not modify it at all, ever.

  • Take a picture, verify sha1 hash
  • Upload the picture to immich, verify sha1 of file in upload directory
  • Download the picture using "download" feature in immich web, verify sha1 of file is the same
  • Download the picture from the mobile app, verify the sha1 of the file is the same
  • Upload the picture again, verify it's a duplicate and not processed

If the sha1 changes at any step, that would be a problem. It might be related to iOS cloud stuff though.

@jrasm91 commented on GitHub (Mar 21, 2023): Instead of looking at sizes it would be best to do a sha1 hash. You should be able to verify the hash at each step and it should be the same at each step. Immich should not modify it at all, ever. - Take a picture, verify sha1 hash - Upload the picture to immich, verify sha1 of file in upload directory - Download the picture using "download" feature in immich web, verify sha1 of file is the same - Download the picture from the mobile app, verify the sha1 of the file is the same - Upload the picture again, verify it's a duplicate and not processed If the sha1 changes at any step, that would be a problem. It might be related to iOS cloud stuff though.
OVERLORD commented 2026-02-04 22:12:32 +03:00
Author
Owner
Copy Link

@Dilanwizzy commented on GitHub (Mar 21, 2023):

I used sha1 hash to verify and the file each time the hash is different. The different sizes show that maybe IOS is doing something with the files.

The same issue also happens on Google Photos. I am not getting this issue on Android as mentioned

@Dilanwizzy commented on GitHub (Mar 21, 2023): I used sha1 hash to verify and the file each time the hash is different. The different sizes show that maybe IOS is doing something with the files. The same issue also happens on Google Photos. I am not getting this issue on Android as mentioned
OVERLORD commented 2026-02-04 22:12:35 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 21, 2023):

Do you have some files you can upload that I can inspect? Ideally, an original and then a download-and-modified version.

@jrasm91 commented on GitHub (Mar 21, 2023): Do you have some files you can upload that I can inspect? Ideally, an original and then a download-and-modified version.
OVERLORD commented 2026-02-04 22:12:38 +03:00
Author
Owner
Copy Link

@Dilanwizzy commented on GitHub (Mar 21, 2023):

So, it seems like the way google handles this is by appending the word original to the file. And then treating the file on the server and the one downloaded as 1 file. Currently, immich would treat them as 2 separate images, with one being uploaded on the server and the other showing as not uploaded.

@Dilanwizzy commented on GitHub (Mar 21, 2023): So, it seems like the way google handles this is by appending the word original to the file. And then treating the file on the server and the one downloaded as 1 file. Currently, immich would treat them as 2 separate images, with one being uploaded on the server and the other showing as not uploaded.
OVERLORD commented 2026-02-04 22:12:39 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Mar 22, 2023):

Yeah, also quick note that I have iCloud disabled and thus the option mentioned is not available for me.

@florihupf commented on GitHub (Mar 22, 2023): Yeah, also quick note that I have iCloud disabled and thus the option mentioned is not available for me.
OVERLORD commented 2026-02-04 22:12:42 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Mar 22, 2023):

Can somebody share/upload a sample image (before and after) that I can investigate with? Otherwise, it is hard to know what is going on.

@jrasm91 commented on GitHub (Mar 22, 2023): Can somebody share/upload a sample image (before and after) that I can investigate with? Otherwise, it is hard to know what is going on.
OVERLORD commented 2026-02-04 22:12:43 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Mar 22, 2023):

Ok, will see to it.

@florihupf commented on GitHub (Mar 22, 2023): Ok, will see to it.
OVERLORD commented 2026-02-04 22:12:47 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Mar 22, 2023):

immich.zip

Here is a zip with a picture uploaded via CLI on Feb/23. I just downloaded onto my phone on Mar/22 and it got uploaded right away as a duplicate. I downloaded them both using the web gui. Note that the zip contains two files with the same name.

HTH and thanks for taking a peek!

@florihupf commented on GitHub (Mar 22, 2023): [immich.zip](https://github.com/immich-app/immich/files/11036921/immich.zip) Here is a zip with a picture uploaded via CLI on Feb/23. I just downloaded onto my phone on Mar/22 and it got uploaded right away as a duplicate. I downloaded them both using the web gui. Note that the zip contains two files with the same name. HTH and thanks for taking a peek!
OVERLORD commented 2026-02-04 22:12:50 +03:00
Author
Owner
Copy Link

@colin715 commented on GitHub (Mar 31, 2023):

I'm seeing similar behavior on Android. My wife (also Android) and I share a user account so that we can see each other's pictures. If I download an image to my device that my wife took, then I end up with a duplicate photo. And the original photo still shows up as not on my device.

@colin715 commented on GitHub (Mar 31, 2023): I'm seeing similar behavior on Android. My wife (also Android) and I share a user account so that we can see each other's pictures. If I download an image to my device that my wife took, then I end up with a duplicate photo. And the original photo still shows up as not on my device.
OVERLORD commented 2026-02-04 22:12:54 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Mar 31, 2023):

Thank you all for the report, we are hoping to address this issue in the next release or so

@alextran1502 commented on GitHub (Mar 31, 2023): Thank you all for the report, we are hoping to address this issue in the next release or so
OVERLORD commented 2026-02-04 22:12:56 +03:00
Author
Owner
Copy Link

@t-shock commented on GitHub (May 22, 2023):

I set up Immich couple days ago and I’m facing the same issue with my iPhone

@t-shock commented on GitHub (May 22, 2023): I set up Immich couple days ago and I’m facing the same issue with my iPhone
OVERLORD commented 2026-02-04 22:13:02 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Jun 10, 2023):

Wohoooooooooooo 💯 🥇

@florihupf commented on GitHub (Jun 10, 2023): Wohoooooooooooo 💯 🥇
OVERLORD commented 2026-02-04 22:13:08 +03:00
Author
Owner
Copy Link

@uhthomas commented on GitHub (Jun 16, 2023):

@florihupf You reported your server and app versions as v1.61.1, which isn't a valid release. The current release is v1.61.0. Would you mind checking both are up to date?

@uhthomas commented on GitHub (Jun 16, 2023): @florihupf You reported your server and app versions as `v1.61.1`, which isn't a valid release. The current release is `v1.61.0`. Would you mind checking both are up to date?
OVERLORD commented 2026-02-04 22:13:16 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Jun 17, 2023):

Yeah, my bad. I am up to date with v1.61.0. Typo on the last digit.

@florihupf commented on GitHub (Jun 17, 2023): Yeah, my bad. I am up to date with v1.61.0. Typo on the last digit.
OVERLORD commented 2026-02-04 22:13:19 +03:00
Author
Owner
Copy Link

@fyfrey commented on GitHub (Jun 19, 2023):

@florihupf these must be different files (maybe iOS does not give access to the original file or a bit error occurred). There can only be exactly 1 asset with the same checksum per user. Please check whether the files have the same checksum (enable advanced error reporting in Immich app settings, show info of the two duplicates)

@fyfrey commented on GitHub (Jun 19, 2023): @florihupf these must be different files (maybe iOS does not give access to the original file or a bit error occurred). There can only be exactly 1 asset with the same checksum per user. Please check whether the files have the same checksum (enable advanced error reporting in Immich app settings, show info of the two duplicates)
OVERLORD commented 2026-02-04 22:13:23 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Jun 19, 2023):

I can repro with more detail.

How do I enable advanced error reporting? In the iOS Immich app, I see Log Level: Info and a trouble shooting toogle. What settings should apply?

As for the repro steps:

a) Just click download on any picture that is not downloaded to the phone yet (but uploaded via CLI to the server).
b) Upon completion of the download, there will be two images in the mobile app (identical looks) and the second image will be uploaded to the server, creating a duplicate there as well (after a delay).

HTH

@florihupf commented on GitHub (Jun 19, 2023): I can repro with more detail. How do I enable advanced error reporting? In the iOS Immich app, I see Log Level: Info and a trouble shooting toogle. What settings should apply? As for the repro steps: a) Just click download on any picture that is not downloaded to the phone yet (but uploaded via CLI to the server). b) Upon completion of the download, there will be two images in the mobile app (identical looks) and the second image will be uploaded to the server, creating a duplicate there as well (after a delay). HTH
OVERLORD commented 2026-02-04 22:13:28 +03:00
Author
Owner
Copy Link

@fyfrey commented on GitHub (Jun 21, 2023):

Enable trouble shooting. Then when clicking the info icon in the full screen view, you see advanced/raw info about an asset.

I bet the checksums are different. But I don't know yet why.

@fyfrey commented on GitHub (Jun 21, 2023): Enable trouble shooting. Then when clicking the info icon in the full screen view, you see advanced/raw info about an asset. I bet the checksums are different. But I don't know yet why.
OVERLORD commented 2026-02-04 22:13:31 +03:00
Author
Owner
Copy Link

@paniman commented on GitHub (Sep 15, 2023):

One solution for the slightly adjusted image to detect as duplicate could be to use phash instead of a another hash algorithm.
Source: https://www.phash.org

@paniman commented on GitHub (Sep 15, 2023): One solution for the slightly adjusted image to detect as duplicate could be to use phash instead of a another hash algorithm. Source: https://www.phash.org
OVERLORD commented 2026-02-04 22:13:35 +03:00
Author
Owner
Copy Link

@florihupf commented on GitHub (Nov 5, 2023):

Hi there,

just tried this again and it works now. Lets close this out. Thanks!

@florihupf commented on GitHub (Nov 5, 2023): Hi there, just tried this again and it works now. Lets close this out. Thanks!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label 📱mobile 🗄️server
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#737
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?