immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

[PR #557] [MERGED] feat(server): de-duplication #8670

New Issue
Closed
opened 2026-02-05 13:51:17 +03:00 by OVERLORD · 0 comments
OVERLORD commented 2026-02-05 13:51:17 +03:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/immich-app/immich/pull/557
Author: @panoti
Created: 8/31/2022
Status: ✅ Merged
Merged: 9/5/2022
Merged by: @alextran1502

Base: main ← Head: feat/deduplication

📝 Commits (10+)

  • f53b648 feat(server): remove un-used deviceAssetId cols.
  • 5782299 feat(server): return 409 if asset is duplicated
  • f1bf608 feat(server): replace old unique constaint
  • 11c892f feat(server): strip deviceId in file path
  • 4d7397c feat(server): skip duplicate asset
  • 3d0da62 chore(server): revert changes
  • 92f1a55 fix(server): asset test spec
  • 6a64fe9 Merge branch 'upstream' into feat/deduplication
  • 637ff9f fix(server): checksum generation for uploaded assets
  • f6d479a fix(server): make sure generation queue run after migraion

📊 Changes

9 files changed (+64 additions, -18 deletions)

View changed files

📝 server/apps/immich/src/api-v1/asset/asset-repository.ts (+17 -0)
📝 server/apps/immich/src/api-v1/asset/asset.service.spec.ts (+1 -0)
📝 server/apps/immich/src/api-v1/asset/asset.service.ts (+23 -9)
📝 server/apps/immich/src/config/asset-upload.config.ts (+2 -2)
📝 server/apps/microservices/src/microservices.service.ts (+3 -1)
📝 server/apps/microservices/src/processors/generate-checksum.processor.ts (+0 -4)
📝 server/libs/database/src/entities/asset.entity.ts (+1 -1)
📝 server/libs/database/src/migrations/1661881837496-AddAssetChecksum.ts (+1 -1)
➕ server/libs/database/src/migrations/1661971370662-UpdateAssetTableWithNewUniqueConstraint.ts (+16 -0)

📄 Description

  • Remove deviceAssetId col in model (create new pull request for this task). Remove unique index @Unique(['deviceAssetId', 'userId', 'deviceId']) (fix https://github.com/immich-app/immich/issues/415).
  • [?] Strip deviceId in file path
  • Create unique index on userId and checksum cols. note: some previous uploaded assets don't have checksum, so we should have a crob job (or migration) to generate checksum (https://github.com/immich-app/immich/pull/558)
  • If duplication occurs
    • Option 1:
      • Return http code 409
      • Show upload error in list (on the web app).
      • Show duplication error on the mobile.
    • Option 2:
      • Skip with http code 201 (like google photos. I can confirm that google photos skips duplicate files (even in different file name) without error message. I tried download some files, then renamed them and re-upload again).

For small change but high efficiency, I decide to leave with Option 2.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/immich-app/immich/pull/557 **Author:** [@panoti](https://github.com/panoti) **Created:** 8/31/2022 **Status:** ✅ Merged **Merged:** 9/5/2022 **Merged by:** [@alextran1502](https://github.com/alextran1502) **Base:** `main` ← **Head:** `feat/deduplication` --- ### 📝 Commits (10+) - [`f53b648`](https://github.com/immich-app/immich/commit/f53b6489843b748bde3c04a26c6f1adca1baacdf) feat(server): remove un-used deviceAssetId cols. - [`5782299`](https://github.com/immich-app/immich/commit/57822994d7fd4884b131d09fc2e842fd21d5b9fc) feat(server): return 409 if asset is duplicated - [`f1bf608`](https://github.com/immich-app/immich/commit/f1bf608f763ccdc117f041eb075029ba5404ad87) feat(server): replace old unique constaint - [`11c892f`](https://github.com/immich-app/immich/commit/11c892fdcbc41e3d4c2da9f408327b88df3c8b07) feat(server): strip deviceId in file path - [`4d7397c`](https://github.com/immich-app/immich/commit/4d7397cd8c59e75b900d1db0e29e7850f48b5e9d) feat(server): skip duplicate asset - [`3d0da62`](https://github.com/immich-app/immich/commit/3d0da628e9a832f699f3b498c4316348fccc1e82) chore(server): revert changes - [`92f1a55`](https://github.com/immich-app/immich/commit/92f1a553408c6a7e729d7351d65f95f7c9190550) fix(server): asset test spec - [`6a64fe9`](https://github.com/immich-app/immich/commit/6a64fe957ef42b9bbf20fa3473e2d1ee293d84ef) Merge branch 'upstream' into feat/deduplication - [`637ff9f`](https://github.com/immich-app/immich/commit/637ff9fdaca8f979fc14ebbf3ca9272931c94225) fix(server): checksum generation for uploaded assets - [`f6d479a`](https://github.com/immich-app/immich/commit/f6d479a19c1925b6a8ee34e096f7048e4efcd315) fix(server): make sure generation queue run after migraion ### 📊 Changes **9 files changed** (+64 additions, -18 deletions) <details> <summary>View changed files</summary> 📝 `server/apps/immich/src/api-v1/asset/asset-repository.ts` (+17 -0) 📝 `server/apps/immich/src/api-v1/asset/asset.service.spec.ts` (+1 -0) 📝 `server/apps/immich/src/api-v1/asset/asset.service.ts` (+23 -9) 📝 `server/apps/immich/src/config/asset-upload.config.ts` (+2 -2) 📝 `server/apps/microservices/src/microservices.service.ts` (+3 -1) 📝 `server/apps/microservices/src/processors/generate-checksum.processor.ts` (+0 -4) 📝 `server/libs/database/src/entities/asset.entity.ts` (+1 -1) 📝 `server/libs/database/src/migrations/1661881837496-AddAssetChecksum.ts` (+1 -1) ➕ `server/libs/database/src/migrations/1661971370662-UpdateAssetTableWithNewUniqueConstraint.ts` (+16 -0) </details> ### 📄 Description - [x] ~~Remove `deviceAssetId` col in model~~ (create new pull request for this task). Remove unique index `@Unique(['deviceAssetId', 'userId', 'deviceId'])` (fix https://github.com/immich-app/immich/issues/415). - [ ] [?] ~Strip deviceId in file path~ - [x] Create unique index on `userId` and `checksum` cols. note: some previous uploaded assets don't have checksum, so we should have a crob job (or migration) to generate checksum (https://github.com/immich-app/immich/pull/558) - [x] If duplication occurs + **Option 1**: * [ ] Return http code 409 * [ ] Show upload error in list (on the web app). * [ ] Show duplication error on the mobile. + **Option 2**: * [x] Skip with http code 201 (like google photos. I can confirm that google photos skips duplicate files (even in different file name) without error message. I tried download some files, then renamed them and re-upload again). For small change but high efficiency, I decide to leave with **Option 2**. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
OVERLORD added the pull-request label 2026-02-05 13:51:17 +03:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#8670
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?