[PR #25316] feat(web/server): synchronise/merge metadata of duplicate images; logic in server #18166

Open
opened 2026-02-05 16:33:46 +03:00 by OVERLORD · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/immich-app/immich/pull/25316
Author: @Phlogi
Created: 1/17/2026
Status: 🔄 Open

Base: mainHead: deduplicate-sync-album


📝 Commits (10+)

  • 19da67f feat(web): Synchronize information from deduplicated images
  • 621af87 Replace addAssetToAlbums with copyAsset
  • 8bea899 fix linter
  • 0408ba7 feat(web): add duplicate sync fields and fix typo
  • 7425c37 feat(web): add tag sync and enhance duplicate resolution
  • 29bfb35 feat(server): move duplicate resolution to backend with sync and stacking
  • 97d530d feat(preferences): enable all duplicate sync settings by default
  • 1f490e1 chore: clean up
  • b1dcd2d chore: clean up
  • f601164 refactor: rename & clean up

📊 Changes

42 files changed (+2540 additions, -160 deletions)

View changed files

docs/docs/features/duplicates-utility.md (+21 -0)
e2e/src/api/specs/duplicate.e2e-spec.ts (+651 -0)
📝 e2e/src/fixtures.ts (+2 -0)
📝 e2e/src/utils.ts (+3 -0)
📝 i18n/en.json (+4 -0)
📝 mobile/openapi/README.md (+4 -0)
📝 mobile/openapi/lib/api.dart (+3 -0)
📝 mobile/openapi/lib/api/duplicates_api.dart (+59 -0)
📝 mobile/openapi/lib/api_client.dart (+6 -0)
📝 mobile/openapi/lib/model/bulk_id_error_reason.dart (+3 -0)
📝 mobile/openapi/lib/model/bulk_id_response_dto.dart (+21 -1)
mobile/openapi/lib/model/duplicate_resolve_dto.dart (+118 -0)
mobile/openapi/lib/model/duplicate_resolve_group_dto.dart (+121 -0)
📝 mobile/openapi/lib/model/duplicate_response_dto.dart (+14 -3)
mobile/openapi/lib/model/duplicate_sync_settings_dto.dart (+147 -0)
📝 open-api/immich-openapi-specs.json (+122 -3)
📝 open-api/typescript-sdk/src/fetch-client.ts (+51 -2)
server/src/controllers/duplicate.controller.spec.ts (+47 -0)
📝 server/src/controllers/duplicate.controller.ts (+16 -3)
📝 server/src/dtos/asset-ids.response.dto.ts (+2 -0)

...and 22 more files

📄 Description

Description

This PR moves duplicate metadata synchronization logic from the frontend to the backend, making it more robust and enabling proper batch operations. This is an improved and refactored version of #13851.

What's new:

  • Server-side metadata sync - Albums, tags, favorites, ratings, descriptions, visibility, and location are now unconditionally synchronized on the backend when resolving duplicates. All metadata from trashed/deleted assets is merged into kept assets automatically — no per-field toggles or settings needed.
  • Batch resolution API - New /duplicates/resolve endpoint handles multiple duplicate groups in a single request
  • Smart keep suggestions - Server provides suggestions for which duplicates to keep based on file size and EXIF completeness (same logic as before)
  • Stack from duplicates - New /duplicates/stack endpoint to organize duplicates as stacks instead of deleting them
  • Tags survive re-extraction - Merged tags are written to asset_exif.tags and the property is locked, so subsequent metadata re-extraction (sidecar write → extract metadata) does not overwrite the merged tag set
  • Correct exif handling - Description and location are properly written to exif data (not asset update), and locked columns are handled by the existing distinctLocked mechanism without redundant wrappers

Not included (planned for follow-up):

  • Bulk "resolve all duplicates" with background job processing - this PR focuses on the core resolution API. If well-received, a follow-up PR will add POST /duplicates/resolve/bulk with job queue support for processing all duplicate groups in the background.

How Has This Been Tested?

  • Comprehensive E2E tests (70+ test cases covering all endpoints and edge cases)
  • Unit tests for metadata merge logic and keep suggestions algorithm
  • Tested all sync options (albums, tags, favorites, ratings, description, location, visibility)
  • Validated permission checks and error handling
  • Manual testing in web UI
  • External automatic testing with a python script and actual images

Screenshots (if appropriate)

image

API Changes

New endpoints:

  • POST /duplicates/resolve - Batch resolve with metadata sync and optional trash/delete
  • POST /duplicates/stack - Create stack from duplicate group assets

Modified endpoints:

  • GET /duplicates - Now includes suggestedKeepAssetIds field in response

Checklist:

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation if applicable - OpenAPI specs updated
  • I have no unrelated changes in the PR. - Only duplicate resolution features
  • I have confirmed that any new dependencies are strictly necessary. - No new dependencies
  • I have written tests for new code (if applicable) - 70+ E2E tests + unit tests + external python scripted tests
  • I have followed naming conventions/patterns in the surrounding code - Follows duplicate/asset patterns
  • All code in src/services/ uses repositories implementations for database calls, filesystem operations, etc. - DuplicateService uses repositories
  • All code in src/repositories/ is pretty basic/simple and does not have any immich specific logic (that belongs in src/services/) - Only data access

Additional external automated python test cases

Click to see the cases and reasoning

Duplicate Test Cases

This document enumerates every duplicate merge test set. Each set groups multiple assets for user Eve where one
or more assets are marked keep and the rest are marked trash to exercise merge
behavior. Each table lists the candidate assets and the expected result.
Total expected duplicate sets: 33.

Favorites (synchronizeFavorites)

Fields tested: favorite

Set Keep Trash Candidates Expected Result
Fav-01-NoneVsTrue favorite=false favorite=true favorite=true
Fav-03-AllFalse favorite=false favorite=false favorite=false
Fav-04-MultiMixed favorite=false favorite=false, true, false favorite=true

Rating (synchronizeRating)

Fields tested: rating

Set Keep Trash Candidates Expected Result
Rat-01-LowVsHigh rating=1 rating=4 rating=4
Rat-03-AllRange rating=0 rating=2, 3, 5 rating=5
Rat-04-ZeroVsZero rating=0 rating=0 rating=0

Tags (synchronizeTags)

Fields tested: tags

Set Keep Trash Candidates Expected Result
Tag-01-NoneVsOne tags=none tags=Beach tags=Beach
Tag-02-KeepHasTags tags=Travel, Food tags=none tags=Travel, Food
Tag-03-Overlap tags=Family, Travel tags=Travel, Sunset tags=Family, Travel, Sunset
Tag-04-MultiMixed tags=Work tags=Work, Party; tags=none; tags=Hiking tags=Work, Party, Hiking

Description (synchronizeDescription)

Fields tested: description (unique non-empty, trimmed)

Set Keep Trash Candidates Expected Result
Desc-01-EmptyVsFilled desc="" desc="A sunset photo" desc="A sunset photo"
Desc-02-FilledVsEmpty desc="Mountain view" desc="" desc="Mountain view"
Desc-03-BothFilled desc="First" desc="Second" desc="First\nSecond"
Desc-04-SameValues desc="Same caption" desc="Same caption" desc="Same caption"
Desc-05-FourMixed desc="Alpha" desc="Beta", "", "Gamma" desc="Alpha\nBeta\nGamma"
Desc-06-MultiLineTrim desc="Line A\nLine B" desc=" Line B \nLine C " desc="Line A\nLine B\nLine C"

Location (synchronizeLocation)

Fields tested: location

Set Keep Trash Candidates Expected Result
Loc-01-NoneVsOne location=null location=NYC location=NYC
Loc-02-OneVsNone location=Paris location=null location=Paris
Loc-03-BothSame location=London location=London location=London
Loc-04-Different location=London location=Tokyo location=London
Loc-05-ThreeDifferent location=NYC location=Paris, London location=NYC
Loc-06-MixedWithNull location=NYC location=null, London location=NYC
Loc-07-AllNull location=null location=null, null location=null

Albums (synchronizeAlbums)

Fields tested: album membership

Set Keep Trash Candidates Expected Result
Album-01-TrashHasAlbum album=none album=TestAlbum-A albums=TestAlbum-A
Album-02-KeepHasAlbum album=TestAlbum-B album=none albums=TestAlbum-B
Album-03-DifferentAlbums album=TestAlbum-C album=TestAlbum-D albums=TestAlbum-C, TestAlbum-D
Album-04-MultipleTrashAlbums album=none album=TestAlbum-E, TestAlbum-F albums=TestAlbum-E, TestAlbum-F

Combined (multiple sync options)

Fields tested: favorite, rating, description, location, album

Set Keep Trash Candidates Expected Result
Combo-01-FavRating favorite=false, rating=1 favorite=true rating=3; favorite=false rating=5 favorite=true; rating=5
Combo-03-AllFields favorite=false rating=1 desc="A" location=NYC favorite=true rating=3 desc="B" location=null; favorite=false rating=5 desc="" location=null favorite=true; rating=5; desc="A\nB"; location=NYC
Combo-04-AllWithAlbum favorite=false rating=2 album=TestAlbum-G favorite=true rating=4 album=TestAlbum-H favorite=true; rating=4; albums=TestAlbum-G, TestAlbum-H

Multi-Keep (multiple keepers)

Fields tested: merged metadata applies to every kept asset
Note: Hidden assets are excluded because they do not show up in the duplicate review UI.

Set Keep Trash Candidates Expected Result
Keep-01-FavRating favorite=false rating=1; favorite=true rating=2 favorite=false rating=5 favorite=true; rating=5
Keep-02-TagsAlbums tags=Work album=TestAlbum-MK-A; tags=Travel album=none tags=Party album=TestAlbum-MK-B tags=Work, Travel, Party; albums=TestAlbum-MK-A, TestAlbum-MK-B
Keep-03-VisDescLoc visibility=timeline desc="Alpha" location=NYC; visibility=archive desc="Beta" location=null visibility=archive desc="" location=null visibility=archive; desc="Alpha\nBeta"; location=NYC

Screenshots

image image image

Please describe to which degree, if any, an LLM was used in creating this pull request.

Claude Code assisted with refactoring the sync logic to the backend, writing test cases, and cleaning up the frontend code. All logic was manually reviewed and tested.


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/immich-app/immich/pull/25316 **Author:** [@Phlogi](https://github.com/Phlogi) **Created:** 1/17/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `deduplicate-sync-album` --- ### 📝 Commits (10+) - [`19da67f`](https://github.com/immich-app/immich/commit/19da67f064f925baea2d25c8197016d20b7c5476) feat(web): Synchronize information from deduplicated images - [`621af87`](https://github.com/immich-app/immich/commit/621af87888cea0bd0af7f5de806a2ad6061676c3) Replace addAssetToAlbums with copyAsset - [`8bea899`](https://github.com/immich-app/immich/commit/8bea89967c84c968d3af1150c1eb1fc18678f1c4) fix linter - [`0408ba7`](https://github.com/immich-app/immich/commit/0408ba76a82cdee9534cb4284a31a27a19888754) feat(web): add duplicate sync fields and fix typo - [`7425c37`](https://github.com/immich-app/immich/commit/7425c377137820d1f6c07604a460c2e44b319e3c) feat(web): add tag sync and enhance duplicate resolution - [`29bfb35`](https://github.com/immich-app/immich/commit/29bfb355fef03632b7baff91a860aff868bb1727) feat(server): move duplicate resolution to backend with sync and stacking - [`97d530d`](https://github.com/immich-app/immich/commit/97d530d5802eeeaf998802ccdeef9f71413e7bd0) feat(preferences): enable all duplicate sync settings by default - [`1f490e1`](https://github.com/immich-app/immich/commit/1f490e1fa91b8f14b957bab42a50e9f05c01681b) chore: clean up - [`b1dcd2d`](https://github.com/immich-app/immich/commit/b1dcd2d963c51076ba04fbbd0383911c5941307c) chore: clean up - [`f601164`](https://github.com/immich-app/immich/commit/f601164a454a3204dbdb8dd123a0e750f64949d1) refactor: rename & clean up ### 📊 Changes **42 files changed** (+2540 additions, -160 deletions) <details> <summary>View changed files</summary> ➕ `docs/docs/features/duplicates-utility.md` (+21 -0) ➕ `e2e/src/api/specs/duplicate.e2e-spec.ts` (+651 -0) 📝 `e2e/src/fixtures.ts` (+2 -0) 📝 `e2e/src/utils.ts` (+3 -0) 📝 `i18n/en.json` (+4 -0) 📝 `mobile/openapi/README.md` (+4 -0) 📝 `mobile/openapi/lib/api.dart` (+3 -0) 📝 `mobile/openapi/lib/api/duplicates_api.dart` (+59 -0) 📝 `mobile/openapi/lib/api_client.dart` (+6 -0) 📝 `mobile/openapi/lib/model/bulk_id_error_reason.dart` (+3 -0) 📝 `mobile/openapi/lib/model/bulk_id_response_dto.dart` (+21 -1) ➕ `mobile/openapi/lib/model/duplicate_resolve_dto.dart` (+118 -0) ➕ `mobile/openapi/lib/model/duplicate_resolve_group_dto.dart` (+121 -0) 📝 `mobile/openapi/lib/model/duplicate_response_dto.dart` (+14 -3) ➕ `mobile/openapi/lib/model/duplicate_sync_settings_dto.dart` (+147 -0) 📝 `open-api/immich-openapi-specs.json` (+122 -3) 📝 `open-api/typescript-sdk/src/fetch-client.ts` (+51 -2) ➕ `server/src/controllers/duplicate.controller.spec.ts` (+47 -0) 📝 `server/src/controllers/duplicate.controller.ts` (+16 -3) 📝 `server/src/dtos/asset-ids.response.dto.ts` (+2 -0) _...and 22 more files_ </details> ### 📄 Description ## Description This PR moves duplicate metadata synchronization logic from the frontend to the backend, making it more robust and enabling proper batch operations. This is an improved and refactored version of #13851. **What's new:** - **Server-side metadata sync** - Albums, tags, favorites, ratings, descriptions, visibility, and location are now unconditionally synchronized on the backend when resolving duplicates. All metadata from trashed/deleted assets is merged into kept assets automatically — no per-field toggles or settings needed. - **Batch resolution API** - New `/duplicates/resolve` endpoint handles multiple duplicate groups in a single request - **Smart keep suggestions** - Server provides suggestions for which duplicates to keep based on file size and EXIF completeness (same logic as before) - **Stack from duplicates** - New `/duplicates/stack` endpoint to organize duplicates as stacks instead of deleting them - **Tags survive re-extraction** - Merged tags are written to `asset_exif.tags` and the property is locked, so subsequent metadata re-extraction (sidecar write → extract metadata) does not overwrite the merged tag set - **Correct exif handling** - Description and location are properly written to exif data (not asset update), and locked columns are handled by the existing `distinctLocked` mechanism without redundant wrappers **Not included (planned for follow-up):** - Bulk "resolve all duplicates" with background job processing - this PR focuses on the core resolution API. If well-received, a follow-up PR will add `POST /duplicates/resolve/bulk` with job queue support for processing all duplicate groups in the background. ## How Has This Been Tested? - [x] Comprehensive E2E tests (70+ test cases covering all endpoints and edge cases) - [x] Unit tests for metadata merge logic and keep suggestions algorithm - [x] Tested all sync options (albums, tags, favorites, ratings, description, location, visibility) - [x] Validated permission checks and error handling - [x] Manual testing in web UI - [x] External automatic testing with a python script and actual images <details><summary><h2>Screenshots (if appropriate)</h2></summary> <!-- Images go below this line. --> <img width="466" height="532" alt="image" src="https://github.com/user-attachments/assets/113dd6d2-0d33-463a-8129-8187d1c79e44" /> </details> ## API Changes **New endpoints:** - `POST /duplicates/resolve` - Batch resolve with metadata sync and optional trash/delete - `POST /duplicates/stack` - Create stack from duplicate group assets **Modified endpoints:** - `GET /duplicates` - Now includes `suggestedKeepAssetIds` field in response ## Checklist: - [x] I have performed a self-review of my own code - [x] I have made corresponding changes to the documentation if applicable - _OpenAPI specs updated_ - [x] I have no unrelated changes in the PR. - _Only duplicate resolution features_ - [x] I have confirmed that any new dependencies are strictly necessary. - _No new dependencies_ - [x] I have written tests for new code (if applicable) - _70+ E2E tests + unit tests + external python scripted tests_ - [x] I have followed naming conventions/patterns in the surrounding code - _Follows duplicate/asset patterns_ - [x] All code in `src/services/` uses repositories implementations for database calls, filesystem operations, etc. - _DuplicateService uses repositories_ - [x] All code in `src/repositories/` is pretty basic/simple and does not have any immich specific logic (that belongs in `src/services/`) - _Only data access_ ## Additional external automated python test cases <details><summary>Click to see the cases and reasoning</summary> <p> <detail> <summary></summary> # Duplicate Test Cases This document enumerates every duplicate merge test set. Each set groups multiple assets for user Eve where one or more assets are marked **keep** and the rest are marked **trash** to exercise merge behavior. Each table lists the candidate assets and the expected result. Total expected duplicate sets: 33. ## Favorites (synchronizeFavorites) Fields tested: favorite | Set | Keep | Trash Candidates | Expected Result | | ----------------- | -------------- | --------------------------- | --------------- | | Fav-01-NoneVsTrue | favorite=false | favorite=true | favorite=true | | Fav-03-AllFalse | favorite=false | favorite=false | favorite=false | | Fav-04-MultiMixed | favorite=false | favorite=false, true, false | favorite=true | ## Rating (synchronizeRating) Fields tested: rating | Set | Keep | Trash Candidates | Expected Result | | ----------------- | -------- | ---------------- | --------------- | | Rat-01-LowVsHigh | rating=1 | rating=4 | rating=4 | | Rat-03-AllRange | rating=0 | rating=2, 3, 5 | rating=5 | | Rat-04-ZeroVsZero | rating=0 | rating=0 | rating=0 | ## Tags (synchronizeTags) Fields tested: tags | Set | Keep | Trash Candidates | Expected Result | | ------------------ | ------------------- | ---------------------------------------- | --------------------------- | | Tag-01-NoneVsOne | tags=none | tags=Beach | tags=Beach | | Tag-02-KeepHasTags | tags=Travel, Food | tags=none | tags=Travel, Food | | Tag-03-Overlap | tags=Family, Travel | tags=Travel, Sunset | tags=Family, Travel, Sunset | | Tag-04-MultiMixed | tags=Work | tags=Work, Party; tags=none; tags=Hiking | tags=Work, Party, Hiking | ## Description (synchronizeDescription) Fields tested: description (unique non-empty, trimmed) | Set | Keep | Trash Candidates | Expected Result | | --------------------- | --------------------- | ------------------------ | ----------------------------- | | Desc-01-EmptyVsFilled | desc="" | desc="A sunset photo" | desc="A sunset photo" | | Desc-02-FilledVsEmpty | desc="Mountain view" | desc="" | desc="Mountain view" | | Desc-03-BothFilled | desc="First" | desc="Second" | desc="First\nSecond" | | Desc-04-SameValues | desc="Same caption" | desc="Same caption" | desc="Same caption" | | Desc-05-FourMixed | desc="Alpha" | desc="Beta", "", "Gamma" | desc="Alpha\nBeta\nGamma" | | Desc-06-MultiLineTrim | desc="Line A\nLine B" | desc=" Line B \nLine C " | desc="Line A\nLine B\nLine C" | ## Location (synchronizeLocation) Fields tested: location | Set | Keep | Trash Candidates | Expected Result | | --------------------- | --------------- | ---------------------- | --------------- | | Loc-01-NoneVsOne | location=null | location=NYC | location=NYC | | Loc-02-OneVsNone | location=Paris | location=null | location=Paris | | Loc-03-BothSame | location=London | location=London | location=London | | Loc-04-Different | location=London | location=Tokyo | location=London | | Loc-05-ThreeDifferent | location=NYC | location=Paris, London | location=NYC | | Loc-06-MixedWithNull | location=NYC | location=null, London | location=NYC | | Loc-07-AllNull | location=null | location=null, null | location=null | ## Albums (synchronizeAlbums) Fields tested: album membership | Set | Keep | Trash Candidates | Expected Result | | ---------------------------- | ----------------- | ------------------------------ | ------------------------------- | | Album-01-TrashHasAlbum | album=none | album=TestAlbum-A | albums=TestAlbum-A | | Album-02-KeepHasAlbum | album=TestAlbum-B | album=none | albums=TestAlbum-B | | Album-03-DifferentAlbums | album=TestAlbum-C | album=TestAlbum-D | albums=TestAlbum-C, TestAlbum-D | | Album-04-MultipleTrashAlbums | album=none | album=TestAlbum-E, TestAlbum-F | albums=TestAlbum-E, TestAlbum-F | ## Combined (multiple sync options) Fields tested: favorite, rating, description, location, album | Set | Keep | Trash Candidates | Expected Result | | --------------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | Combo-01-FavRating | favorite=false, rating=1 | favorite=true rating=3; favorite=false rating=5 | favorite=true; rating=5 | | Combo-03-AllFields | favorite=false rating=1 desc="A" location=NYC | favorite=true rating=3 desc="B" location=null; favorite=false rating=5 desc="" location=null | favorite=true; rating=5; desc="A\nB"; location=NYC | | Combo-04-AllWithAlbum | favorite=false rating=2 album=TestAlbum-G | favorite=true rating=4 album=TestAlbum-H | favorite=true; rating=4; albums=TestAlbum-G, TestAlbum-H | ## Multi-Keep (multiple keepers) Fields tested: merged metadata applies to every kept asset Note: Hidden assets are excluded because they do not show up in the duplicate review UI. | Set | Keep | Trash Candidates | Expected Result | | ------------------ | ------------------------------------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------- | | Keep-01-FavRating | favorite=false rating=1; favorite=true rating=2 | favorite=false rating=5 | favorite=true; rating=5 | | Keep-02-TagsAlbums | tags=Work album=TestAlbum-MK-A; tags=Travel album=none | tags=Party album=TestAlbum-MK-B | tags=Work, Travel, Party; albums=TestAlbum-MK-A, TestAlbum-MK-B | | Keep-03-VisDescLoc | visibility=timeline desc="Alpha" location=NYC; visibility=archive desc="Beta" location=null | visibility=archive desc="" location=null | visibility=archive; desc="Alpha\nBeta"; location=NYC | # Screenshots <img width="1389" height="675" alt="image" src="https://github.com/user-attachments/assets/bca6381c-0c85-41ca-8f4a-ca2956d9c180" /> <img width="1194" height="524" alt="image" src="https://github.com/user-attachments/assets/209fc727-fd00-4fc5-abe6-25ddebf50f9d" /> <img width="859" height="517" alt="image" src="https://github.com/user-attachments/assets/1bb01735-6a24-4d23-8cfe-e34ff5dca6fd" /> </p> </details> ## Please describe to which degree, if any, an LLM was used in creating this pull request. Claude Code assisted with refactoring the sync logic to the backend, writing test cases, and cleaning up the frontend code. All logic was manually reviewed and tested. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
OVERLORD added the pull-request label 2026-02-05 16:33:46 +03:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#18166