immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 08:41:28 +03:00
Code Issues 27 Packages Projects Releases 10 Wiki Activity

[BUG] Data-loss-potential: Live video entities "originalPath" is incorrect #1881

New Issue
Closed
opened 2026-02-05 04:17:55 +03:00 by OVERLORD · 5 comments
OVERLORD commented 2026-02-05 04:17:55 +03:00
Owner
Copy Link

Originally created by @truongsinh on GitHub (Dec 26, 2023).

The bug

Notice, I personally think this is a critical bug that can result in data loss

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.91.4

Version of Immich Mobile App

v1.91.4 (Android)

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:

  syncthing:
    image: syncthing/syncthing
    container_name: syncthing
    hostname: starm-syncthing
    environment:
    #   - PUID=1000
    #   - PGID=1000
      - PUID=0
      - PGID=0
    volumes:
      - /root/docker-data/syncthing/config:/var/syncthing/config
      - /root/docker-data/syncthing/Sync:/var/syncthing/Sync
      - ${UPLOAD_LOCATION}/library:/var/syncthing/library
    ports:
      - 8384:8384 # Web UI
      - 22000:22000/tcp # TCP file transfers
      - 22000:22000/udp # QUIC file transfers
      - 21027:21027/udp # Receive local discovery broadcasts
    restart: unless-stopped
   
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION}
    command: ["start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    ports:
      - 2283:3001    

    depends_on:
      - redis
      - database
    restart: always
    networks:
      - caddy

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION}
    command: ["start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /mnt/ffmpeg.sh:/usr/bin/ffmpeg:ro
      - /home/truongsinh/.ssh:/root/.ssh:ro
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - caddy

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - stack.env
    environment:
      - NODE_ENV=production
    restart: always
    networks:
      - caddy

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always
    ports:
      - 6379:6379
    networks:
      - caddy

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    env_file:
      - stack.env
    networks:
      - caddy
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
    ports:
      - 5432:5432

volumes:
  pgdata:
  model-cache:

networks:
  caddy:
    external: true

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=/srv/btrfs/docker/immich-user-data
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
IMMICH_VERSION=v1.91.4
UPLOAD_LOCATION_OLD=/srv/mergerfs/genesis

Reproduction steps

1. Use Android, take a picture with "Top shot"
2. Sync to Immich
3. Query the image entity to get its live photo id, then query its "originalPath" and, "originalFileName"
or just simply (assuming "349425fc-1ca5-4ced-a5bb-18576e753d90" is the ID of the photo):



SELECT id, "originalPath", "encodedVideoPath", "originalFileName"
	FROM public.assets
	where 
	id=(SELECT "livePhotoVideoId" from public.assets where id='349425fc-1ca5-4ced-a5bb-18576e753d90')
	
	;


## Expected behavior:
`originalPath` is in `library` subfolder

## Observed behaviour, `originalPath` is in `encoded-video` subfolder

Additional information

Anything NOT in library is supposed to be derivative/generated/transient (enconded-library, thumbs, upload). Having original data inside enconded-library is a critical bug that can definitely result in data loss.

Originally created by @truongsinh on GitHub (Dec 26, 2023). ### The bug **Notice, I personally think this is a critical bug that can result in data loss** ### The OS that Immich Server is running on Debian ### Version of Immich Server v1.91.4 ### Version of Immich Mobile App v1.91.4 (Android) ### Platform with the issue - [X] Server - [ ] Web - [X] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: syncthing: image: syncthing/syncthing container_name: syncthing hostname: starm-syncthing environment: # - PUID=1000 # - PGID=1000 - PUID=0 - PGID=0 volumes: - /root/docker-data/syncthing/config:/var/syncthing/config - /root/docker-data/syncthing/Sync:/var/syncthing/Sync - ${UPLOAD_LOCATION}/library:/var/syncthing/library ports: - 8384:8384 # Web UI - 22000:22000/tcp # TCP file transfers - 22000:22000/udp # QUIC file transfers - 21027:21027/udp # Receive local discovery broadcasts restart: unless-stopped immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION} command: ["start-server.sh"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload env_file: - stack.env environment: - NODE_ENV=production ports: - 2283:3001 depends_on: - redis - database restart: always networks: - caddy immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION} command: ["start-microservices.sh"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /mnt/ffmpeg.sh:/usr/bin/ffmpeg:ro - /home/truongsinh/.ssh:/root/.ssh:ro env_file: - stack.env environment: - NODE_ENV=production depends_on: - redis - database restart: always networks: - caddy immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION} volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - model-cache:/cache env_file: - stack.env environment: - NODE_ENV=production restart: always networks: - caddy redis: container_name: immich_redis image: redis:6.2 restart: always ports: - 6379:6379 networks: - caddy database: container_name: immich_postgres image: tensorchord/pgvecto-rs:pg14-v0.1.11 env_file: - stack.env networks: - caddy environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} PG_DATA: /var/lib/postgresql/data volumes: - pgdata:/var/lib/postgresql/data restart: always ports: - 5432:5432 volumes: pgdata: model-cache: networks: caddy: external: true ``` ### Your .env content ```Shell DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_PASSWORD=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis UPLOAD_LOCATION=/srv/btrfs/docker/immich-user-data PUBLIC_LOGIN_PAGE_MESSAGE= IMMICH_WEB_URL=http://immich-web:3000 IMMICH_SERVER_URL=http://immich-server:3001 IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003 IMMICH_VERSION=v1.91.4 UPLOAD_LOCATION_OLD=/srv/mergerfs/genesis ``` ### Reproduction steps ```bash 1. Use Android, take a picture with "Top shot" 2. Sync to Immich 3. Query the image entity to get its live photo id, then query its "originalPath" and, "originalFileName" or just simply (assuming "349425fc-1ca5-4ced-a5bb-18576e753d90" is the ID of the photo): SELECT id, "originalPath", "encodedVideoPath", "originalFileName" FROM public.assets where id=(SELECT "livePhotoVideoId" from public.assets where id='349425fc-1ca5-4ced-a5bb-18576e753d90') ; ## Expected behavior: `originalPath` is in `library` subfolder ## Observed behaviour, `originalPath` is in `encoded-video` subfolder ``` ### Additional information Anything NOT in `library` is supposed to be derivative/generated/transient (enconded-library, thumbs, upload). Having original data inside `enconded-library` is a critical bug that can definitely result in data loss.
OVERLORD commented 2026-02-05 04:18:00 +03:00
Author
Owner
Copy Link

@truongsinh commented on GitHub (Dec 26, 2023):

could be related to https://github.com/immich-app/immich/issues/4444

@truongsinh commented on GitHub (Dec 26, 2023): could be related to https://github.com/immich-app/immich/issues/4444
OVERLORD commented 2026-02-05 04:18:03 +03:00
Author
Owner
Copy Link

@danieldietzler commented on GitHub (Dec 26, 2023):

Hey, this is not a bug. Live photos are a bit confusing though, I'll try to outlie it:

iOS and Android store live photos differently. iOS has literally 2 files, a still and a motion part. Both are linked by the livePhoto metadata tags.

Android has one file (a jpg) and stores the video within that (kinda weird, I know). If you want to look at how it's done, you can check it out with something like exiftool.

In both cases, just like we do with any other media, we generate encoded files from that. Thus, both the motion and the still part will be a file ("the apple approach" if you like). However, for Android there is only one original file, because that contains the video as well. Therefore, the video is in fact only an encoded video (thus in the respective folder), because there is no "original" video file.

@danieldietzler commented on GitHub (Dec 26, 2023): Hey, this is *not* a bug. Live photos are a bit confusing though, I'll try to outlie it: iOS and Android store live photos differently. iOS has literally 2 files, a still and a motion part. Both are linked by the livePhoto metadata tags. Android has one file (a jpg) and stores the video within that (kinda weird, I know). If you want to look at how it's done, you can check it out with something like exiftool. In both cases, just like we do with any other media, we generate encoded files from that. Thus, both the motion and the still part will be a file ("the apple approach" if you like). However, for Android there is only *one* original file, because that contains the video as well. Therefore, the video is in fact only an encoded video (thus in the respective folder), because there is no "original" video file.
OVERLORD commented 2026-02-05 04:18:06 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Dec 26, 2023):

Like @danieldietzler said, this file is indeed generated and can be recreated from the original, which is stored in the library folder.

@jrasm91 commented on GitHub (Dec 26, 2023): Like @danieldietzler said, this file is indeed generated and can be recreated from the original, which is stored in the library folder.
OVERLORD commented 2026-02-05 04:18:09 +03:00
Author
Owner
Copy Link

@truongsinh commented on GitHub (Dec 26, 2023):

Thanks @jrasm91 and @danieldietzler , that's a relief. (let me know if this follow-up should be a new github issue) Though, if this is the case, how do I re-generate live-photo asset from exisiting jpeg file? Currently, when running repair, I have lots of orphans like this:

    {
        "entityId": "0b6ce070-2eb1-4645-be2e-6792de7413ac",
        "entityType": "asset",
        "checksum": "4AbUK2PWV8XumktykSVOL86K6Mw=",
        "pathType": "original",
        "pathValue": "/usr/src/app/upload/encoded-video/8b114f63-4094-41cf-824c-192f85650c30/d0/da/d0da817d-1edd-488c-90ab-0fd509ab8dd7-MP.mp4"
    },

I have confirm that the asset to the original photo exist in the DB (by the following query)

SELECT id, "deviceAssetId", "ownerId", "deviceId", type, "originalPath", "resizePath", "fileCreatedAt", "fileModifiedAt", "isFavorite", duration, "webpPath", "encodedVideoPath", checksum, "isVisible", "livePhotoVideoId", "updatedAt", "createdAt", "isArchived", "originalFileName", "sidecarPath", "isReadOnly", thumbhash, "isOffline", "libraryId", "isExternal", "deletedAt", "localDateTime", "stackParentId"
	FROM public.assets
	
	where "livePhotoVideoId" in( SELECT  id  FROM public.assets where id = '0b6ce070-2eb1-4645-be2e-6792de7413ac') 
	limit 10
	;

and that there are no reported orphans of those images; all orphans are of "pathType": "original" with path %encoded-video%. I have run GENERATE THUMBNAILS & TRANSCODE VIDEOS for "All", but nothing changes.

@truongsinh commented on GitHub (Dec 26, 2023): Thanks @jrasm91 and @danieldietzler , that's a relief. (let me know if this follow-up should be a new github issue) Though, if this is the case, how do I re-generate live-photo asset from exisiting jpeg file? Currently, when running repair, I have lots of `orphans` like this: ``` { "entityId": "0b6ce070-2eb1-4645-be2e-6792de7413ac", "entityType": "asset", "checksum": "4AbUK2PWV8XumktykSVOL86K6Mw=", "pathType": "original", "pathValue": "/usr/src/app/upload/encoded-video/8b114f63-4094-41cf-824c-192f85650c30/d0/da/d0da817d-1edd-488c-90ab-0fd509ab8dd7-MP.mp4" }, ``` I have confirm that the asset to the original photo exist in the DB (by the following query) ```SQL SELECT id, "deviceAssetId", "ownerId", "deviceId", type, "originalPath", "resizePath", "fileCreatedAt", "fileModifiedAt", "isFavorite", duration, "webpPath", "encodedVideoPath", checksum, "isVisible", "livePhotoVideoId", "updatedAt", "createdAt", "isArchived", "originalFileName", "sidecarPath", "isReadOnly", thumbhash, "isOffline", "libraryId", "isExternal", "deletedAt", "localDateTime", "stackParentId" FROM public.assets where "livePhotoVideoId" in( SELECT id FROM public.assets where id = '0b6ce070-2eb1-4645-be2e-6792de7413ac') limit 10 ; ``` and that there are no reported orphans of those images; all orphans are of `"pathType": "original"` with path `%encoded-video%`. I have run GENERATE THUMBNAILS & TRANSCODE VIDEOS for "All", but nothing changes.
OVERLORD commented 2026-02-05 04:18:12 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Dec 26, 2023):

Yeah there are some bugs/outstanding issues with regenerating those files right now.

@jrasm91 commented on GitHub (Dec 26, 2023): Yeah there are some bugs/outstanding issues with regenerating those files right now.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1881
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?