Images appear to be randomly corrupted (at least on RK build) with no logs or known reason #8129

Open
opened 2026-02-05 13:32:53 +03:00 by OVERLORD · 7 comments
Owner

Originally created by @platima on GitHub (Dec 25, 2025).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • Yes

The bug

I'm not 100% on the cause in the title, but it seems likely due to how the files present.

Using the same two CR2 files as a test, and repeatedly uploading, they break about 50% of the time.

The broken ones, when downloaded, still have a correct preview image, but the RAW/CR2 image data appears corrupted:

I went through the .JPG copies of all the same photos (as my camera was set to save both) and there's no issues.

Strange!

Happy Holidays everyone regardless 😊

EDIT: This is happening with some JPGs too.

The OS that Immich Server is running on

Ubuntu 24.04.3 LTS (Josh Riek)

Version of Immich Server

v2.4.1-rknn

Version of Immich Mobile App

N/A

Platform with the issue

  • Server
  • Web
  • Mobile

Device make and model

ArmSoM Sige7 / Banana Pi BPi-M7 (RK3588)

Your docker-compose.yml content

name: immich

networks:
  netdev-frigate:
    external: true
  immich-internal:
    driver: bridge
    
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    security_opt: # enables full access to /sys and /proc, still far better than privileged: true
      - systempaths=unconfined
      - apparmor=unconfined
    group_add:
      - video
    devices:
      - /dev/rga:/dev/rga
      - /dev/dri:/dev/dri
      - /dev/dma_heap:/dev/dma_heap
      - /dev/mpp_service:/dev/mpp_service
      - /dev/mali0:/dev/mali0 # only required to enable OpenCL-accelerated HDR -> SDR tonemapping
    volumes:
      - /etc/OpenCL:/etc/OpenCL:ro # only required to enable OpenCL-accelerated HDR -> SDR tonemapping
      - /usr/lib/aarch64-linux-gnu/libmali-x11/libmali-valhall-g610-g13p0-x11-wayland-gbm.so:/usr/lib/aarch64-linux-gnu/libmali.so.1:ro # only required to enable OpenCL-accelerated HDR -> SDR tonemapping
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/data
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    dns:
      - 1.1.1.1
      - 8.8.8.8
    networks:
      netdev-frigate:
        ipv4_address: 10.12.20.88
      immich-internal:
    #ports:
    #  - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-rknn
    security_opt:
      - systempaths=unconfined
      - apparmor=unconfined
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    networks:
      - immich-internal
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8@sha256:81db6d39e1bba3b3ff32bd3a1b19a6d69690f94a3954ec131277b9a26b95b3aa
    healthcheck:
      test: redis-cli ping || exit 1
    networks:
      - immich-internal
    restart: always

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    networks:
      - immich-internal
    shm_size: 128mb
    restart: always

volumes:
  model-cache:

Your .env content

UPLOAD_LOCATION=/data/off-host/docker/immich/library
DB_DATA_LOCATION=/data/off-host/immich/database
IMMICH_VERSION=v2
DB_PASSWORD=postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
MACHINE_LEARNING_RKNN_THREADS=3
IMMICH_PORT=80

Reproduction steps

  1. Upload 'good' CR2 images to Immich (via web UI in my case)
  2. Refresh - some might be broken
  3. Delete and retry if not

Relevant log output


Additional information

No response

Originally created by @platima on GitHub (Dec 25, 2025). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug I'm not 100% on the cause in the title, but it seems likely due to how the files present. Using the same two CR2 files as a test, and repeatedly uploading, they break about 50% of the time. The broken ones, when downloaded, still have a correct preview image, but the RAW/CR2 image data appears corrupted: - https://plati.ma/wp-content/uploads/2025/12/Test-Files-Good.zip - https://plati.ma/wp-content/uploads/2025/12/Test-Files-Broken.zip I went through the .JPG copies of all the same photos (as my camera was set to save both) and there's no issues. Strange! Happy Holidays everyone regardless 😊 EDIT: This is happening with some JPGs too. ### The OS that Immich Server is running on Ubuntu 24.04.3 LTS (Josh Riek) ### Version of Immich Server v2.4.1-rknn ### Version of Immich Mobile App N/A ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Device make and model ArmSoM Sige7 / Banana Pi BPi-M7 (RK3588) ### Your docker-compose.yml content ```YAML name: immich networks: netdev-frigate: external: true immich-internal: driver: bridge services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} security_opt: # enables full access to /sys and /proc, still far better than privileged: true - systempaths=unconfined - apparmor=unconfined group_add: - video devices: - /dev/rga:/dev/rga - /dev/dri:/dev/dri - /dev/dma_heap:/dev/dma_heap - /dev/mpp_service:/dev/mpp_service - /dev/mali0:/dev/mali0 # only required to enable OpenCL-accelerated HDR -> SDR tonemapping volumes: - /etc/OpenCL:/etc/OpenCL:ro # only required to enable OpenCL-accelerated HDR -> SDR tonemapping - /usr/lib/aarch64-linux-gnu/libmali-x11/libmali-valhall-g610-g13p0-x11-wayland-gbm.so:/usr/lib/aarch64-linux-gnu/libmali.so.1:ro # only required to enable OpenCL-accelerated HDR -> SDR tonemapping # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/data - /etc/localtime:/etc/localtime:ro env_file: - stack.env dns: - 1.1.1.1 - 8.8.8.8 networks: netdev-frigate: ipv4_address: 10.12.20.88 immich-internal: #ports: # - '2283:2283' depends_on: - redis - database restart: always healthcheck: disable: false immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-rknn security_opt: - systempaths=unconfined - apparmor=unconfined devices: - /dev/dri:/dev/dri volumes: - model-cache:/cache env_file: - stack.env networks: - immich-internal restart: always healthcheck: disable: false redis: container_name: immich_redis image: docker.io/valkey/valkey:8@sha256:81db6d39e1bba3b3ff32bd3a1b19a6d69690f94a3954ec131277b9a26b95b3aa healthcheck: test: redis-cli ping || exit 1 networks: - immich-internal restart: always database: container_name: immich_postgres image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: '--data-checksums' # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs # DB_STORAGE_TYPE: 'HDD' volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data networks: - immich-internal shm_size: 128mb restart: always volumes: model-cache: ``` ### Your .env content ```Shell UPLOAD_LOCATION=/data/off-host/docker/immich/library DB_DATA_LOCATION=/data/off-host/immich/database IMMICH_VERSION=v2 DB_PASSWORD=postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich MACHINE_LEARNING_RKNN_THREADS=3 IMMICH_PORT=80 ``` ### Reproduction steps 1. Upload 'good' CR2 images to Immich (via web UI in my case) 2. Refresh - some might be broken 3. Delete and retry if not ### Relevant log output ```shell ``` ### Additional information _No response_
Author
Owner

@bo0tzz commented on GitHub (Dec 25, 2025):

Can you specify more clearly what you mean by broken? Are you using the "prefer embedded thumbnails" setting and have you tried changing that?

@bo0tzz commented on GitHub (Dec 25, 2025): Can you specify more clearly what you mean by broken? Are you using the "prefer embedded thumbnails" setting and have you tried changing that?
Author
Owner

@platima commented on GitHub (Dec 26, 2025):

Can you specify more clearly what you mean by broken? Are you using the "prefer embedded thumbnails" setting and have you tried changing that?

Sorry that's why I linked to the ZIP files. They are completely corrupted after being uploaded to Immich:

Image

This corruption is present on the image when viewed in Immich after uploading, when downloaded from Immich and viewed locally, and then even when manually copied off the filesystem. The filesystem itself is perfectly fine though, and it's only sometimes when you upload a CR2, not every time.

The embedded thumbnail is not broken at all though. I am just thinking it might be related.

I'm guessing you meant "Prefer embedded preview"? If so, that's turned off, but should not matter as the preview is unharmed.

@platima commented on GitHub (Dec 26, 2025): > Can you specify more clearly what you mean by broken? Are you using the "prefer embedded thumbnails" setting and have you tried changing that? Sorry that's why I linked to the ZIP files. They are completely corrupted after being uploaded to Immich: <img width="2862" height="1536" alt="Image" src="https://github.com/user-attachments/assets/d9d6899f-4f42-415b-8b6f-5336931e2403" /> This corruption is present on the image when viewed in Immich after uploading, when downloaded from Immich and viewed locally, and then even when manually copied off the filesystem. The filesystem itself is perfectly fine though, and it's only sometimes when you upload a CR2, not every time. The embedded thumbnail is not broken at all though. I am just thinking it might be related. I'm guessing you meant "Prefer embedded preview"? If so, that's turned off, but should not matter as the preview is unharmed.
Author
Owner

@platima commented on GitHub (Dec 26, 2025):

UPDATE: I've found some JPEG files are being corrupted too:

Image
@platima commented on GitHub (Dec 26, 2025): UPDATE: I've found some JPEG files are being corrupted too: <img width="3840" height="2088" alt="Image" src="https://github.com/user-attachments/assets/1d8a9ed4-2890-4e57-b9c9-fd18d86ba349" />
Author
Owner

@platima commented on GitHub (Jan 4, 2026):

Can you specify more clearly what you mean by broken? Are you using the "prefer embedded thumbnails" setting and have you tried changing that?

Hey happy new years, just making sure you saw the updates 😅

@platima commented on GitHub (Jan 4, 2026): > Can you specify more clearly what you mean by broken? Are you using the "prefer embedded thumbnails" setting and have you tried changing that? Hey happy new years, just making sure you saw the updates 😅
Author
Owner

@mertalev commented on GitHub (Jan 4, 2026):

I wonder if the storage medium is dying or faulty? The preview having artifacts is one thing, but the original file definitely should not be corrupt in any way beyond what was uploaded.

@mertalev commented on GitHub (Jan 4, 2026): I wonder if the storage medium is dying or faulty? The preview having artifacts is one thing, but the original file definitely should not be corrupt in any way beyond what was uploaded.
Author
Owner

@platima commented on GitHub (Jan 4, 2026):

No storage medium issues - brand new M.2 SSD, and smartctl shows it's happy and healthy.

Also I can copy 500 photos to the SSD, and back again, without corruption. It's only when I upload via Immich that some become corrupted.

Whilst it should not matter, as it's TCP, it's all on gigabit LAN and the switchport is showing no RX/TX errors or CRC failures.

@platima commented on GitHub (Jan 4, 2026): No storage medium issues - brand new M.2 SSD, and `smartctl` shows it's happy and healthy. Also I can copy 500 photos to the SSD, and back again, without corruption. It's only when I upload via Immich that some become corrupted. Whilst it should not matter, as it's TCP, it's all on gigabit LAN and the switchport is showing no RX/TX errors or CRC failures.
Author
Owner

@platima commented on GitHub (Jan 25, 2026):

I did some more testing;

  • Increased shm; no change
  • Reduced RKNN cores to 1 (guaranteed available); no change
  • Increased memory; no change
  • Disabled swapping; no change
  • Re-pulled without rknn; no change

Then I did a bit more debugging

  • Deleted and re-uploaded files repeatedly; corruption is not always the same files
  • Watched system temp, dmesg, rknn logs, frigate logs, etc whilst uploading; nothing notable

So these files are just being corrupted for some reason that I simply cannot figure out.

@platima commented on GitHub (Jan 25, 2026): I did some more testing; - Increased shm; no change - Reduced RKNN cores to 1 (guaranteed available); no change - Increased memory; no change - Disabled swapping; no change - Re-pulled without rknn; no change Then I did a bit more debugging - Deleted and re-uploaded files repeatedly; corruption is not always the same files - Watched system temp, dmesg, rknn logs, frigate logs, etc whilst uploading; nothing notable So these files are just being corrupted for some reason that I simply cannot figure out.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#8129