immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-24 05:44:37 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Very long check times with borg and rclone after updating from 1.122.1 to 1.124.2 #5145

New Issue
Closed
opened 2026-02-05 11:11:53 +03:00 by OVERLORD · 6 comments
OVERLORD commented 2026-02-05 11:11:53 +03:00
Owner
Copy Link

Originally created by @gullmar on GitHub (Jan 14, 2025).

The bug

I'm running Immich with Docker Compose and I have and external backup solution in place, relying on borg and rclone.
Shortly, I have a script which runs every day that stops the Immich container, performs a backup of the data folder to a USB hard drive using borg, restarts the Immich container, and sync the backup to OneDrive using rclone.
Two days ago, I updated Immich to the last version: there are no breaking changes mentioned in the changelogs, so I didn't change anything in the configuration.
From then, the duration of backup operations really increased:

  • backup with borg: 2 minutes -> 3 hours
  • sync with rclone: 1 hour -> 15+ hours (I've stopped it)

Both the operations seem to be stuck for a lot of time on a checking phase.
I've attached borg's logs, and I've put a note on the line where the process hangs.
On the other hand, rclone syncs the backups for many other services too, so it may or may not be related to this.
The other services did not have any problem with borg, which is executed individually for each one of them.
The same problem with Immich happened today, for the second time.
I'm not sure where could I gather further information to investigate this problem.
I did NOT add a huge amount of pictures in the last days, I've actually used Immich really little, and I'm the only user.
My next move would be deleting my current backup, including the remote copy, and trying to make a new backup from scratch, since, for now, I don't care about my backup history.
But this could be a problem in other circumstances, and maybe something similar happened to other users, so I'm posting this here.
Thank you for all your great work!

The OS that Immich Server is running on

Debian 12

Version of Immich Server

v1.124.2

Version of Immich Mobile App

v1.123.0 build.172 (f-droid)

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
    immich-server:
        container_name: immich_server
        image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
        extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
            file: hwaccel.transcoding.yml
            service: quicksync # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
        volumes:
            # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
            - ${UPLOAD_LOCATION}:/usr/src/app/upload
            - /etc/localtime:/etc/localtime:ro
        env_file:
            - .env
        ports:
            - 2283:2283
        depends_on:
            - redis
            - database
        restart: always
        healthcheck:
            disable: false

    immich-machine-learning:
        container_name: immich_machine_learning
        # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
        # Example tag: ${IMMICH_VERSION:-release}-cuda
        image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
        # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
        #   file: hwaccel.ml.yml
        #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
        volumes:
            - model-cache:/cache
        env_file:
            - .env
        restart: always
        healthcheck:
            disable: false

    redis:
        container_name: immich_redis
        image: docker.io/redis:6.2-alpine@sha256:eaba718fecd1196d88533de7ba49bf903ad33664a92debb24660a922ecd9cac8
        healthcheck:
            test: redis-cli ping || exit 1
        restart: always

    database:
        container_name: immich_postgres
        image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
        environment:
            POSTGRES_PASSWORD: ${DB_PASSWORD}
            POSTGRES_USER: ${DB_USERNAME}
            POSTGRES_DB: ${DB_DATABASE_NAME}
            POSTGRES_INITDB_ARGS: "--data-checksums"
        volumes:
            # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
            - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
        healthcheck:
            test: >-
                pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
                Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
                --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
                echo "checksum failure count is $$Chksum";
                [ "$$Chksum" = '0' ] || exit 1
            interval: 5m
            start_interval: 30s
            start_period: 5m
        command: >-
            postgres
            -c shared_preload_libraries=vectors.so
            -c 'search_path="$$user", public, vectors'
            -c logging_collector=on
            -c max_wal_size=2GB
            -c shared_buffers=512MB
            -c wal_compression=on
        restart: always

volumes:
    model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=../../data/immich/data

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=MYPASSWORD

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_DATA_LOCATION=../../data/immich/db

REDIS_HOSTNAME=immich_redis

Reproduction steps

  1. Perform a few backups using borg on version v1.122.1: the last backup should take just a few minutes if there are not many new pictures
  2. Update to version v1.124.2
  3. Perform a new backup with borg: checking the backup takes a lot of time, much more than before

...

Relevant log output

Creating archive at "/borg/repo::08e6a3499f99_2025-01-13"
------------------------------------------------------------------------------
Repository: /borg/repo
Archive name: 08e6a3499f99_2025-01-13
Archive fingerprint: 4e16baa904b86c3f51cb4f7c158c18052f792b036c321a302f2f80eb5d756c2b
Time (start): Mon, 2025-01-13 02:03:54
Time (end):   Mon, 2025-01-13 02:03:58
Duration: 4.05 seconds
Number of files: 5212
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               13.30 GB             12.80 GB             65.73 MB
All archives:              185.45 GB            177.91 GB             13.13 GB

                       Unique chunks         Total chunks
Chunk index:                   10023               132647
------------------------------------------------------------------------------
terminating with success status, rc 0
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:              -13.35 GB            -12.81 GB             -2.65 MB
All archives:              172.10 GB            165.11 GB             13.13 GB

                       Unique chunks         Total chunks
Chunk index:                    9998               123085
------------------------------------------------------------------------------
terminating with success status, rc 0
compaction freed about 2.66 MB repository space.
terminating with success status, rc 0
Starting repository check <---------------------------- HERE IS WHERE THE PROCESS HANGS!
finished segment check at segment 440
Starting repository index check
Index object count match.
Finished full repository check, no problems found.
Starting archive consistency check...
Analyzing archive 15038bfa10c8_2024-12-08 (1/13)
Analyzing archive a861331fbc49_2024-12-15 (2/13)
Analyzing archive 6dc76f73687d_2024-12-22 (3/13)
Analyzing archive 49f3e83539dc_2024-12-29 (4/13)
Analyzing archive 7758d10ce14e_2024-12-31 (5/13)
Analyzing archive c63364c6950b_2025-01-05 (6/13)
Analyzing archive f5b068bab784_2025-01-07 (7/13)
Analyzing archive f4681da47a70_2025-01-08 (8/13)
Analyzing archive d2fa9213ae5e_2025-01-09 (9/13)
Analyzing archive ee3ee08afd30_2025-01-10 (10/13)
Analyzing archive f0db6c503566_2025-01-11 (11/13)
Analyzing archive 7c22264a4db7_2025-01-12 (12/13)
Analyzing archive 08e6a3499f99_2025-01-13 (13/13)
Archive consistency check complete, no problems found.
terminating with success status, rc 0

Additional information

Borg is run using docker, it was last updated two months ago:

pschiffe/borg                                   latest    5acfe3500414
Originally created by @gullmar on GitHub (Jan 14, 2025). ### The bug I'm running Immich with Docker Compose and I have and external backup solution in place, relying on borg and rclone. Shortly, I have a script which runs every day that stops the Immich container, performs a backup of the data folder to a USB hard drive using borg, restarts the Immich container, and sync the backup to OneDrive using rclone. Two days ago, I updated Immich to the last version: there are no breaking changes mentioned in the changelogs, so I didn't change anything in the configuration. From then, the duration of backup operations really increased: - backup with borg: 2 minutes -> 3 hours - sync with rclone: 1 hour -> 15+ hours (I've stopped it) Both the operations seem to be stuck for a lot of time on a checking phase. I've attached borg's logs, and I've put a note on the line where the process hangs. On the other hand, rclone syncs the backups for many other services too, so it may or may not be related to this. The other services did not have any problem with borg, which is executed individually for each one of them. The same problem with Immich happened today, for the second time. I'm not sure where could I gather further information to investigate this problem. I did NOT add a huge amount of pictures in the last days, I've actually used Immich really little, and I'm the only user. My next move would be deleting my current backup, including the remote copy, and trying to make a new backup from scratch, since, for now, I don't care about my backup history. But this could be a problem in other circumstances, and maybe something similar happened to other users, so I'm posting this here. Thank you for all your great work! ### The OS that Immich Server is running on Debian 12 ### Version of Immich Server v1.124.2 ### Version of Immich Mobile App v1.123.0 build.172 (f-droid) ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding file: hwaccel.transcoding.yml service: quicksync # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:2283 depends_on: - redis - database restart: always healthcheck: disable: false immich-machine-learning: container_name: immich_machine_learning # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag. # Example tag: ${IMMICH_VERSION:-release}-cuda image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration # file: hwaccel.ml.yml # service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable volumes: - model-cache:/cache env_file: - .env restart: always healthcheck: disable: false redis: container_name: immich_redis image: docker.io/redis:6.2-alpine@sha256:eaba718fecd1196d88533de7ba49bf903ad33664a92debb24660a922ecd9cac8 healthcheck: test: redis-cli ping || exit 1 restart: always database: container_name: immich_postgres image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_INITDB_ARGS: "--data-checksums" volumes: # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file - ${DB_DATA_LOCATION}:/var/lib/postgresql/data healthcheck: test: >- pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1; Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1 interval: 5m start_interval: 30s start_period: 5m command: >- postgres -c shared_preload_libraries=vectors.so -c 'search_path="$$user", public, vectors' -c logging_collector=on -c max_wal_size=2GB -c shared_buffers=512MB -c wal_compression=on restart: always volumes: model-cache: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=../../data/immich/data # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secret for postgres. You should change it to a random password DB_PASSWORD=MYPASSWORD # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich DB_DATA_LOCATION=../../data/immich/db REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps 1. Perform a few backups using borg on version v1.122.1: the last backup should take just a few minutes if there are not many new pictures 2. Update to version v1.124.2 3. Perform a new backup with borg: checking the backup takes a lot of time, much more than before 4. ... ### Relevant log output ```shell Creating archive at "/borg/repo::08e6a3499f99_2025-01-13" ------------------------------------------------------------------------------ Repository: /borg/repo Archive name: 08e6a3499f99_2025-01-13 Archive fingerprint: 4e16baa904b86c3f51cb4f7c158c18052f792b036c321a302f2f80eb5d756c2b Time (start): Mon, 2025-01-13 02:03:54 Time (end): Mon, 2025-01-13 02:03:58 Duration: 4.05 seconds Number of files: 5212 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 13.30 GB 12.80 GB 65.73 MB All archives: 185.45 GB 177.91 GB 13.13 GB Unique chunks Total chunks Chunk index: 10023 132647 ------------------------------------------------------------------------------ terminating with success status, rc 0 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size Deleted data: -13.35 GB -12.81 GB -2.65 MB All archives: 172.10 GB 165.11 GB 13.13 GB Unique chunks Total chunks Chunk index: 9998 123085 ------------------------------------------------------------------------------ terminating with success status, rc 0 compaction freed about 2.66 MB repository space. terminating with success status, rc 0 Starting repository check <---------------------------- HERE IS WHERE THE PROCESS HANGS! finished segment check at segment 440 Starting repository index check Index object count match. Finished full repository check, no problems found. Starting archive consistency check... Analyzing archive 15038bfa10c8_2024-12-08 (1/13) Analyzing archive a861331fbc49_2024-12-15 (2/13) Analyzing archive 6dc76f73687d_2024-12-22 (3/13) Analyzing archive 49f3e83539dc_2024-12-29 (4/13) Analyzing archive 7758d10ce14e_2024-12-31 (5/13) Analyzing archive c63364c6950b_2025-01-05 (6/13) Analyzing archive f5b068bab784_2025-01-07 (7/13) Analyzing archive f4681da47a70_2025-01-08 (8/13) Analyzing archive d2fa9213ae5e_2025-01-09 (9/13) Analyzing archive ee3ee08afd30_2025-01-10 (10/13) Analyzing archive f0db6c503566_2025-01-11 (11/13) Analyzing archive 7c22264a4db7_2025-01-12 (12/13) Analyzing archive 08e6a3499f99_2025-01-13 (13/13) Archive consistency check complete, no problems found. terminating with success status, rc 0 ``` ### Additional information Borg is run using docker, it was last updated two months ago: ``` pschiffe/borg latest 5acfe3500414 ```
OVERLORD commented 2026-02-05 11:11:57 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jan 14, 2025):

Hello, Immich has no control over how long backups take. This is likely a network or hardware issue.

@mmomjian commented on GitHub (Jan 14, 2025): Hello, Immich has no control over how long backups take. This is likely a network or hardware issue.
OVERLORD commented 2026-02-05 11:11:58 +03:00
Author
Owner
Copy Link

@gullmar commented on GitHub (Jan 14, 2025):

I thought that it may be relevant because I have no problem with other services, running with Docker and being backed up with the same strategy.
The problem appears to be affecting only Immich after the update.
I'm sorry for not being able to provide more details, I'm still investigating it.

@gullmar commented on GitHub (Jan 14, 2025): I thought that it may be relevant because I have no problem with other services, running with Docker and being backed up with the same strategy. The problem appears to be affecting only Immich after the update. I'm sorry for not being able to provide more details, I'm still investigating it.
OVERLORD commented 2026-02-05 11:11:59 +03:00
Author
Owner
Copy Link

@gullmar commented on GitHub (Jan 14, 2025):

And by the way, I did not want to sound rude at all: thank you for your quick answer and for your time.

@gullmar commented on GitHub (Jan 14, 2025): And by the way, I did not want to sound rude at all: thank you for your quick answer and for your time.
OVERLORD commented 2026-02-05 11:12:00 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Jan 14, 2025):

Perhaps other services don't have that many directories like Immich? And USB drives usually don't have the fastest IO speed

@alextran1502 commented on GitHub (Jan 14, 2025): Perhaps other services don't have that many directories like Immich? And USB drives usually don't have the fastest IO speed
OVERLORD commented 2026-02-05 11:12:01 +03:00
Author
Owner
Copy Link

@gullmar commented on GitHub (Jan 14, 2025):

It could certainly be, and I'm aware that both writing to the USB drive and uploading to OneDrive are slow.
I thought of writing this issue because:

  • it was not happening before updating Immich: incremental borg backups for Immich took just a few minutes, now it takes hours;
  • borg backup's performance did not change for other services.

Nevertheless, I understand that this issue is very generic and difficult, if not impossible, to replicate, especially if nobody else is reporting anything similar. It may also be something not directly related to Immich which, for some reason, only affected Immich.
I will try to fix it on my own and then report here.
Thank you.

@gullmar commented on GitHub (Jan 14, 2025): It could certainly be, and I'm aware that both writing to the USB drive and uploading to OneDrive are slow. I thought of writing this issue because: - it was not happening before updating Immich: incremental borg backups for Immich took just a few minutes, now it takes hours; - borg backup's performance did not change for other services. Nevertheless, I understand that this issue is very generic and difficult, if not impossible, to replicate, especially if nobody else is reporting anything similar. It may also be something not directly related to Immich which, for some reason, only affected Immich. I will try to fix it on my own and then report here. Thank you.
OVERLORD commented 2026-02-05 11:12:02 +03:00
Author
Owner
Copy Link

@gullmar commented on GitHub (Mar 6, 2025):

Just an update: Borg checks still take a very long time as per v1.129.0, but Rclone check times are now somehow acceptable on my setup.
I've disabled Borg checks, for now, since even the example script in this guide does not perform the check: https://immich.app/docs/guides/template-backup-script/.

@gullmar commented on GitHub (Mar 6, 2025): Just an update: Borg checks still take a very long time as per v1.129.0, but Rclone check times are now somehow acceptable on my setup. I've disabled Borg checks, for now, since even the example script in this guide does not perform the check: https://immich.app/docs/guides/template-backup-script/.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#5145
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?