Background backup throws error when proxypassed #2069

Closed
opened 2026-02-05 04:58:11 +03:00 by OVERLORD · 1 comment
Owner

Originally created by @wociscz on GitHub (Jan 27, 2024).

The bug

Immich android app throws error when immich is behind (nginx) proxypass with valid https certificate.
Error (notification) is "background backup failed, trying again" (or ~smth like this) after taking a photo.
There is no log event in access/error log on nginx side during this error.

My setup:

  • immich runs in docker compose in VM (hostname docker.internal)
  • nginx proxy is running in different VM (immich.internal) configured with valid https cert from my .internal CA proxying requests to http://docker.internal:2283
  • phone has .internal CA certificate added into trust store. Other sites in my network works without issue, also immich.internal when opened in the chrome browser on android works and is "trusted".
  • everything works in browser without any issue if I connect to https://immich.internal or directly to http://docker.internal:2283.
  • network is configure to use DHCP - phone uses internal DNS so is able to resolve whatever.internal to proper ip address.

Problem is when I configure android app to use https://immich.internal as host. I can use the app without problem with foreground backup. Background backup throws mentioned error. I have also configured "Allow self-signed certificates" even though the cert is NOT self-signed from the android system point of view - because I have CA cert for .internal imported and it is trusted system-wide.

If I configure the app pointing directly to http://docker.internal:2283 background backup starts working as well without any issue.

The OS that Immich Server is running on

Ubuntu 22.04 with docker compose

Version of Immich Server

v1.93.3

Version of Immich Mobile App

v1.93.2

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

/mnt/photo/immich is cifs mount to NAS with proper permissions.

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/photo/immich/

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. create internal CA for desired internal `.tld` of your choose (beyond this reproduction howto - check step-ca for example)
2. create https proxypass on different machine pointing to immich docker compose port
3. try to configure android app pointing to proxypass hostname

Additional information

nginx proxypass server definition:

server {
        listen 443 ssl http2;
        ssl_certificate /etc/letsencrypt/live/immich.internal/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/immich.internal/privkey.pem;
        ssl_trusted_certificate /etc/letsencrypt/live/immich.internal/chain.pem;
        access_log /var/log/nginx/access_immich.log;
        error_log /var/log/nginx/error_immich.log;
        include /etc/nginx/ssl.conf;

        server_name immich.internal;


    client_max_body_size 50000M;

    location / {
        set $upstream docker.internal:2283;
        proxy_pass http://$upstream;
        proxy_set_header Host              $http_host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # http://nginx.org/en/docs/http/websocket.html
        proxy_http_version 1.1;
        proxy_set_header   Upgrade    $http_upgrade;
        proxy_set_header   Connection "upgrade";
        proxy_redirect off;
    }
}
Originally created by @wociscz on GitHub (Jan 27, 2024). ### The bug Immich android app throws error when immich is behind (nginx) proxypass with valid https certificate. Error (notification) is "background backup failed, trying again" (or ~smth like this) after taking a photo. There is no log event in access/error log on nginx side during this error. My setup: - immich runs in docker compose in VM (hostname `docker.internal`) - nginx proxy is running in different VM (`immich.internal`) configured with valid https cert from my `.internal` CA proxying requests to `http://docker.internal:2283` - phone has `.internal` CA certificate added into trust store. Other sites in my network works without issue, also `immich.internal` when opened in the chrome browser on android works and is "trusted". - everything works in browser without any issue if I connect to `https://immich.internal` or directly to `http://docker.internal:2283`. - network is configure to use DHCP - phone uses internal DNS so is able to resolve `whatever.internal` to proper ip address. Problem is when I configure android app to use `https://immich.internal` as host. I can use the app without problem with foreground backup. Background backup throws mentioned error. I have also configured "Allow self-signed certificates" even though the cert is NOT self-signed from the android system point of view - because I have CA cert for `.internal` imported and it is trusted system-wide. If I configure the app pointing directly to `http://docker.internal:2283` background backup starts working as well without any issue. ### The OS that Immich Server is running on Ubuntu 22.04 with docker compose ### Version of Immich Server v1.93.3 ### Version of Immich Mobile App v1.93.2 ### Platform with the issue - [X] Server - [ ] Web - [X] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 depends_on: - redis - database restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc restart: always database: container_name: immich_postgres image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always volumes: pgdata: model-cache: ``` ### Your .env content `/mnt/photo/immich` is cifs mount to NAS with proper permissions. ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=/mnt/photo/immich/ # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secret for postgres. You should change it to a random password DB_PASSWORD=postgres # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ``` 1. create internal CA for desired internal `.tld` of your choose (beyond this reproduction howto - check step-ca for example) 2. create https proxypass on different machine pointing to immich docker compose port 3. try to configure android app pointing to proxypass hostname ``` ### Additional information nginx proxypass server definition: ``` server { listen 443 ssl http2; ssl_certificate /etc/letsencrypt/live/immich.internal/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/immich.internal/privkey.pem; ssl_trusted_certificate /etc/letsencrypt/live/immich.internal/chain.pem; access_log /var/log/nginx/access_immich.log; error_log /var/log/nginx/error_immich.log; include /etc/nginx/ssl.conf; server_name immich.internal; client_max_body_size 50000M; location / { set $upstream docker.internal:2283; proxy_pass http://$upstream; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # http://nginx.org/en/docs/http/websocket.html proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_redirect off; } } ```
Author
Owner

@bo0tzz commented on GitHub (Jan 27, 2024):

Potentially a duplicate of #5562

@bo0tzz commented on GitHub (Jan 27, 2024): Potentially a duplicate of #5562
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2069