[BUG] 502 Bad Gateway after restarting the Browser (Firefox, Gnome Web) #688

Closed
opened 2026-02-04 21:55:17 +03:00 by OVERLORD · 9 comments
Owner

Originally created by @akoyaxd on GitHub (Feb 14, 2023).

The bug

Since upgrading to 1.47.2 from 1.46.0 I get a 502 Bad Gateway when I try to open Immich. Deleting Cockies and Website data solves the issue until it appears again.

The OS that Immich Server is running on

Docker on Debian Host

Version of Immich Server

1.47.2

Version of Immich Mobile App

1.47.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.3"

volumes:
  immich_data:
    external: true

services:
  server:
    image: altran1502/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - immich_data:/usr/src/app/upload
    environment:
      - NODE_ENV=production
      - DB_HOSTNAME=postgres
      - DB_USERNAME=postgres
      - DB_PASSWORD=${DB_PASSWORD}
      - DB_DATABASE_NAME=immichdb
      - REDIS_HOSTNAME=redis
      - JWT_SECRET=${JWT_SECRET}
      - ENABLE_MAPBOX=false
      - MAPBOX_KEY=${MAPBOX_KEY}
      - VITE_LOGIN_PAGE_MESSAGE="Immich Instance des Koyax Project"
      - ENABLE_MAPBOX=true
    env_file:
      - env
    hostname: immich-server
    depends_on:
      - redis
      - postgres
    restart: unless-stopped

  microservices:
    image: altran1502/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    volumes:
      - immich_data:/usr/src/app/upload
    environment:
      - NODE_ENV=production
      - DB_HOSTNAME=postgres
      - DB_USERNAME=postgres
      - DB_PASSWORD=${DB_PASSWORD}
      - DB_DATABASE_NAME=immichdb
      - REDIS_HOSTNAME=redis
      - JWT_SECRET=${JWT_SECRET}
      - ENABLE_MAPBOX=false
      - MAPBOX_KEY=${MAPBOX_KEY}
      - VITE_LOGIN_PAGE_MESSAGE="Immich Instance des Koyax Project"
      - ENABLE_MAPBOX=true
    env_file:
      - env
    depends_on:
      - redis
      - postgres
    restart: unless-stopped
    hostname: immich-microservices
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 4G

  machine-learning:
    image: altran1502/immich-machine-learning:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    volumes:
      - immich_data:/usr/src/app/upload
    environment:
      - NODE_ENV=production
      - DB_HOSTNAME=postgres
      - DB_USERNAME=postgres
      - DB_PASSWORD=${DB_PASSWORD}
      - DB_DATABASE_NAME=immichdb
      - REDIS_HOSTNAME=redis
      - JWT_SECRET=${JWT_SECRET}
      - ENABLE_MAPBOX=false
      - MAPBOX_KEY=${MAPBOX_KEY}
      - VITE_LOGIN_PAGE_MESSAGE="Immich Instance des Koyax Project"
      - ENABLE_MAPBOX=true
    env_file:
      - env
    depends_on:
      - postgres
    restart: unless-stopped
    hostname: immich-machine-learning

  web:
    image: altran1502/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    environment:
      - DB_HOSTNAME=postgres
      - DB_USERNAME=postgres
      - DB_PASSWORD=${DB_PASSWORD}
      - DB_DATABASE_NAME=immichdb
      - REDIS_HOSTNAME=redis
      - JWT_SECRET=${JWT_SECRET}
      - ENABLE_MAPBOX=false
      - MAPBOX_KEY=${MAPBOX_KEY}
      - VITE_LOGIN_PAGE_MESSAGE="Immich Instance des Koyax Project"
      - ENABLE_MAPBOX=true
    env_file:
      - env
    restart: unless-stopped
    hostname: immich-web

  redis:
    image: redis:6
    restart: unless-stopped
    hostname: immich_redis

  postgres:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    env_file:
      - env
    volumes:
      - /opt/immich-postgres:/var/lib/postgresql/data
    restart: unless-stopped
    hostname: immich_postgres

  proxy:
    image: altran1502/immich-proxy:release
    ports:
      - 2283:8080
    depends_on:
      - server
    restart: unless-stopped

Your .env content

OAUTH_ENABLED=true
OAUTH_ISSUER_URL=https://account.koyax.org/application/o/immich/
OAUTH_CLIENT_ID=${OPEN_ID_ID}
OAUTH_CLIENT_SECRET=${OPEN_ID_SECRET}
OAUTH_BUTTON_TEXT=Login mit Koyax ID

Reproduction steps

1. Login to Immich
2. Watch you precious assets
3. Close the browser
4. Retry to open the immich page
5. 502
...

Additional information

  • I use Authentik for Login with OIDC
  • Sensitive Information is set in Portainer
Originally created by @akoyaxd on GitHub (Feb 14, 2023). ### The bug Since upgrading to 1.47.2 from 1.46.0 I get a 502 Bad Gateway when I try to open Immich. Deleting Cockies and Website data solves the issue until it appears again. ### The OS that Immich Server is running on Docker on Debian Host ### Version of Immich Server 1.47.2 ### Version of Immich Mobile App 1.47.0 ### Platform with the issue - [ ] Server - [X] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.3" volumes: immich_data: external: true services: server: image: altran1502/immich-server:release entrypoint: ["/bin/sh", "./start-server.sh"] volumes: - immich_data:/usr/src/app/upload environment: - NODE_ENV=production - DB_HOSTNAME=postgres - DB_USERNAME=postgres - DB_PASSWORD=${DB_PASSWORD} - DB_DATABASE_NAME=immichdb - REDIS_HOSTNAME=redis - JWT_SECRET=${JWT_SECRET} - ENABLE_MAPBOX=false - MAPBOX_KEY=${MAPBOX_KEY} - VITE_LOGIN_PAGE_MESSAGE="Immich Instance des Koyax Project" - ENABLE_MAPBOX=true env_file: - env hostname: immich-server depends_on: - redis - postgres restart: unless-stopped microservices: image: altran1502/immich-server:release entrypoint: ["/bin/sh", "./start-microservices.sh"] volumes: - immich_data:/usr/src/app/upload environment: - NODE_ENV=production - DB_HOSTNAME=postgres - DB_USERNAME=postgres - DB_PASSWORD=${DB_PASSWORD} - DB_DATABASE_NAME=immichdb - REDIS_HOSTNAME=redis - JWT_SECRET=${JWT_SECRET} - ENABLE_MAPBOX=false - MAPBOX_KEY=${MAPBOX_KEY} - VITE_LOGIN_PAGE_MESSAGE="Immich Instance des Koyax Project" - ENABLE_MAPBOX=true env_file: - env depends_on: - redis - postgres restart: unless-stopped hostname: immich-microservices deploy: resources: limits: cpus: '4.0' memory: 4G machine-learning: image: altran1502/immich-machine-learning:release entrypoint: ["/bin/sh", "./entrypoint.sh"] volumes: - immich_data:/usr/src/app/upload environment: - NODE_ENV=production - DB_HOSTNAME=postgres - DB_USERNAME=postgres - DB_PASSWORD=${DB_PASSWORD} - DB_DATABASE_NAME=immichdb - REDIS_HOSTNAME=redis - JWT_SECRET=${JWT_SECRET} - ENABLE_MAPBOX=false - MAPBOX_KEY=${MAPBOX_KEY} - VITE_LOGIN_PAGE_MESSAGE="Immich Instance des Koyax Project" - ENABLE_MAPBOX=true env_file: - env depends_on: - postgres restart: unless-stopped hostname: immich-machine-learning web: image: altran1502/immich-web:release entrypoint: ["/bin/sh", "./entrypoint.sh"] environment: - DB_HOSTNAME=postgres - DB_USERNAME=postgres - DB_PASSWORD=${DB_PASSWORD} - DB_DATABASE_NAME=immichdb - REDIS_HOSTNAME=redis - JWT_SECRET=${JWT_SECRET} - ENABLE_MAPBOX=false - MAPBOX_KEY=${MAPBOX_KEY} - VITE_LOGIN_PAGE_MESSAGE="Immich Instance des Koyax Project" - ENABLE_MAPBOX=true env_file: - env restart: unless-stopped hostname: immich-web redis: image: redis:6 restart: unless-stopped hostname: immich_redis postgres: image: postgres:14 environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} PG_DATA: /var/lib/postgresql/data env_file: - env volumes: - /opt/immich-postgres:/var/lib/postgresql/data restart: unless-stopped hostname: immich_postgres proxy: image: altran1502/immich-proxy:release ports: - 2283:8080 depends_on: - server restart: unless-stopped ``` ### Your .env content ```Shell OAUTH_ENABLED=true OAUTH_ISSUER_URL=https://account.koyax.org/application/o/immich/ OAUTH_CLIENT_ID=${OPEN_ID_ID} OAUTH_CLIENT_SECRET=${OPEN_ID_SECRET} OAUTH_BUTTON_TEXT=Login mit Koyax ID ``` ### Reproduction steps ```bash 1. Login to Immich 2. Watch you precious assets 3. Close the browser 4. Retry to open the immich page 5. 502 ... ``` ### Additional information * I use Authentik for Login with OIDC * Sensitive Information is set in Portainer
Author
Owner

@Morethanevil commented on GitHub (Feb 14, 2023):

I have the same problem with the new version. Version 1.46 works fine.

@Morethanevil commented on GitHub (Feb 14, 2023): I have the same problem with the new version. Version 1.46 works fine.
Author
Owner

@michelheusschen commented on GitHub (Feb 14, 2023):

I'm having a hard time reproducing this issue. Does the error provide any more information besides '502 Bad Gateway' and is there any error logged in the browser's console? A bad gateway error could also be caused by a problem in a container. Do any of the containers have log messages that indicate an error?

@michelheusschen commented on GitHub (Feb 14, 2023): I'm having a hard time reproducing this issue. Does the error provide any more information besides '502 Bad Gateway' and is there any error logged in the browser's console? A bad gateway error could also be caused by a problem in a container. Do any of the containers have log messages that indicate an error?
Author
Owner

@gauravpandey44 commented on GitHub (Feb 14, 2023):

I am also getting the same error after upgrading .
The issue resolves after clearing the cookies . Then again refreshing the page shows the same error message.

My issue resolved as suggested by @akoyaxd :

setting this annotations on k8 ingress
nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"

Thank you 👍

@gauravpandey44 commented on GitHub (Feb 14, 2023): I am also getting the same error after upgrading . The issue resolves after clearing the cookies . Then again refreshing the page shows the same error message. My issue resolved as suggested by @akoyaxd : setting this annotations on k8 ingress nginx.ingress.kubernetes.io/proxy-buffer-size: "128k" Thank you :+1:
Author
Owner

@akoyaxd commented on GitHub (Feb 14, 2023):

I found the issue. It belongs to my Reverse Proxy. The default proxy buffer size was to small by default on my nginx v1.18.0.

I got the following error in my reverse proxy:

2023/02/14 12:27:45 [error] 2822440#2822440: *9613752 upstream sent too big header while reading response header from upstream, client: 2003:dc:bf27:24f4:5cfc:8b5b:74cd:e202, server: photos.koyax.org, request: "GET /photos HTTP/2.0", upstream: "http://127.0.0.1:2283/photos", host: "photos.koyax.org"

Setting the proxy_buffer_size in my reverse proxy solved it for me.

Example for nginx. Set this inside your location blog:

        proxy_buffer_size   128k;
        proxy_buffers   4 256k;
        proxy_busy_buffers_size   256k;

@akoyaxd commented on GitHub (Feb 14, 2023): I found the issue. It belongs to my Reverse Proxy. The default proxy buffer size was to small by default on my nginx v1.18.0. I got the following error in my reverse proxy: ``` 2023/02/14 12:27:45 [error] 2822440#2822440: *9613752 upstream sent too big header while reading response header from upstream, client: 2003:dc:bf27:24f4:5cfc:8b5b:74cd:e202, server: photos.koyax.org, request: "GET /photos HTTP/2.0", upstream: "http://127.0.0.1:2283/photos", host: "photos.koyax.org" ``` Setting the proxy_buffer_size in my reverse proxy solved it for me. Example for nginx. Set this inside your location blog: ``` proxy_buffer_size 128k; proxy_buffers 4 256k; proxy_busy_buffers_size 256k; ```
Author
Owner

@alextran1502 commented on GitHub (Feb 14, 2023):

@akoyaxd Just out of curiosity did you change anything regarding your proxy prior to the update? I am trying to understand why there wasn't any issue before 1.47 and all of the sudden we have many reported problem

@alextran1502 commented on GitHub (Feb 14, 2023): @akoyaxd Just out of curiosity did you change anything regarding your proxy prior to the update? I am trying to understand why there wasn't any issue before 1.47 and all of the sudden we have many reported problem
Author
Owner

@akoyaxd commented on GitHub (Feb 14, 2023):

I was running this nginx config for month without issues.

The bad Gateway only seems to appear on the /photos endpoint. The prior redirect from / works just fine.

One thing I found to be quite large is the link value in the header:



link | <./_app/immutable/assets/_layout-60bc0101.css>;  rel="preload";as="style"; nopush,  <./_app/immutable/assets/asset-grid-9cb813b8.css>;  rel="preload";as="style"; nopush,  <./_app/immutable/assets/asset-viewer-36281e22.css>;  rel="preload";as="style"; nopush,  <./_app/immutable/assets/circle-icon-button-3e5300b3.css>;  rel="preload";as="style"; nopush,  <./_app/immutable/assets/setting-accordion-5439f12e.css>;  rel="preload";as="style"; nopush,  <./_app/immutable/assets/setting-switch-af459860.css>;  rel="preload";as="…rel="modulepreload"; nopush,  <./_app/immutable/chunks/setting-input-field-8f4d2429.js>;  rel="modulepreload"; nopush,  <./_app/immutable/chunks/handle-error-9c6d9d1c.js>;  rel="modulepreload"; nopush,  <./_app/immutable/chunks/status-box-ad613fff.js>;  rel="modulepreload"; nopush,  <./_app/immutable/chunks/theme-button-26a6e752.js>;  rel="modulepreload"; nopush,  <./_app/immutable/chunks/constants-12ef1026.js>;  rel="modulepreload"; nopush,  <./_app/immutable/chunks/side-bar-7127487d.js>;  rel="modulepreload"; nopush
-- | --

Is that one new? Or could it have become larger? That would explain why the header size suddenly exceeds the nginx defaults.

@akoyaxd commented on GitHub (Feb 14, 2023): I was running this nginx config for month without issues. The bad Gateway only seems to appear on the /photos endpoint. The prior redirect from / works just fine. One thing I found to be quite large is the `link` value in the header: ``` link | <./_app/immutable/assets/_layout-60bc0101.css>; rel="preload";as="style"; nopush, <./_app/immutable/assets/asset-grid-9cb813b8.css>; rel="preload";as="style"; nopush, <./_app/immutable/assets/asset-viewer-36281e22.css>; rel="preload";as="style"; nopush, <./_app/immutable/assets/circle-icon-button-3e5300b3.css>; rel="preload";as="style"; nopush, <./_app/immutable/assets/setting-accordion-5439f12e.css>; rel="preload";as="style"; nopush, <./_app/immutable/assets/setting-switch-af459860.css>; rel="preload";as="…rel="modulepreload"; nopush, <./_app/immutable/chunks/setting-input-field-8f4d2429.js>; rel="modulepreload"; nopush, <./_app/immutable/chunks/handle-error-9c6d9d1c.js>; rel="modulepreload"; nopush, <./_app/immutable/chunks/status-box-ad613fff.js>; rel="modulepreload"; nopush, <./_app/immutable/chunks/theme-button-26a6e752.js>; rel="modulepreload"; nopush, <./_app/immutable/chunks/constants-12ef1026.js>; rel="modulepreload"; nopush, <./_app/immutable/chunks/side-bar-7127487d.js>; rel="modulepreload"; nopush -- | -- ``` Is that one new? Or could it have become larger? That would explain why the header size suddenly exceeds the nginx defaults.
Author
Owner

@alextran1502 commented on GitHub (Feb 14, 2023):

@michelheusschen this issue might related to caching the asset PR

@alextran1502 commented on GitHub (Feb 14, 2023): @michelheusschen this issue might related to caching the asset PR
Author
Owner

@michelheusschen commented on GitHub (Feb 14, 2023):

Just had a quick look and it seems indirectly related to the asset caching PR #1724. A lot of components are getting preloaded in the link header and in that PR the ImmichLogo component also gets added to that list, which could be just enough to trigger this error.

Then the current nginx config seems a bit strange to me. The proxy_buffering directive is set to off and according to the nginx docs, only the proxy_buffer_size should influence anything related to buffering, but there are a bunch of other directives related to proxy buffering being set.

Doubling the current proxy_buffer_size from 16k to 32k should be sufficient for now (untested), but the whole nginx buffering configuration should be revisited.

@michelheusschen commented on GitHub (Feb 14, 2023): Just had a quick look and it seems indirectly related to the asset caching PR #1724. A lot of components are getting preloaded in the `link` header and in that PR the ImmichLogo component also gets added to that list, which could be just enough to trigger this error. Then the current nginx config seems a bit strange to me. The `proxy_buffering` directive is set to `off` and according to the [nginx docs](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering), only the `proxy_buffer_size` should influence anything related to buffering, but there are a bunch of other directives related to proxy buffering being set. Doubling the current `proxy_buffer_size` from 16k to 32k should be sufficient for now (untested), but the whole nginx buffering configuration should be revisited.
Author
Owner

@akoyaxd commented on GitHub (Feb 14, 2023):

Just for your information:
I used a separate nginx proxy in front of my docker compose deployment. I didn't change the config from an nginx server inside the container.

@akoyaxd commented on GitHub (Feb 14, 2023): Just for your information: I used a separate nginx proxy in front of my docker compose deployment. I didn't change the config from an nginx server inside the container.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#688