immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Database migration failure #6344

New Issue
Closed
opened 2026-02-05 12:13:00 +03:00 by OVERLORD · 18 comments
OVERLORD commented 2026-02-05 12:13:00 +03:00
Owner
Copy Link

Originally created by @rstat1 on GitHub (Jun 22, 2025).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • Yes

The bug

Did the update to 1.135.3 the other day, and noticed today when looking at some other stuff that the update did not succeed due to a database migration failure.

alter database "immich-db" set role to immich-db;
Seem to be caused by the last parameter to this query not being quoted, as in my testing the query succeeds when it is quoted.

I'm running Immich on a Kubernetes cluster, using an external instance of Postgres that's shared with other things on the cluster.

The OS that Immich Server is running on

Debian 12

Version of Immich Server

1.135.3

Version of Immich Mobile App

1.135.1

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

Don't have one of these

Your .env content

IMMICH_LOG_LEVEL: debug
      REDIS_HOSTNAME: '{{ printf "%s-redis-master" .Release.Name }}'
      IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}'
      DB_HOSTNAME:
        valueFrom:
          secretKeyRef:
            name: immich-db-secret-immich-db
            key: HOST
      DB_USERNAME:
        valueFrom:
          secretKeyRef:
            name: immich-db-secret-immich-db
            key: LOGIN
      DB_DATABASE_NAME:
        valueFrom:
          secretKeyRef:
            name: immich-db-secret-immich-db
            key: DATABASE_NAME
      # -- You should provide your own secret outside of this helm-chart and use `postgresql.global.postgresql.auth.existingSecret` to provide credentials to the postgresql instance
      DB_PASSWORD:
        valueFrom:
          secretKeyRef:
            name: immich-db-secret-immich-db
            key: PASSWORD

Reproduction steps

  1. Updated to 1.135.3
  2. CrashLoopBackoff
  3. ???
    ...

Relevant log output

Query failed : {
   durationMs: 7.36540100000002,
   error: PostgresError: syntax error at or near "-"
       at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
       at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
       at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
       at Socket.emit (node:events:518:28)
       at addChunk (node:internal/streams/readable:561:12)
       at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
       at Readable.push (node:internal/streams/readable:392:5)
       at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
     severity_local: 'ERROR',
     severity: 'ERROR',
     code: '42601',
     position: '46',
     file: 'scan.l',
     line: '1244',
     routine: 'scanner_yyerror'
   },
   sql: 'alter database "immich-db" set role to immich-db;',
   params: []
 }
 [Nest] 7  - 06/22/2025, 6:36:03 PM     LOG [Microservices:DatabaseRepository] Migration "1750189909087-AddAlbumUserCreateFields" succeeded
 [Nest] 7  - 06/22/2025, 6:36:03 PM   ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: syntax error at or near "-"
 [Nest] 7  - 06/22/2025, 6:36:03 PM    WARN [Microservices:DatabaseRepository] Migration "1750323941566-UnsetPrewarmDimParameter" failed

Additional information

No response

Originally created by @rstat1 on GitHub (Jun 22, 2025). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug Did the update to 1.135.3 the other day, and noticed today when looking at some other stuff that the update did not succeed due to a database migration failure. ```alter database "immich-db" set role to immich-db;``` Seem to be caused by the last parameter to this query not being quoted, as in my testing the query succeeds when it is quoted. I'm running Immich on a Kubernetes cluster, using an external instance of Postgres that's shared with other things on the cluster. ### The OS that Immich Server is running on Debian 12 ### Version of Immich Server 1.135.3 ### Version of Immich Mobile App 1.135.1 ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML Don't have one of these ``` ### Your .env content ```Shell IMMICH_LOG_LEVEL: debug REDIS_HOSTNAME: '{{ printf "%s-redis-master" .Release.Name }}' IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}' DB_HOSTNAME: valueFrom: secretKeyRef: name: immich-db-secret-immich-db key: HOST DB_USERNAME: valueFrom: secretKeyRef: name: immich-db-secret-immich-db key: LOGIN DB_DATABASE_NAME: valueFrom: secretKeyRef: name: immich-db-secret-immich-db key: DATABASE_NAME # -- You should provide your own secret outside of this helm-chart and use `postgresql.global.postgresql.auth.existingSecret` to provide credentials to the postgresql instance DB_PASSWORD: valueFrom: secretKeyRef: name: immich-db-secret-immich-db key: PASSWORD ``` ### Reproduction steps 1. Updated to 1.135.3 2. CrashLoopBackoff 3. ??? ... ### Relevant log output ```shell Query failed : { durationMs: 7.36540100000002, error: PostgresError: syntax error at or near "-" at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) at Socket.emit (node:events:518:28) at addChunk (node:internal/streams/readable:561:12) at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) at Readable.push (node:internal/streams/readable:392:5) at TCP.onStreamRead (node:internal/stream_base_commons:189:23) { severity_local: 'ERROR', severity: 'ERROR', code: '42601', position: '46', file: 'scan.l', line: '1244', routine: 'scanner_yyerror' }, sql: 'alter database "immich-db" set role to immich-db;', params: [] } [Nest] 7 - 06/22/2025, 6:36:03 PM LOG [Microservices:DatabaseRepository] Migration "1750189909087-AddAlbumUserCreateFields" succeeded [Nest] 7 - 06/22/2025, 6:36:03 PM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: syntax error at or near "-" [Nest] 7 - 06/22/2025, 6:36:03 PM WARN [Microservices:DatabaseRepository] Migration "1750323941566-UnsetPrewarmDimParameter" failed ``` ### Additional information _No response_
OVERLORD added the 🗄️server label 2026-02-05 12:13:00 +03:00
OVERLORD commented 2026-02-05 12:13:02 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 22, 2025):

Can you please connect to the DB and run the following command:

SELECT * FROM pg_db_role_setting left join pg_database d on d.oid = setdatabase left join pg_roles r on r.oid = setrole \gx

@mmomjian commented on GitHub (Jun 22, 2025): Can you please connect to the DB and run the following command: `SELECT * FROM pg_db_role_setting left join pg_database d on d.oid = setdatabase left join pg_roles r on r.oid = setrole \gx`
OVERLORD commented 2026-02-05 12:13:02 +03:00
Author
Owner
Copy Link

@rstat1 commented on GitHub (Jun 22, 2025):

Here ya go:

Command output 
[
  {
    "setdatabase": 0,
    "setrole": 17831,
    "setconfig": [
      "role=immich-db"
    ],
    "oid": null,
    "datname": null,
    "datdba": null,
    "encoding": null,
    "datlocprovider": null,
    "datistemplate": null,
    "datallowconn": null,
    "datconnlimit": null,
    "datfrozenxid": null,
    "datminmxid": null,
    "dattablespace": null,
    "datcollate": null,
    "datctype": null,
    "daticulocale": null,
    "daticurules": null,
    "datcollversion": null,
    "datacl": null,
    "rolname": "immich-db-c3WD9F",
    "rolsuper": true,
    "rolinherit": true,
    "rolcreaterole": false,
    "rolcreatedb": false,
    "rolcanlogin": true,
    "rolreplication": false,
    "rolconnlimit": -1,
    "rolpassword": "********",
    "rolvaliduntil": null,
    "rolbypassrls": false,
    "rolconfig": [
      "role=immich-db"
    ],
    "oid (1)": 17831
  },
  {
    "setdatabase": 0,
    "setrole": 79318,
    "setconfig": [
      "role=****-db"
    ],
    "oid": null,
    "datname": null,
    "datdba": null,
    "encoding": null,
    "datlocprovider": null,
    "datistemplate": null,
    "datallowconn": null,
    "datconnlimit": null,
    "datfrozenxid": null,
    "datminmxid": null,
    "dattablespace": null,
    "datcollate": null,
    "datctype": null,
    "daticulocale": null,
    "daticurules": null,
    "datcollversion": null,
    "datacl": null,
    "rolname": "****-db-Gyz4PT",
    "rolsuper": false,
    "rolinherit": true,
    "rolcreaterole": false,
    "rolcreatedb": false,
    "rolcanlogin": true,
    "rolreplication": false,
    "rolconnlimit": -1,
    "rolpassword": "********",
    "rolvaliduntil": null,
    "rolbypassrls": false,
    "rolconfig": [
      "role=****-db"
    ],
    "oid (1)": 79318
  },
  {
    "setdatabase": 0,
    "setrole": 125145,
    "setconfig": [
      "role=****-db"
    ],
    "oid": null,
    "datname": null,
    "datdba": null,
    "encoding": null,
    "datlocprovider": null,
    "datistemplate": null,
    "datallowconn": null,
    "datconnlimit": null,
    "datfrozenxid": null,
    "datminmxid": null,
    "dattablespace": null,
    "datcollate": null,
    "datctype": null,
    "daticulocale": null,
    "daticurules": null,
    "datcollversion": null,
    "datacl": null,
    "rolname": "****-db-c3WD9F",
    "rolsuper": false,
    "rolinherit": true,
    "rolcreaterole": false,
    "rolcreatedb": false,
    "rolcanlogin": true,
    "rolreplication": false,
    "rolconnlimit": -1,
    "rolpassword": "********",
    "rolvaliduntil": null,
    "rolbypassrls": false,
    "rolconfig": [
      "role=****-db"
    ],
    "oid (1)": 125145
  },
  {
    "setdatabase": 17112,
    "setrole": 0,
    "setconfig": [
      "search_path=\"$user\", public, vectors",
      "role=immich-db"
    ],
    "oid": 17112,
    "datname": "immich-db",
    "datdba": 17111,
    "encoding": 6,
    "datlocprovider": "c",
    "datistemplate": false,
    "datallowconn": true,
    "datconnlimit": -1,
    "datfrozenxid": "722",
    "datminmxid": "1",
    "dattablespace": 1663,
    "datcollate": "C",
    "datctype": "C",
    "daticulocale": null,
    "daticurules": null,
    "datcollversion": null,
    "datacl": "{\"=Tc/\\\"immich-db\\\"\",\"\\\"immich-db\\\"=CTc/\\\"immich-db\\\"\",\"\\\"immich-db-c3WD9F\\\"=CTc/\\\"immich-db\\\"\"}",
    "rolname": null,
    "rolsuper": null,
    "rolinherit": null,
    "rolcreaterole": null,
    "rolcreatedb": null,
    "rolcanlogin": null,
    "rolreplication": null,
    "rolconnlimit": null,
    "rolpassword": null,
    "rolvaliduntil": null,
    "rolbypassrls": null,
    "rolconfig": null,
    "oid (1)": null
  }
]
@rstat1 commented on GitHub (Jun 22, 2025): Here ya go: <details> <summary>Command output</summary> ``` [ { "setdatabase": 0, "setrole": 17831, "setconfig": [ "role=immich-db" ], "oid": null, "datname": null, "datdba": null, "encoding": null, "datlocprovider": null, "datistemplate": null, "datallowconn": null, "datconnlimit": null, "datfrozenxid": null, "datminmxid": null, "dattablespace": null, "datcollate": null, "datctype": null, "daticulocale": null, "daticurules": null, "datcollversion": null, "datacl": null, "rolname": "immich-db-c3WD9F", "rolsuper": true, "rolinherit": true, "rolcreaterole": false, "rolcreatedb": false, "rolcanlogin": true, "rolreplication": false, "rolconnlimit": -1, "rolpassword": "********", "rolvaliduntil": null, "rolbypassrls": false, "rolconfig": [ "role=immich-db" ], "oid (1)": 17831 }, { "setdatabase": 0, "setrole": 79318, "setconfig": [ "role=****-db" ], "oid": null, "datname": null, "datdba": null, "encoding": null, "datlocprovider": null, "datistemplate": null, "datallowconn": null, "datconnlimit": null, "datfrozenxid": null, "datminmxid": null, "dattablespace": null, "datcollate": null, "datctype": null, "daticulocale": null, "daticurules": null, "datcollversion": null, "datacl": null, "rolname": "****-db-Gyz4PT", "rolsuper": false, "rolinherit": true, "rolcreaterole": false, "rolcreatedb": false, "rolcanlogin": true, "rolreplication": false, "rolconnlimit": -1, "rolpassword": "********", "rolvaliduntil": null, "rolbypassrls": false, "rolconfig": [ "role=****-db" ], "oid (1)": 79318 }, { "setdatabase": 0, "setrole": 125145, "setconfig": [ "role=****-db" ], "oid": null, "datname": null, "datdba": null, "encoding": null, "datlocprovider": null, "datistemplate": null, "datallowconn": null, "datconnlimit": null, "datfrozenxid": null, "datminmxid": null, "dattablespace": null, "datcollate": null, "datctype": null, "daticulocale": null, "daticurules": null, "datcollversion": null, "datacl": null, "rolname": "****-db-c3WD9F", "rolsuper": false, "rolinherit": true, "rolcreaterole": false, "rolcreatedb": false, "rolcanlogin": true, "rolreplication": false, "rolconnlimit": -1, "rolpassword": "********", "rolvaliduntil": null, "rolbypassrls": false, "rolconfig": [ "role=****-db" ], "oid (1)": 125145 }, { "setdatabase": 17112, "setrole": 0, "setconfig": [ "search_path=\"$user\", public, vectors", "role=immich-db" ], "oid": 17112, "datname": "immich-db", "datdba": 17111, "encoding": 6, "datlocprovider": "c", "datistemplate": false, "datallowconn": true, "datconnlimit": -1, "datfrozenxid": "722", "datminmxid": "1", "dattablespace": 1663, "datcollate": "C", "datctype": "C", "daticulocale": null, "daticurules": null, "datcollversion": null, "datacl": "{\"=Tc/\\\"immich-db\\\"\",\"\\\"immich-db\\\"=CTc/\\\"immich-db\\\"\",\"\\\"immich-db-c3WD9F\\\"=CTc/\\\"immich-db\\\"\"}", "rolname": null, "rolsuper": null, "rolinherit": null, "rolcreaterole": null, "rolcreatedb": null, "rolcanlogin": null, "rolreplication": null, "rolconnlimit": null, "rolpassword": null, "rolvaliduntil": null, "rolbypassrls": null, "rolconfig": null, "oid (1)": null } ] ``` </details>
OVERLORD commented 2026-02-05 12:13:03 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 22, 2025):

I am not sure why you have all these role=immich-db setconfigs, nor what they mean. I don't think this is an actual PG param. Have you ever run something like ALTER DATABASE 'immich-db' SET role TO 'immich-db';?

@mmomjian commented on GitHub (Jun 22, 2025): I am not sure why you have all these `role=immich-db` setconfigs, nor what they mean. I don't think this is an actual PG param. Have you ever run something like `ALTER DATABASE 'immich-db' SET role TO 'immich-db';`?
OVERLORD commented 2026-02-05 12:13:04 +03:00
Author
Owner
Copy Link

@rstat1 commented on GitHub (Jun 22, 2025):

Yes I did so before reporting this issue, to see if running that command manually would be an effective workaround for the crash loop. Which it seems to have been because after a redeploy its no longer crashing.

@rstat1 commented on GitHub (Jun 22, 2025): Yes I did so before reporting this issue, to see if running that command manually would be an effective workaround for the crash loop. Which it seems to have been because after a redeploy its no longer crashing.
OVERLORD commented 2026-02-05 12:13:04 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 23, 2025):

fixed in #19457

@mmomjian commented on GitHub (Jun 23, 2025): fixed in #19457
OVERLORD commented 2026-02-05 12:13:05 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 23, 2025):

@rstat1 can you please test using the pr-19462 docker image to see if it fixes it? It might be a few minutes until the image is built

@mmomjian commented on GitHub (Jun 23, 2025): @rstat1 can you please test using the `pr-19462` docker image to see if it fixes it? It might be a few minutes until the image is built
OVERLORD commented 2026-02-05 12:13:06 +03:00
Author
Owner
Copy Link

@tgkenney commented on GitHub (Jun 23, 2025):

I'm not sure if this is related or not, but I'm also getting a database migration error [Nest] 7 - 06/23/2025, 10:36:00 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: relation "kysely_migrations_lock" does not exist.

I was using the release docker tag, but I get the same error with that tag you suggested testing pr-19462

Before I pulled that tag though, I was getting a different error [Nest] 7 - 06/23/2025, 10:33:50 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim"

I can create a new issue if this isn't related

@tgkenney commented on GitHub (Jun 23, 2025): I'm not sure if this is related or not, but I'm also getting a database migration error `[Nest] 7 - 06/23/2025, 10:36:00 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: relation "kysely_migrations_lock" does not exist`. I was using the `release` docker tag, but I get the same error with that tag you suggested testing `pr-19462` Before I pulled that tag though, I was getting a different error `[Nest] 7 - 06/23/2025, 10:33:50 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim"` I can create a new issue if this isn't related
OVERLORD commented 2026-02-05 12:13:06 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 23, 2025):

I'm not sure if this is related or not, but I'm also getting a database migration error [Nest] 7 - 06/23/2025, 10:36:00 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: relation "kysely_migrations_lock" does not exist.

I was using the release docker tag, but I get the same error with that tag you suggested testing pr-19462

Before I pulled that tag though, I was getting a different error [Nest] 7 - 06/23/2025, 10:33:50 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim"

I can create a new issue if this isn't related

This sounds like you are using your own postgres database, with immich not having superuser permissions?

Actually, what version of immich were you running when you got "permission denied to set parameter "vchordrq.prewarm_dim"`" ?

@mmomjian commented on GitHub (Jun 23, 2025): > I'm not sure if this is related or not, but I'm also getting a database migration error `[Nest] 7 - 06/23/2025, 10:36:00 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: relation "kysely_migrations_lock" does not exist`. > > I was using the `release` docker tag, but I get the same error with that tag you suggested testing `pr-19462` > > Before I pulled that tag though, I was getting a different error `[Nest] 7 - 06/23/2025, 10:33:50 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim"` > > I can create a new issue if this isn't related This sounds like you are using your own postgres database, with immich not having superuser permissions? Actually, what version of immich were you running when you got "permission denied to set parameter "vchordrq.prewarm_dim"`" ?
OVERLORD commented 2026-02-05 12:13:07 +03:00
Author
Owner
Copy Link

@tgkenney commented on GitHub (Jun 23, 2025):

It does have it's own database on my postgres server yeah, but it should have full permissions. It's been a while since I set it up and it was working fine till this.

Here's the full log, but it's hard to say exactly when it started because the container just keeps looping on this error.
EDIT: Actually, my uptime monitor shows the /api/server/ping endpoint went down at 2025-06-20 02:08:40

Initializing Immich v1.135.1
Detected CPU Cores: 24
Starting api worker
Starting microservices worker
[Nest] 7  - 06/23/2025, 9:36:29 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7  - 06/23/2025, 9:36:29 AM     LOG [Microservices:DatabaseRepository] Running migrations, this may take a while
[Nest] 17  - 06/23/2025, 9:36:29 AM     LOG [Api:EventRepository] Initialized websocket server
Query failed : {
  durationMs: 4.2896660000005795,
  error: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim"
      at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
      at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
      at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
      at Socket.emit (node:events:518:28)
      at addChunk (node:internal/streams/readable:561:12)
      at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
      at Readable.push (node:internal/streams/readable:392:5)
      at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
    severity_local: 'ERROR',
    severity: 'ERROR',
    code: '42501',
    file: 'guc.c',
    line: '11874',
    routine: 'validate_option_array_item'
  },
  sql: 'ALTER DATABASE "immich" RESET vchordrq.prewarm_dim;',
  params: []
}
[Nest] 7  - 06/23/2025, 9:36:30 AM   ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim"
[Nest] 7  - 06/23/2025, 9:36:30 AM     LOG [Microservices:DatabaseRepository] Migration "1750189909087-AddAlbumUserCreateFields" succeeded
[Nest] 7  - 06/23/2025, 9:36:30 AM    WARN [Microservices:DatabaseRepository] Migration "1750323941566-UnsetPrewarmDimParameter" failed
PostgresError: permission denied to set parameter "vchordrq.prewarm_dim"
    at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
    at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
    at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
  severity_local: 'ERROR',
  severity: 'ERROR',
  code: '42501',
  file: 'guc.c',
  line: '11874',
  routine: 'validate_option_array_item'
}
microservices worker error: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim", stack: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim"
    at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
    at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
    at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
microservices worker exited with code 1```
@tgkenney commented on GitHub (Jun 23, 2025): It does have it's own database on my postgres server yeah, but it should have full permissions. It's been a while since I set it up and it was working fine till this. Here's the full log, but it's hard to say exactly when it started because the container just keeps looping on this error. EDIT: Actually, my uptime monitor shows the `/api/server/ping` endpoint went down at 2025-06-20 02:08:40 ``` Initializing Immich v1.135.1 Detected CPU Cores: 24 Starting api worker Starting microservices worker [Nest] 7 - 06/23/2025, 9:36:29 AM LOG [Microservices:EventRepository] Initialized websocket server [Nest] 7 - 06/23/2025, 9:36:29 AM LOG [Microservices:DatabaseRepository] Running migrations, this may take a while [Nest] 17 - 06/23/2025, 9:36:29 AM LOG [Api:EventRepository] Initialized websocket server Query failed : { durationMs: 4.2896660000005795, error: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim" at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) at Socket.emit (node:events:518:28) at addChunk (node:internal/streams/readable:561:12) at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) at Readable.push (node:internal/streams/readable:392:5) at TCP.onStreamRead (node:internal/stream_base_commons:189:23) { severity_local: 'ERROR', severity: 'ERROR', code: '42501', file: 'guc.c', line: '11874', routine: 'validate_option_array_item' }, sql: 'ALTER DATABASE "immich" RESET vchordrq.prewarm_dim;', params: [] } [Nest] 7 - 06/23/2025, 9:36:30 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim" [Nest] 7 - 06/23/2025, 9:36:30 AM LOG [Microservices:DatabaseRepository] Migration "1750189909087-AddAlbumUserCreateFields" succeeded [Nest] 7 - 06/23/2025, 9:36:30 AM WARN [Microservices:DatabaseRepository] Migration "1750323941566-UnsetPrewarmDimParameter" failed PostgresError: permission denied to set parameter "vchordrq.prewarm_dim" at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) at Socket.emit (node:events:518:28) at addChunk (node:internal/streams/readable:561:12) at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) at Readable.push (node:internal/streams/readable:392:5) at TCP.onStreamRead (node:internal/stream_base_commons:189:23) { severity_local: 'ERROR', severity: 'ERROR', code: '42501', file: 'guc.c', line: '11874', routine: 'validate_option_array_item' } microservices worker error: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim", stack: PostgresError: permission denied to set parameter "vchordrq.prewarm_dim" at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) at Socket.emit (node:events:518:28) at addChunk (node:internal/streams/readable:561:12) at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) at Readable.push (node:internal/streams/readable:392:5) at TCP.onStreamRead (node:internal/stream_base_commons:189:23) microservices worker exited with code 1```
OVERLORD commented 2026-02-05 12:13:07 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 23, 2025):

Please upgrade to 1.135.3 I would not expect to see that error on that version

@mmomjian commented on GitHub (Jun 23, 2025): Please upgrade to 1.135.3 I would not expect to see that error on that version
OVERLORD commented 2026-02-05 12:13:08 +03:00
Author
Owner
Copy Link

@tgkenney commented on GitHub (Jun 23, 2025):

Well I was using the release tag which should be equivalent, not sure why it didn't pull that, but I changed it to v1.135.3 and tried again. Same error:

Initializing Immich v1.135.3
Detected CPU Cores: 24
Starting api worker
Starting microservices worker
[Nest] 7  - 06/23/2025, 10:54:29 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7  - 06/23/2025, 10:54:30 AM     LOG [Microservices:DatabaseRepository] Running migrations, this may take a while
Query failed : {
  durationMs: 5.253556000000572,
  error: PostgresError: relation "kysely_migrations_lock" does not exist
      at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
      at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
      at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
      at Socket.emit (node:events:518:28)
      at addChunk (node:internal/streams/readable:561:12)
      at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
      at Readable.push (node:internal/streams/readable:392:5)
      at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
    severity_local: 'ERROR',
    severity: 'ERROR',
    code: '42P01',
    position: '18',
    file: 'parse_relation.c',
    line: '1392',
    routine: 'parserOpenTable'
  },
  sql: 'select "id" from "kysely_migrations_lock" where "id" = $1',
  params: [ 'migration_lock' ]
}
[Nest] 7  - 06/23/2025, 10:54:30 AM   ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: relation "kysely_migrations_lock" does not exist
PostgresError: relation "kysely_migrations_lock" does not exist
    at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
    at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
    at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
  severity_local: 'ERROR',
  severity: 'ERROR',
  code: '42P01',
  position: '18',
  file: 'parse_relation.c',
  line: '1392',
  routine: 'parserOpenTable'
}
microservices worker error: PostgresError: relation "kysely_migrations_lock" does not exist, stack: PostgresError: relation "kysely_migrations_lock" does not exist
    at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
    at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
    at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
microservices worker exited with code 1
Killing api process
@tgkenney commented on GitHub (Jun 23, 2025): Well I was using the `release` tag which should be equivalent, not sure why it didn't pull that, but I changed it to `v1.135.3` and tried again. Same error: ``` Initializing Immich v1.135.3 Detected CPU Cores: 24 Starting api worker Starting microservices worker [Nest] 7 - 06/23/2025, 10:54:29 AM LOG [Microservices:EventRepository] Initialized websocket server [Nest] 7 - 06/23/2025, 10:54:30 AM LOG [Microservices:DatabaseRepository] Running migrations, this may take a while Query failed : { durationMs: 5.253556000000572, error: PostgresError: relation "kysely_migrations_lock" does not exist at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) at Socket.emit (node:events:518:28) at addChunk (node:internal/streams/readable:561:12) at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) at Readable.push (node:internal/streams/readable:392:5) at TCP.onStreamRead (node:internal/stream_base_commons:189:23) { severity_local: 'ERROR', severity: 'ERROR', code: '42P01', position: '18', file: 'parse_relation.c', line: '1392', routine: 'parserOpenTable' }, sql: 'select "id" from "kysely_migrations_lock" where "id" = $1', params: [ 'migration_lock' ] } [Nest] 7 - 06/23/2025, 10:54:30 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: relation "kysely_migrations_lock" does not exist PostgresError: relation "kysely_migrations_lock" does not exist at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) at Socket.emit (node:events:518:28) at addChunk (node:internal/streams/readable:561:12) at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) at Readable.push (node:internal/streams/readable:392:5) at TCP.onStreamRead (node:internal/stream_base_commons:189:23) { severity_local: 'ERROR', severity: 'ERROR', code: '42P01', position: '18', file: 'parse_relation.c', line: '1392', routine: 'parserOpenTable' } microservices worker error: PostgresError: relation "kysely_migrations_lock" does not exist, stack: PostgresError: relation "kysely_migrations_lock" does not exist at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) at Socket.emit (node:events:518:28) at addChunk (node:internal/streams/readable:561:12) at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) at Readable.push (node:internal/streams/readable:392:5) at TCP.onStreamRead (node:internal/stream_base_commons:189:23) microservices worker exited with code 1 Killing api process ```
OVERLORD commented 2026-02-05 12:13:08 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 23, 2025):

Well I was using the release tag which should be equivalent, not sure why it didn't pull that, but I changed it to v1.135.3 and tried again. Same error:

Initializing Immich v1.135.3
Detected CPU Cores: 24
Starting api worker
Starting microservices worker
[Nest] 7  - 06/23/2025, 10:54:29 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7  - 06/23/2025, 10:54:30 AM     LOG [Microservices:DatabaseRepository] Running migrations, this may take a while
Query failed : {
  durationMs: 5.253556000000572,
  error: PostgresError: relation "kysely_migrations_lock" does not exist
      at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
      at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
      at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
      at Socket.emit (node:events:518:28)
      at addChunk (node:internal/streams/readable:561:12)
      at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
      at Readable.push (node:internal/streams/readable:392:5)
      at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
    severity_local: 'ERROR',
    severity: 'ERROR',
    code: '42P01',
    position: '18',
    file: 'parse_relation.c',
    line: '1392',
    routine: 'parserOpenTable'
  },
  sql: 'select "id" from "kysely_migrations_lock" where "id" = $1',
  params: [ 'migration_lock' ]
}
[Nest] 7  - 06/23/2025, 10:54:30 AM   ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: relation "kysely_migrations_lock" does not exist
PostgresError: relation "kysely_migrations_lock" does not exist
    at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
    at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
    at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23) {
  severity_local: 'ERROR',
  severity: 'ERROR',
  code: '42P01',
  position: '18',
  file: 'parse_relation.c',
  line: '1392',
  routine: 'parserOpenTable'
}
microservices worker error: PostgresError: relation "kysely_migrations_lock" does not exist, stack: PostgresError: relation "kysely_migrations_lock" does not exist
    at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26)
    at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6)
    at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
microservices worker exited with code 1
Killing api process

This error is different. Your DB is missing the kysely_migrations_lock table. I am not sure how that could ever happen. Did you have any database crashes/corruption or anything like that? I assume that some instability in the database deployment has resulted in the loss of some tables.

I think you should probably join the discord or make a new discussion thread on github with database details, list of all tables, etc. I don't see how this would be an immich bug, and I've never seen this before.

@mmomjian commented on GitHub (Jun 23, 2025): > Well I was using the `release` tag which should be equivalent, not sure why it didn't pull that, but I changed it to `v1.135.3` and tried again. Same error: > > ``` > Initializing Immich v1.135.3 > Detected CPU Cores: 24 > Starting api worker > Starting microservices worker > [Nest] 7 - 06/23/2025, 10:54:29 AM LOG [Microservices:EventRepository] Initialized websocket server > [Nest] 7 - 06/23/2025, 10:54:30 AM LOG [Microservices:DatabaseRepository] Running migrations, this may take a while > Query failed : { > durationMs: 5.253556000000572, > error: PostgresError: relation "kysely_migrations_lock" does not exist > at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) > at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) > at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) > at Socket.emit (node:events:518:28) > at addChunk (node:internal/streams/readable:561:12) > at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) > at Readable.push (node:internal/streams/readable:392:5) > at TCP.onStreamRead (node:internal/stream_base_commons:189:23) { > severity_local: 'ERROR', > severity: 'ERROR', > code: '42P01', > position: '18', > file: 'parse_relation.c', > line: '1392', > routine: 'parserOpenTable' > }, > sql: 'select "id" from "kysely_migrations_lock" where "id" = $1', > params: [ 'migration_lock' ] > } > [Nest] 7 - 06/23/2025, 10:54:30 AM ERROR [Microservices:DatabaseRepository] Kysely migrations failed: PostgresError: relation "kysely_migrations_lock" does not exist > PostgresError: relation "kysely_migrations_lock" does not exist > at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) > at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) > at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) > at Socket.emit (node:events:518:28) > at addChunk (node:internal/streams/readable:561:12) > at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) > at Readable.push (node:internal/streams/readable:392:5) > at TCP.onStreamRead (node:internal/stream_base_commons:189:23) { > severity_local: 'ERROR', > severity: 'ERROR', > code: '42P01', > position: '18', > file: 'parse_relation.c', > line: '1392', > routine: 'parserOpenTable' > } > microservices worker error: PostgresError: relation "kysely_migrations_lock" does not exist, stack: PostgresError: relation "kysely_migrations_lock" does not exist > at ErrorResponse (/usr/src/app/node_modules/postgres/cjs/src/connection.js:790:26) > at handle (/usr/src/app/node_modules/postgres/cjs/src/connection.js:476:6) > at Socket.data (/usr/src/app/node_modules/postgres/cjs/src/connection.js:315:9) > at Socket.emit (node:events:518:28) > at addChunk (node:internal/streams/readable:561:12) > at readableAddChunkPushByteMode (node:internal/streams/readable:512:3) > at Readable.push (node:internal/streams/readable:392:5) > at TCP.onStreamRead (node:internal/stream_base_commons:189:23) > microservices worker exited with code 1 > Killing api process > ``` This error is different. Your DB is missing the `kysely_migrations_lock` table. I am not sure how that could ever happen. Did you have any database crashes/corruption or anything like that? I assume that some instability in the database deployment has resulted in the loss of some tables. I think you should probably join the discord or make a new discussion thread on github with database details, list of all tables, etc. I don't see how this would be an immich bug, and I've never seen this before.
OVERLORD commented 2026-02-05 12:13:09 +03:00
Author
Owner
Copy Link

@tgkenney commented on GitHub (Jun 23, 2025):

Correcting myself: The previous error Migration "1750323941566-UnsetPrewarmDimParameter" failed was always happening, but then I pulled the pr-19462 tag as suggested earlier, and since then both v1.135.1 and v1.135.3 are giving the Kysely error. There hasn't been any corruption that I'm aware of anyway.

I'll try a rollback of the DB and then pull v1.135.3 just to make sure it's not me

@tgkenney commented on GitHub (Jun 23, 2025): Correcting myself: The previous error `Migration "1750323941566-UnsetPrewarmDimParameter" failed` was always happening, but then I pulled the `pr-19462` tag as suggested earlier, and since then both v1.135.1 and v1.135.3 are giving the Kysely error. There hasn't been any corruption that I'm aware of anyway. I'll try a rollback of the DB and then pull v1.135.3 just to make sure it's not me
OVERLORD commented 2026-02-05 12:13:10 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 23, 2025):

Correcting myself: The previous error Migration "1750323941566-UnsetPrewarmDimParameter" failed was always happening, but then I pulled the pr-19462 tag as suggested earlier, and since then both v1.135.1 and v1.135.3 are giving the Kysely error. There hasn't been any corruption that I'm aware of anyway.

I'll try a rollback of the DB and then pull v1.135.3 just to make sure it's not me

Hmm. Can you please run select * from pg_db_role_setting; in PG?

@mmomjian commented on GitHub (Jun 23, 2025): > Correcting myself: The previous error `Migration "1750323941566-UnsetPrewarmDimParameter" failed` was always happening, but then I pulled the `pr-19462` tag as suggested earlier, and since then both v1.135.1 and v1.135.3 are giving the Kysely error. There hasn't been any corruption that I'm aware of anyway. > > I'll try a rollback of the DB and then pull v1.135.3 just to make sure it's not me Hmm. Can you please run `select * from pg_db_role_setting;` in PG?
OVERLORD commented 2026-02-05 12:13:10 +03:00
Author
Owner
Copy Link

@tgkenney commented on GitHub (Jun 23, 2025):

setdatabase,setrole,setconfig
313608,0,{"search_path=\"\"\"$user\"\", public, vectors\"",role=immich}
@tgkenney commented on GitHub (Jun 23, 2025): ``` setdatabase,setrole,setconfig 313608,0,{"search_path=\"\"\"$user\"\", public, vectors\"",role=immich} ```
OVERLORD commented 2026-02-05 12:13:11 +03:00
Author
Owner
Copy Link

@mmomjian commented on GitHub (Jun 23, 2025):

setdatabase,setrole,setconfig
313608,0,{"search_path=\"\"\"$user\"\", public, vectors\"",role=immich}

If you are using vchord (0.3 or 0.4) please connect to PG manually and run alter database <DB_DATABASE_NAME> reset all; alter database <DB_DATABASE_NAME> set role to immich; and see if that helps anything

Somehow your search_path has an extra set of quotes

@mmomjian commented on GitHub (Jun 23, 2025): > ``` > setdatabase,setrole,setconfig > 313608,0,{"search_path=\"\"\"$user\"\", public, vectors\"",role=immich} > ``` If you are using vchord (0.3 or 0.4) please connect to PG manually and run `alter database <DB_DATABASE_NAME> reset all; alter database <DB_DATABASE_NAME> set role to immich;` and see if that helps anything Somehow your search_path has an extra set of quotes
OVERLORD commented 2026-02-05 12:13:11 +03:00
Author
Owner
Copy Link

@tgkenney commented on GitHub (Jun 23, 2025):

Yeah you can probably ignore my report now, I restored the database to backup from this morning, and brought the container up with v1.135.3 and it came right up

@tgkenney commented on GitHub (Jun 23, 2025): Yeah you can probably ignore my report now, I restored the database to backup from this morning, and brought the container up with v1.135.3 and it came right up
OVERLORD commented 2026-02-05 12:13:12 +03:00
Author
Owner
Copy Link

@rstat1 commented on GitHub (Jun 23, 2025):

@mmomjian

I did what you asked and there were no crashes or errors during startup like there was before .

But I should add I did also manually run the failing query right after reporting this issue, in an attempt to workaround the crash I was seeing

@rstat1 commented on GitHub (Jun 23, 2025): @mmomjian I did what you asked and there were no crashes or errors during startup like there was before . But I should add I did also manually run the failing query right after reporting this issue, in an attempt to workaround the crash I was seeing
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label 🗄️server
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#6344
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?