immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 17 Packages Projects Releases 10 Wiki Activity

[BUG] v1.91.0 Crash Loop Relating to Postgres Statement Error #1799

New Issue
Closed
opened 2026-02-05 03:53:08 +03:00 by OVERLORD · 5 comments
OVERLORD commented 2026-02-05 03:53:08 +03:00
Owner
Copy Link

Originally created by @LocalAreaNitwit on GitHub (Dec 16, 2023).

The bug

After upgrading to v1.91.0 and implementing the breaking change instructions the server container does not start. PostgreSQL has the following error in the log:

2023-12-16 12:41:46.590 UTC [62] ERROR:  assertion failed: !array.contains_nulls()
2023-12-16 12:41:46.590 UTC [62] STATEMENT:  
	        INSERT INTO smart_search("assetId", embedding)
	        SELECT si."assetId", si."clipEmbedding"
	        FROM smart_info si
	        WHERE "clipEmbedding" IS NOT NULL

The OS that Immich Server is running on

Arch Linux/Kubernetes

Version of Immich Server

v1.91.0

Version of Immich Mobile App

N/A

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: immich-server
  name: immich-server
  namespace: immich
spec:
  replicas: 3
  selector:
    matchLabels:
      app: immich-server
  template:
    metadata:
      labels:
        app: immich-server
    spec:
      volumes:
        - name: immich-photos
          persistentVolumeClaim:
            claimName: immich-photos
        - name: geocoding-dump
          emptyDir: {}
        - name: transformers-cache
          emptyDir: {}
      containers:
        - name: immich
          image: docker.io/altran1502/immich-server:v1.91.0
          imagePullPolicy: IfNotPresent
          args: ["start.sh", "immich"]
          livenessProbe:
            httpGet:
              path: /server-info/ping
              port: 3001
            initialDelaySeconds: 5
            failureThreshold: 3
            timeoutSeconds: 1
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /server-info/ping
              port: 3001
            initialDelaySeconds: 5
            failureThreshold: 3
            timeoutSeconds: 1
            periodSeconds: 10
          resources:
            limits:
              cpu: 500m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 200Mi
          env:
            - name: DB_HOSTNAME
              value: immich-psql
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_USER
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_PASSWORD
            - name: DB_DATABASE_NAME
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_DB
            - name: REDIS_HOSTNAME
              value: immich-redis
            - name: IMMICH_MACHINE_LEARNING_URL
              value: http://immich-machine-learning:3003
          volumeMounts:
            - mountPath: /usr/src/app/upload
              name: immich-photos
            - name: geocoding-dump
              mountPath: /usr/src/app/.reverse-geocoding-dump
            - name: transformers-cache
              mountPath: /usr/src/app/.transformers_cache
      restartPolicy: Always
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: immich-machine-learning
  name: immich-machine-learning
  namespace: immich
spec:
  replicas: 1
  selector:
    matchLabels:
      app: immich-machine-learning
  template:
    metadata:
      labels:
        app: immich-machine-learning
    spec:
      volumes:
        - name: immich-photos
          persistentVolumeClaim:
            claimName: immich-photos
        - name: immich-cache
          emptyDir: {}
        - name: geocoding-dump
          emptyDir: {}
        - name: transformers-cache
          emptyDir: {}
      containers:
        - name: immich-machine-learning
          image: docker.io/altran1502/immich-machine-learning:v1.91.0
          imagePullPolicy: IfNotPresent
          livenessProbe:
            tcpSocket:
              port: 3003
            initialDelaySeconds: 0
            failureThreshold: 3
            timeoutSeconds: 1
            periodSeconds: 10
          readinessProbe:
            tcpSocket:
              port: 3003
            initialDelaySeconds: 0
            failureThreshold: 3
            timeoutSeconds: 1
            periodSeconds: 10
          startupProbe:
            tcpSocket:
              port: 3003
            initialDelaySeconds: 0
            failureThreshold: 30
            timeoutSeconds: 1
            periodSeconds: 5
          resources:
            limits:
              cpu: 2000m
              memory: 2048Mi
            requests:
              cpu: 200m
              memory: 200Mi
          env:
            - name: DB_HOSTNAME
              value: immich-psql
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_USER
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_PASSWORD
            - name: DB_DATABASE_NAME
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_DB
            - name: REDIS_HOSTNAME
              value: immich-redis
            - name: TRANSFORMERS_CACHE
              value: /usr/src/app/.transformers_cache
          volumeMounts:
            - mountPath: /usr/src/app/upload
              name: immich-photos
            - mountPath: /cache
              name: immich-cache
            - name: geocoding-dump
              mountPath: /usr/src/app/.reverse-geocoding-dump
            - name: transformers-cache
              mountPath: /usr/src/app/.transformers_cache
      restartPolicy: Always
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: immich-microservice
  name: immich-microservice
  namespace: immich
spec:
  replicas: 1
  selector:
    matchLabels:
      app: immich-microservice
  template:
    metadata:
      labels:
        app: immich-microservice
    spec:
      volumes:
        - name: immich-photos
          persistentVolumeClaim:
            claimName: immich-photos
        - name: geocoding-dump
          emptyDir: {}
        - name: transformers-cache
          emptyDir: {}
      containers:
        - name: immich
          image: docker.io/altran1502/immich-server:v1.91.0
          imagePullPolicy: IfNotPresent
          args: ["start.sh", "microservices"]
          resources:
            limits:
              cpu: 2000m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 512Mi
          env:
            - name: DB_HOSTNAME
              value: immich-psql
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_USER
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_PASSWORD
            - name: DB_DATABASE_NAME
              valueFrom:
                secretKeyRef:
                  name: immich-database
                  key: POSTGRES_DB
            - name: REDIS_HOSTNAME
              value: immich-redis
            - name: IMMICH_MACHINE_LEARNING_URL
              value: http://immich-machine-learning:3003
          volumeMounts:
            - mountPath: /usr/src/app/upload
              name: immich-photos
            - name: geocoding-dump
              mountPath: /usr/src/app/.reverse-geocoding-dump
            - name: transformers-cache
              mountPath: /usr/src/app/.transformers_cache
      restartPolicy: Always

Your .env content

N/A

Reproduction steps

This is running on Kubernetes, the following changes have been made to adhere to the breaking change:

1. PostgreSQL image has been changed to tensorchord/pgvecto-rs:pg15-v0.1.11
2. Typesense deployment removed, including environment variable references

Additional information

No response

Originally created by @LocalAreaNitwit on GitHub (Dec 16, 2023). ### The bug After upgrading to v1.91.0 and implementing the breaking change instructions the server container does not start. PostgreSQL has the following error in the log: ``` 2023-12-16 12:41:46.590 UTC [62] ERROR: assertion failed: !array.contains_nulls() 2023-12-16 12:41:46.590 UTC [62] STATEMENT: INSERT INTO smart_search("assetId", embedding) SELECT si."assetId", si."clipEmbedding" FROM smart_info si WHERE "clipEmbedding" IS NOT NULL ``` ### The OS that Immich Server is running on Arch Linux/Kubernetes ### Version of Immich Server v1.91.0 ### Version of Immich Mobile App N/A ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML apiVersion: apps/v1 kind: Deployment metadata: labels: app: immich-server name: immich-server namespace: immich spec: replicas: 3 selector: matchLabels: app: immich-server template: metadata: labels: app: immich-server spec: volumes: - name: immich-photos persistentVolumeClaim: claimName: immich-photos - name: geocoding-dump emptyDir: {} - name: transformers-cache emptyDir: {} containers: - name: immich image: docker.io/altran1502/immich-server:v1.91.0 imagePullPolicy: IfNotPresent args: ["start.sh", "immich"] livenessProbe: httpGet: path: /server-info/ping port: 3001 initialDelaySeconds: 5 failureThreshold: 3 timeoutSeconds: 1 periodSeconds: 10 readinessProbe: httpGet: path: /server-info/ping port: 3001 initialDelaySeconds: 5 failureThreshold: 3 timeoutSeconds: 1 periodSeconds: 10 resources: limits: cpu: 500m memory: 1024Mi requests: cpu: 100m memory: 200Mi env: - name: DB_HOSTNAME value: immich-psql - name: DB_USERNAME valueFrom: secretKeyRef: name: immich-database key: POSTGRES_USER - name: DB_PASSWORD valueFrom: secretKeyRef: name: immich-database key: POSTGRES_PASSWORD - name: DB_DATABASE_NAME valueFrom: secretKeyRef: name: immich-database key: POSTGRES_DB - name: REDIS_HOSTNAME value: immich-redis - name: IMMICH_MACHINE_LEARNING_URL value: http://immich-machine-learning:3003 volumeMounts: - mountPath: /usr/src/app/upload name: immich-photos - name: geocoding-dump mountPath: /usr/src/app/.reverse-geocoding-dump - name: transformers-cache mountPath: /usr/src/app/.transformers_cache restartPolicy: Always --- apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: immich-machine-learning name: immich-machine-learning namespace: immich spec: replicas: 1 selector: matchLabels: app: immich-machine-learning template: metadata: labels: app: immich-machine-learning spec: volumes: - name: immich-photos persistentVolumeClaim: claimName: immich-photos - name: immich-cache emptyDir: {} - name: geocoding-dump emptyDir: {} - name: transformers-cache emptyDir: {} containers: - name: immich-machine-learning image: docker.io/altran1502/immich-machine-learning:v1.91.0 imagePullPolicy: IfNotPresent livenessProbe: tcpSocket: port: 3003 initialDelaySeconds: 0 failureThreshold: 3 timeoutSeconds: 1 periodSeconds: 10 readinessProbe: tcpSocket: port: 3003 initialDelaySeconds: 0 failureThreshold: 3 timeoutSeconds: 1 periodSeconds: 10 startupProbe: tcpSocket: port: 3003 initialDelaySeconds: 0 failureThreshold: 30 timeoutSeconds: 1 periodSeconds: 5 resources: limits: cpu: 2000m memory: 2048Mi requests: cpu: 200m memory: 200Mi env: - name: DB_HOSTNAME value: immich-psql - name: DB_USERNAME valueFrom: secretKeyRef: name: immich-database key: POSTGRES_USER - name: DB_PASSWORD valueFrom: secretKeyRef: name: immich-database key: POSTGRES_PASSWORD - name: DB_DATABASE_NAME valueFrom: secretKeyRef: name: immich-database key: POSTGRES_DB - name: REDIS_HOSTNAME value: immich-redis - name: TRANSFORMERS_CACHE value: /usr/src/app/.transformers_cache volumeMounts: - mountPath: /usr/src/app/upload name: immich-photos - mountPath: /cache name: immich-cache - name: geocoding-dump mountPath: /usr/src/app/.reverse-geocoding-dump - name: transformers-cache mountPath: /usr/src/app/.transformers_cache restartPolicy: Always --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: immich-microservice name: immich-microservice namespace: immich spec: replicas: 1 selector: matchLabels: app: immich-microservice template: metadata: labels: app: immich-microservice spec: volumes: - name: immich-photos persistentVolumeClaim: claimName: immich-photos - name: geocoding-dump emptyDir: {} - name: transformers-cache emptyDir: {} containers: - name: immich image: docker.io/altran1502/immich-server:v1.91.0 imagePullPolicy: IfNotPresent args: ["start.sh", "microservices"] resources: limits: cpu: 2000m memory: 1024Mi requests: cpu: 100m memory: 512Mi env: - name: DB_HOSTNAME value: immich-psql - name: DB_USERNAME valueFrom: secretKeyRef: name: immich-database key: POSTGRES_USER - name: DB_PASSWORD valueFrom: secretKeyRef: name: immich-database key: POSTGRES_PASSWORD - name: DB_DATABASE_NAME valueFrom: secretKeyRef: name: immich-database key: POSTGRES_DB - name: REDIS_HOSTNAME value: immich-redis - name: IMMICH_MACHINE_LEARNING_URL value: http://immich-machine-learning:3003 volumeMounts: - mountPath: /usr/src/app/upload name: immich-photos - name: geocoding-dump mountPath: /usr/src/app/.reverse-geocoding-dump - name: transformers-cache mountPath: /usr/src/app/.transformers_cache restartPolicy: Always ``` ### Your .env content ```Shell N/A ``` ### Reproduction steps ```bash This is running on Kubernetes, the following changes have been made to adhere to the breaking change: 1. PostgreSQL image has been changed to tensorchord/pgvecto-rs:pg15-v0.1.11 2. Typesense deployment removed, including environment variable references ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 03:53:15 +03:00
Author
Owner
Copy Link

@Arbide commented on GitHub (Dec 16, 2023):

I had very similar problems after updating 1 hour ago.
You may need to assing more memory to PostgreSQL.

@Arbide commented on GitHub (Dec 16, 2023): I had very similar problems after updating 1 hour ago. You may need to assing more memory to PostgreSQL.
OVERLORD commented 2026-02-05 03:53:21 +03:00
Author
Owner
Copy Link

@LocalAreaNitwit commented on GitHub (Dec 16, 2023):

I had very similar problems after updating 1 hour ago. You may need to assing more memory to PostgreSQL.

Thanks for the reply!
The PostgreSQL deployment has no limits set currently. The node it's running on has 4GB free memory.

Did you end up with success?

@LocalAreaNitwit commented on GitHub (Dec 16, 2023): > I had very similar problems after updating 1 hour ago. You may need to assing more memory to PostgreSQL. Thanks for the reply! The PostgreSQL deployment has no limits set currently. The node it's running on has 4GB free memory. Did you end up with success?
OVERLORD commented 2026-02-05 03:53:23 +03:00
Author
Owner
Copy Link

@Arbide commented on GitHub (Dec 16, 2023):

I had very similar problems after updating 1 hour ago. You may need to assing more memory to PostgreSQL.

Thanks for the reply!
The PostgreSQL deployment has no limits set currently. The node it's running on has 4GB free memory.

Did you end up with success?

In my case, after adding memory to db container, all the errors was solved.

@Arbide commented on GitHub (Dec 16, 2023): > > I had very similar problems after updating 1 hour ago. You may need to assing more memory to PostgreSQL. > > Thanks for the reply! > The PostgreSQL deployment has no limits set currently. The node it's running on has 4GB free memory. > > Did you end up with success? In my case, after adding memory to db container, all the errors was solved.
OVERLORD commented 2026-02-05 03:53:26 +03:00
Author
Owner
Copy Link

@alexsands commented on GitHub (Dec 16, 2023):

I'm also experiencing this error without any apparent limitations on memory (6GB allocated and only ~1GB used). It looks like the clipEmbedding column was never created in smart_info:

immich=# \d immich.public.smart_info
                                                                                                    Tab
le "public.smart_info"
            Column             |   Type   | Collation | Nullable |                                     
                                       Default                                                         
                   
-------------------------------+----------+-----------+----------+-------------------------------------
-------------------------------------------------------------------------------------------------------
-------------------
 assetId                       | uuid     |           | not null | 
 tags                          | text[]   |           |          | 
 objects                       | text[]   |           |          | 
 smartInfoTextSearchableColumn | tsvector |           | not null | generated always as (to_tsvector('en
glish'::regconfig, f_concat_ws(' '::text, COALESCE(tags, ARRAY[]::text[]) || COALESCE(objects, ARRAY[]:
:text[])))) stored
Indexes:
    "PK_5e3753aadd956110bf3ec0244ac" PRIMARY KEY, btree ("assetId")
    "si_tags" gin (tags)
    "smart_info_text_searchable_idx" gin ("smartInfoTextSearchableColumn")
Foreign-key constraints:
    "FK_5e3753aadd956110bf3ec0244ac" FOREIGN KEY ("assetId") REFERENCES assets(id) ON DELETE CASCADE

Is there any advice for manually creating this column?

@alexsands commented on GitHub (Dec 16, 2023): I'm also experiencing this error without any apparent limitations on memory (6GB allocated and only ~1GB used). It looks like the `clipEmbedding` column was never created in `smart_info`: ``` immich=# \d immich.public.smart_info Tab le "public.smart_info" Column | Type | Collation | Nullable | Default -------------------------------+----------+-----------+----------+------------------------------------- ------------------------------------------------------------------------------------------------------- ------------------- assetId | uuid | | not null | tags | text[] | | | objects | text[] | | | smartInfoTextSearchableColumn | tsvector | | not null | generated always as (to_tsvector('en glish'::regconfig, f_concat_ws(' '::text, COALESCE(tags, ARRAY[]::text[]) || COALESCE(objects, ARRAY[]: :text[])))) stored Indexes: "PK_5e3753aadd956110bf3ec0244ac" PRIMARY KEY, btree ("assetId") "si_tags" gin (tags) "smart_info_text_searchable_idx" gin ("smartInfoTextSearchableColumn") Foreign-key constraints: "FK_5e3753aadd956110bf3ec0244ac" FOREIGN KEY ("assetId") REFERENCES assets(id) ON DELETE CASCADE ``` Is there any advice for manually creating this column?
OVERLORD commented 2026-02-05 03:53:31 +03:00
Author
Owner
Copy Link

@alexsands commented on GitHub (Dec 16, 2023):

Actually, looking into this further I think my error message is slightly different than the one @LocalAreaNitwit is experiencing (I'll open a separate issue if the clipEmbedding error persists, but I was able to get the server container to start. @LocalAreaNitwit, you may be able to resolve the clipEmbedding NULL assertion error using the fix that was mentioned in #5740.

@alexsands commented on GitHub (Dec 16, 2023): Actually, looking into this further I think my error message is slightly different than the one @LocalAreaNitwit is experiencing (I'll open a separate issue if the `clipEmbedding` error persists, but I was able to get the server container to start. @LocalAreaNitwit, you may be able to resolve the clipEmbedding NULL assertion error using the fix that was mentioned in #5740.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1799
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?