immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 08:41:28 +03:00
Code Issues 26 Packages Projects Releases 10 Wiki Activity

[BUG] pgvecto.rs: IPC connection is closed unexpected. #1847

New Issue
Closed
opened 2026-02-05 04:10:45 +03:00 by OVERLORD · 5 comments
OVERLORD commented 2026-02-05 04:10:45 +03:00
Owner
Copy Link

Originally created by @mglants on GitHub (Dec 20, 2023).

The bug

[Nest] 8  - 12/20/2023, 1:24:29 AM   ERROR [JobService] QueryFailedError: pgvecto.rs: pgvecto.rs: IPC connection is closed unexpected.
ADVICE: The error is raisen by background worker errors. Please check the full Postgresql log to get more information.
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async SmartInfoRepository.upsertEmbedding (/usr/src/app/dist/infra/repositories/smart-info.repository.js:110:9)
    at async SmartInfoRepository.upsert (/usr/src/app/dist/infra/repositories/smart-info.repository.js:103:9)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:103:9)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)

The OS that Immich Server is running on

Kubernetes, 1.27

Version of Immich Server

v1.91.4

Version of Immich Mobile App

v1.0.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "8"
    meta.helm.sh/release-name: immich-microservices
    meta.helm.sh/release-namespace: apps
    reloader.stakater.com/auto: "true"
  creationTimestamp: "2023-11-07T23:53:01Z"
  generation: 9
  labels:
    app.kubernetes.io/component: main
    app.kubernetes.io/instance: immich-microservices
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: immich-microservices
    helm.sh/chart: app-template-2.0.3
    helm.toolkit.fluxcd.io/name: immich-microservices
    helm.toolkit.fluxcd.io/namespace: apps
  name: immich-microservices
  namespace: apps
  resourceVersion: "277312664"
  uid: b1aa4cb1-9e7e-4e70-ba5a-c2010d7c5661
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app.kubernetes.io/component: main
      app.kubernetes.io/instance: immich-microservices
      app.kubernetes.io/name: immich-microservices
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: main
        app.kubernetes.io/instance: immich-microservices
        app.kubernetes.io/name: immich-microservices
    spec:
      automountServiceAccountToken: true
      containers:
      - args:
        - start-microservices.sh
        env:
        - name: DB_DATABASE_NAME
          valueFrom:
            secretKeyRef:
              key: dbname
              name: immich-pgsql-pguser-immich
        - name: DB_HOSTNAME
          valueFrom:
            secretKeyRef:
              key: host
              name: immich-pgsql-pguser-immich
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: immich-pgsql-pguser-immich
        - name: DB_PORT
          value: "5432"
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              key: user
              name: immich-pgsql-pguser-immich
        - name: DISABLE_REVERSE_GEOCODING
          value: "true"
        - name: IMMICH_MEDIA_LOCATION
          value: /usr/src/app/upload
        - name: LOG_LEVEL
          value: log
        - name: MICROSERVICES_PORT
          value: "3002"
        - name: REDIS_HOSTNAME
          valueFrom:
            configMapKeyRef:
              key: REDIS_HOSTNAME
              name: immich-configmap
        - name: REDIS_PORT
          valueFrom:
            configMapKeyRef:
              key: REDIS_PORT
              name: immich-configmap
        - name: STAKATER_IMMICH_CONFIGMAP_CONFIGMAP
          value: 0cd40a1ba7a4c36cbfd8a510784480b5a1973ea3
        image: ghcr.io/immich-app/immich-server:v1.91.4
        imagePullPolicy: IfNotPresent
        name: main
        resources:
          limits:
            memory: 8000Mi
          requests:
            cpu: 100m
            memory: 512Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/src/app/.reverse-geocoding-dump
          name: geocoding-dump
        - mountPath: /usr/src/app/node_modules/local-reverse-geocoder/geonames_dump
          name: geoname-dump
        - mountPath: /usr/src/app/upload
          name: library
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 568
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 568
        runAsUser: 568
        supplementalGroups:
        - 44
        - 109
        - 65542
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/name: immich-microservices
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
      volumes:
      - emptyDir: {}
        name: geocoding-dump
      - emptyDir: {}
        name: geoname-dump
      - name: library
        persistentVolumeClaim:
          claimName: immich-nfs
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-11-07T23:53:01Z"
    lastUpdateTime: "2023-12-20T01:15:12Z"
    message: ReplicaSet "immich-microservices-68ccc9ccf4" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2023-12-20T01:23:56Z"
    lastUpdateTime: "2023-12-20T01:23:56Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 9
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1




### Your .env content

```Shell
- name: DB_DATABASE_NAME
          valueFrom:
            secretKeyRef:
              key: dbname
              name: immich-pgsql-pguser-immich
        - name: DB_HOSTNAME
          valueFrom:
            secretKeyRef:
              key: host
              name: immich-pgsql-pguser-immich
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: immich-pgsql-pguser-immich
        - name: DB_PORT
          value: "5432"
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              key: user
              name: immich-pgsql-pguser-immich
        - name: DISABLE_REVERSE_GEOCODING
          value: "true"
        - name: IMMICH_MEDIA_LOCATION
          value: /usr/src/app/upload
        - name: LOG_LEVEL
          value: log
        - name: MICROSERVICES_PORT
          value: "3002"
        - name: REDIS_HOSTNAME
          valueFrom:
            configMapKeyRef:
              key: REDIS_HOSTNAME
              name: immich-configmap
        - name: REDIS_PORT
          valueFrom:
            configMapKeyRef:
              key: REDIS_PORT
              name: immich-configmap

Reproduction steps

1. Run microservices, any task running is errored with  pgvecto.rs:

Additional information

pgvecto.rs = 0.1.11

Originally created by @mglants on GitHub (Dec 20, 2023). ### The bug ``` [Nest] 8 - 12/20/2023, 1:24:29 AM ERROR [JobService] QueryFailedError: pgvecto.rs: pgvecto.rs: IPC connection is closed unexpected. ADVICE: The error is raisen by background worker errors. Please check the full Postgresql log to get more information. at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:211:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33) at async SmartInfoRepository.upsertEmbedding (/usr/src/app/dist/infra/repositories/smart-info.repository.js:110:9) at async SmartInfoRepository.upsert (/usr/src/app/dist/infra/repositories/smart-info.repository.js:103:9) at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:103:9) at async /usr/src/app/dist/domain/job/job.service.js:113:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24) ``` ### The OS that Immich Server is running on Kubernetes, 1.27 ### Version of Immich Server v1.91.4 ### Version of Immich Mobile App v1.0.0 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "8" meta.helm.sh/release-name: immich-microservices meta.helm.sh/release-namespace: apps reloader.stakater.com/auto: "true" creationTimestamp: "2023-11-07T23:53:01Z" generation: 9 labels: app.kubernetes.io/component: main app.kubernetes.io/instance: immich-microservices app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: immich-microservices helm.sh/chart: app-template-2.0.3 helm.toolkit.fluxcd.io/name: immich-microservices helm.toolkit.fluxcd.io/namespace: apps name: immich-microservices namespace: apps resourceVersion: "277312664" uid: b1aa4cb1-9e7e-4e70-ba5a-c2010d7c5661 spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 3 selector: matchLabels: app.kubernetes.io/component: main app.kubernetes.io/instance: immich-microservices app.kubernetes.io/name: immich-microservices strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app.kubernetes.io/component: main app.kubernetes.io/instance: immich-microservices app.kubernetes.io/name: immich-microservices spec: automountServiceAccountToken: true containers: - args: - start-microservices.sh env: - name: DB_DATABASE_NAME valueFrom: secretKeyRef: key: dbname name: immich-pgsql-pguser-immich - name: DB_HOSTNAME valueFrom: secretKeyRef: key: host name: immich-pgsql-pguser-immich - name: DB_PASSWORD valueFrom: secretKeyRef: key: password name: immich-pgsql-pguser-immich - name: DB_PORT value: "5432" - name: DB_USERNAME valueFrom: secretKeyRef: key: user name: immich-pgsql-pguser-immich - name: DISABLE_REVERSE_GEOCODING value: "true" - name: IMMICH_MEDIA_LOCATION value: /usr/src/app/upload - name: LOG_LEVEL value: log - name: MICROSERVICES_PORT value: "3002" - name: REDIS_HOSTNAME valueFrom: configMapKeyRef: key: REDIS_HOSTNAME name: immich-configmap - name: REDIS_PORT valueFrom: configMapKeyRef: key: REDIS_PORT name: immich-configmap - name: STAKATER_IMMICH_CONFIGMAP_CONFIGMAP value: 0cd40a1ba7a4c36cbfd8a510784480b5a1973ea3 image: ghcr.io/immich-app/immich-server:v1.91.4 imagePullPolicy: IfNotPresent name: main resources: limits: memory: 8000Mi requests: cpu: 100m memory: 512Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /usr/src/app/.reverse-geocoding-dump name: geocoding-dump - mountPath: /usr/src/app/node_modules/local-reverse-geocoder/geonames_dump name: geoname-dump - mountPath: /usr/src/app/upload name: library dnsPolicy: ClusterFirst enableServiceLinks: false restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 568 fsGroupChangePolicy: OnRootMismatch runAsGroup: 568 runAsUser: 568 supplementalGroups: - 44 - 109 - 65542 serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 topologySpreadConstraints: - labelSelector: matchLabels: app.kubernetes.io/name: immich-microservices maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule volumes: - emptyDir: {} name: geocoding-dump - emptyDir: {} name: geoname-dump - name: library persistentVolumeClaim: claimName: immich-nfs status: availableReplicas: 1 conditions: - lastTransitionTime: "2023-11-07T23:53:01Z" lastUpdateTime: "2023-12-20T01:15:12Z" message: ReplicaSet "immich-microservices-68ccc9ccf4" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing - lastTransitionTime: "2023-12-20T01:23:56Z" lastUpdateTime: "2023-12-20T01:23:56Z" message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available observedGeneration: 9 readyReplicas: 1 replicas: 1 updatedReplicas: 1 ``` ``` ### Your .env content ```Shell - name: DB_DATABASE_NAME valueFrom: secretKeyRef: key: dbname name: immich-pgsql-pguser-immich - name: DB_HOSTNAME valueFrom: secretKeyRef: key: host name: immich-pgsql-pguser-immich - name: DB_PASSWORD valueFrom: secretKeyRef: key: password name: immich-pgsql-pguser-immich - name: DB_PORT value: "5432" - name: DB_USERNAME valueFrom: secretKeyRef: key: user name: immich-pgsql-pguser-immich - name: DISABLE_REVERSE_GEOCODING value: "true" - name: IMMICH_MEDIA_LOCATION value: /usr/src/app/upload - name: LOG_LEVEL value: log - name: MICROSERVICES_PORT value: "3002" - name: REDIS_HOSTNAME valueFrom: configMapKeyRef: key: REDIS_HOSTNAME name: immich-configmap - name: REDIS_PORT valueFrom: configMapKeyRef: key: REDIS_PORT name: immich-configmap ``` ### Reproduction steps ```bash 1. Run microservices, any task running is errored with pgvecto.rs: ``` ### Additional information pgvecto.rs = 0.1.11
OVERLORD commented 2026-02-05 04:10:58 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Dec 20, 2023):

This sounds like it's probably a pgvecto.rs issue. Can you post the postgres log?

@bo0tzz commented on GitHub (Dec 20, 2023): This sounds like it's probably a pgvecto.rs issue. Can you post the postgres log?
OVERLORD commented 2026-02-05 04:11:05 +03:00
Author
Owner
Copy Link

@VoVAllen commented on GitHub (Dec 21, 2023):

I'm the developer of pgvecto.rs and sorry to hear that. May I ask what's your system version and kernel version? And does this error happen immediately after upgrade?

@VoVAllen commented on GitHub (Dec 21, 2023): I'm the developer of pgvecto.rs and sorry to hear that. May I ask what's your system version and kernel version? And does this error happen immediately after upgrade?
OVERLORD commented 2026-02-05 04:11:12 +03:00
Author
Owner
Copy Link

@mathieugfortin commented on GitHub (Jan 23, 2024):

FYI, yesterday I did a fresh reinstall of version of 1.93.3, nuked the DB, nuked the immich photo directory and had the same pgvecto and postgres errors. Asset import worked were not impacted.

I just did a compose down/update stack and now the face detection job runs normally

@mathieugfortin commented on GitHub (Jan 23, 2024): FYI, yesterday I did a fresh reinstall of version of 1.93.3, nuked the DB, nuked the immich photo directory and had the same pgvecto and postgres errors. Asset import worked were not impacted. I just did a compose down/update stack and now the face detection job runs normally
OVERLORD commented 2026-02-05 04:11:16 +03:00
Author
Owner
Copy Link

@YentlFrickx commented on GitHub (Feb 1, 2024):

I have the same issue, restarted db, was successful, but following error was shown:

"[2024-02-01T18:28:24Z ERROR vectors::bgworker] Panickied. Info: PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Error(\"unknown variant `Cosine`, expected one of `L2`, `Cos`, `Dot`\", line: 1, column: 67)), location: Location { file: \"crates/service/src/utils/file_atomic.rs\", line: 51, col: 52 }, can_unwind: true, force_no_backtrace: false }. Backtrace: disabled backtrace."

After that deleted immich-server pod, that fails to start and then following logs appear from postgres:

"2024-02-01 18:28:51.940 UTC [38] ERROR:  pgvecto.rs: pgvecto.rs: IPC connection is closed unexpected."
"\tADVICE: The error is raisen by background worker errors. Please check the full Postgresql log to get more information."
"2024-02-01 18:28:51.940 UTC [38] STATEMENT:  COMMIT"
"2024-02-01 18:28:51.981 UTC [38] WARNING:  there is no transaction in progress"

Small update: reverting to previous version (v1.93.3) works, so seems to be something in immich (didn't even restart the db pod)

@YentlFrickx commented on GitHub (Feb 1, 2024): I have the same issue, restarted db, was successful, but following error was shown: ``` "[2024-02-01T18:28:24Z ERROR vectors::bgworker] Panickied. Info: PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Error(\"unknown variant `Cosine`, expected one of `L2`, `Cos`, `Dot`\", line: 1, column: 67)), location: Location { file: \"crates/service/src/utils/file_atomic.rs\", line: 51, col: 52 }, can_unwind: true, force_no_backtrace: false }. Backtrace: disabled backtrace." ``` After that deleted immich-server pod, that fails to start and then following logs appear from postgres: ``` "2024-02-01 18:28:51.940 UTC [38] ERROR: pgvecto.rs: pgvecto.rs: IPC connection is closed unexpected." "\tADVICE: The error is raisen by background worker errors. Please check the full Postgresql log to get more information." "2024-02-01 18:28:51.940 UTC [38] STATEMENT: COMMIT" "2024-02-01 18:28:51.981 UTC [38] WARNING: there is no transaction in progress" ``` Small update: reverting to previous version (v1.93.3) works, so seems to be something in immich (didn't even restart the db pod)
OVERLORD commented 2026-02-05 04:11:27 +03:00
Author
Owner
Copy Link

@nothing2obvi commented on GitHub (Jun 11, 2024):

I'm getting this error on v1.105.1. What do I do?

@nothing2obvi commented on GitHub (Jun 11, 2024): I'm getting this error on `v1.105.1`. What do I do?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1847
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?