error: could not find function "vectors_amhandler_wrapper" on 1.94.1 arm64 image (pg16) #2100

Closed
opened 2026-02-05 05:06:39 +03:00 by OVERLORD · 4 comments
Owner

Originally created by @prometheanfire on GitHub (Feb 1, 2024).

The bug

it looks like a driver is missing on the arm64 image?

Migration "SetAssetFaceNullOnPersonDelete1704943345360" failed, error: could not find function "vectors_amhandler_wrapper" in file "/usr/lib/postgresql/16/lib/vectors.so"
/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219
            throw new QueryFailedError_1.QueryFailedError(query, parameters, err);
                  ^

QueryFailedError: could not find function "vectors_amhandler_wrapper" in file "/usr/lib/postgresql/16/lib/vectors.so"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async SetAssetFaceNullOnPersonDelete1704943345360.up (/usr/src/app/dist/infra/migrations/1704943345360-SetAssetFaceNullOnPersonDelete.js:6:9)
    at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
    at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:263:35)
    at async DatabaseRepository.runMigrations (/usr/src/app/dist/infra/repositories/database.repository.js:42:9)
    at async DatabaseService.init (/usr/src/app/dist/domain/database/database.service.js:33:9)
    at async AppService.init (/usr/src/app/dist/immich/app.service.js:56:9)
    at async AppModule.onModuleInit (/usr/src/app/dist/immich/app.module.js:32:9)
    at async callModuleInitHook (/usr/src/app/node_modules/@nestjs/core/hooks/on-module-init.hook.js:51:9) {
  query: '\n' +
    '            ALTER TABLE "asset_faces"\n' +
    '            DROP CONSTRAINT "FK_95ad7106dd7b484275443f580f9",\n' +
    '            ADD CONSTRAINT "FK_95ad7106dd7b484275443f580f9" \n' +
    '            FOREIGN KEY ("personId") REFERENCES "person"("id")\n' +
    '            ON DELETE SET NULL ON UPDATE CASCADE\n' +
    '    ',
  parameters: undefined,
  driverError: error: could not find function "vectors_amhandler_wrapper" in file "/usr/lib/postgresql/16/lib/vectors.so"
      at /usr/src/app/node_modules/pg/lib/client.js:526:17
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
      at async SetAssetFaceNullOnPersonDelete1704943345360.up (/usr/src/app/dist/infra/migrations/1704943345360-SetAssetFaceNullOnPersonDelete.js:6:9)
      at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
      at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:263:35)
      at async DatabaseRepository.runMigrations (/usr/src/app/dist/infra/repositories/database.repository.js:42:9)
      at async DatabaseService.init (/usr/src/app/dist/domain/database/database.service.js:33:9)
      at async AppService.init (/usr/src/app/dist/immich/app.service.js:56:9)
      at async AppModule.onModuleInit (/usr/src/app/dist/immich/app.module.js:32:9) {
    length: 386,
    severity: 'ERROR',
    code: '42883',
    detail: undefined,
    hint: undefined,
    position: undefined,
    internalPosition: undefined,
    internalQuery: undefined,
    where: 'SQL statement "SELECT fk."personId" FROM ONLY "public"."asset_faces" fk LEFT OUTER JOIN ONLY "public"."person" pk ON ( pk."id" OPERATOR(pg_catalog.=) fk."personId") WHERE pk."id" IS NULL AND (fk."personId" IS NOT NULL)"',
    schema: undefined,
    table: undefined,
    column: undefined,
    dataType: undefined,
    constraint: undefined,
    file: 'dfmgr.c',
    line: '127',
    routine: 'load_external_function'
  },
  length: 386,
  severity: 'ERROR',
  code: '42883',
  detail: undefined,
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: 'SQL statement "SELECT fk."personId" FROM ONLY "public"."asset_faces" fk LEFT OUTER JOIN ONLY "public"."person" pk ON ( pk."id" OPERATOR(pg_catalog.=) fk."personId") WHERE pk."id" IS NULL AND (fk."personId" IS NOT NULL)"',
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'dfmgr.c',
  line: '127',
  routine: 'load_external_function'
}

Node.js v20.11.0

The OS that Immich Server is running on

kubernetes deployment

Version of Immich Server

v1.94.1@sha256:e0655cdd4351a8406eb87ec379a9c45e7bc2a9d051d25c93e2953d92e01713b6

Version of Immich Mobile App

N/A

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

---
env:
  DB_PASSWORD:
    valueFrom:
      secretKeyRef:
        name: immich-pg-secret
        key: password

# https://github.com/immich-app/immich/pkgs/container/immich-server
image:
  tag: v1.94.1@sha256:e0655cdd4351a8406eb87ec379a9c45e7bc2a9d051d25c93e2953d92e01713b6

immich:
  persistence:
    library:
      existingClaim: immich-storage-pvc

# https://hub.docker.com/r/tensorchord/pgvecto-rs/tags
postgresql:
  enabled: true
  image:
    # error happens with both versions
    #tag: pg16-v0.1.13@sha256:23c46f011028e141c2361ef9a5b40608df73e7d2773aaccfdb8e088430e8de1c
    tag: pg16-v0.2.0-alpha.2@sha256:329cd81686974608589595abe7baf9c9deb847ed20d17e1e183f11d272d117a2
  global:
    postgresql:
      auth:
        existingSecret: immich-pg-secret

redis:
  enabled: true

microservices:
  controller:
    replicas: 1
    strategy: RollingUpdate
  persistence:
    geodata-cache:
      enabled: true
      type: pvc
      storageClass: nfs-csi
      accessMode: ReadWriteMany
      size: 20Gi
  resources:
    requests:
      cpu: 1500m
      memory: 2048M

# https://github.com/immich-app/immich/pkgs/container/immich-machine-learning/
machine-learning:
  image:
    tag: v1.94.1@sha256:48a9f0056983703b1f50728b4e4dd4879f3bff307579910c00a4f62204f113dc
  controller:
    replicas: 1
    strategy: RollingUpdate
  resources:
    requests:
      cpu: 500m
      memory: 2048M
  persistence:
    cache:
      enabled: true
      type: pvc
      storageClass: nfs-csi
      accessMode: ReadWriteMany
      size: 20Gi
  probes:
    liveness:
      enabled: false
      spec:
        initialDelaySeconds: 120
        failureThreshold: 3
    readiness:
      enabled: false
      spec:
        initialDelaySeconds: 120
        failureThreshold: 3
    startup:
      enabled: false
      spec:
        initialDelaySeconds: 120
        failureThreshold: 3

server:
  probes:
    liveness:
      enabled: false
      spec:
        initialDelaySeconds: 120
        failureThreshold: 3
    readiness:
      enabled: false
      spec:
        initialDelaySeconds: 120
        failureThreshold: 3
    startup:
      enabled: false
      spec:
        initialDelaySeconds: 120
        failureThreshold: 3
  ingress:
    main:
      enabled: true
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod-rfc2136
        traefik.ingress.kubernetes.io/router.entrypoints: websecure
      hosts:
        - host: FOO
          paths:
            - path: /
      tls:
        - secretName: immich-tls
          hosts:
            - FOO

Your .env content

N/A

Reproduction steps

run pg16 update from 1.92.1 to 1.94.1

Additional information

I had a side issue with running pgvector 0.1.13 (updated from 0.1.11 iirc). that was solved by updating to 0.2.0 and running the upgrade instructions. The vectors_amhandler_wrapper error occurred both before and after.

Originally created by @prometheanfire on GitHub (Feb 1, 2024). ### The bug it looks like a driver is missing on the arm64 image? ``` Migration "SetAssetFaceNullOnPersonDelete1704943345360" failed, error: could not find function "vectors_amhandler_wrapper" in file "/usr/lib/postgresql/16/lib/vectors.so" /usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219 throw new QueryFailedError_1.QueryFailedError(query, parameters, err); ^ QueryFailedError: could not find function "vectors_amhandler_wrapper" in file "/usr/lib/postgresql/16/lib/vectors.so" at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async SetAssetFaceNullOnPersonDelete1704943345360.up (/usr/src/app/dist/infra/migrations/1704943345360-SetAssetFaceNullOnPersonDelete.js:6:9) at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17) at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:263:35) at async DatabaseRepository.runMigrations (/usr/src/app/dist/infra/repositories/database.repository.js:42:9) at async DatabaseService.init (/usr/src/app/dist/domain/database/database.service.js:33:9) at async AppService.init (/usr/src/app/dist/immich/app.service.js:56:9) at async AppModule.onModuleInit (/usr/src/app/dist/immich/app.module.js:32:9) at async callModuleInitHook (/usr/src/app/node_modules/@nestjs/core/hooks/on-module-init.hook.js:51:9) { query: '\n' + ' ALTER TABLE "asset_faces"\n' + ' DROP CONSTRAINT "FK_95ad7106dd7b484275443f580f9",\n' + ' ADD CONSTRAINT "FK_95ad7106dd7b484275443f580f9" \n' + ' FOREIGN KEY ("personId") REFERENCES "person"("id")\n' + ' ON DELETE SET NULL ON UPDATE CASCADE\n' + ' ', parameters: undefined, driverError: error: could not find function "vectors_amhandler_wrapper" in file "/usr/lib/postgresql/16/lib/vectors.so" at /usr/src/app/node_modules/pg/lib/client.js:526:17 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25) at async SetAssetFaceNullOnPersonDelete1704943345360.up (/usr/src/app/dist/infra/migrations/1704943345360-SetAssetFaceNullOnPersonDelete.js:6:9) at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17) at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:263:35) at async DatabaseRepository.runMigrations (/usr/src/app/dist/infra/repositories/database.repository.js:42:9) at async DatabaseService.init (/usr/src/app/dist/domain/database/database.service.js:33:9) at async AppService.init (/usr/src/app/dist/immich/app.service.js:56:9) at async AppModule.onModuleInit (/usr/src/app/dist/immich/app.module.js:32:9) { length: 386, severity: 'ERROR', code: '42883', detail: undefined, hint: undefined, position: undefined, internalPosition: undefined, internalQuery: undefined, where: 'SQL statement "SELECT fk."personId" FROM ONLY "public"."asset_faces" fk LEFT OUTER JOIN ONLY "public"."person" pk ON ( pk."id" OPERATOR(pg_catalog.=) fk."personId") WHERE pk."id" IS NULL AND (fk."personId" IS NOT NULL)"', schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'dfmgr.c', line: '127', routine: 'load_external_function' }, length: 386, severity: 'ERROR', code: '42883', detail: undefined, hint: undefined, position: undefined, internalPosition: undefined, internalQuery: undefined, where: 'SQL statement "SELECT fk."personId" FROM ONLY "public"."asset_faces" fk LEFT OUTER JOIN ONLY "public"."person" pk ON ( pk."id" OPERATOR(pg_catalog.=) fk."personId") WHERE pk."id" IS NULL AND (fk."personId" IS NOT NULL)"', schema: undefined, table: undefined, column: undefined, dataType: undefined, constraint: undefined, file: 'dfmgr.c', line: '127', routine: 'load_external_function' } Node.js v20.11.0 ``` ### The OS that Immich Server is running on kubernetes deployment ### Version of Immich Server v1.94.1@sha256:e0655cdd4351a8406eb87ec379a9c45e7bc2a9d051d25c93e2953d92e01713b6 ### Version of Immich Mobile App N/A ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML --- env: DB_PASSWORD: valueFrom: secretKeyRef: name: immich-pg-secret key: password # https://github.com/immich-app/immich/pkgs/container/immich-server image: tag: v1.94.1@sha256:e0655cdd4351a8406eb87ec379a9c45e7bc2a9d051d25c93e2953d92e01713b6 immich: persistence: library: existingClaim: immich-storage-pvc # https://hub.docker.com/r/tensorchord/pgvecto-rs/tags postgresql: enabled: true image: # error happens with both versions #tag: pg16-v0.1.13@sha256:23c46f011028e141c2361ef9a5b40608df73e7d2773aaccfdb8e088430e8de1c tag: pg16-v0.2.0-alpha.2@sha256:329cd81686974608589595abe7baf9c9deb847ed20d17e1e183f11d272d117a2 global: postgresql: auth: existingSecret: immich-pg-secret redis: enabled: true microservices: controller: replicas: 1 strategy: RollingUpdate persistence: geodata-cache: enabled: true type: pvc storageClass: nfs-csi accessMode: ReadWriteMany size: 20Gi resources: requests: cpu: 1500m memory: 2048M # https://github.com/immich-app/immich/pkgs/container/immich-machine-learning/ machine-learning: image: tag: v1.94.1@sha256:48a9f0056983703b1f50728b4e4dd4879f3bff307579910c00a4f62204f113dc controller: replicas: 1 strategy: RollingUpdate resources: requests: cpu: 500m memory: 2048M persistence: cache: enabled: true type: pvc storageClass: nfs-csi accessMode: ReadWriteMany size: 20Gi probes: liveness: enabled: false spec: initialDelaySeconds: 120 failureThreshold: 3 readiness: enabled: false spec: initialDelaySeconds: 120 failureThreshold: 3 startup: enabled: false spec: initialDelaySeconds: 120 failureThreshold: 3 server: probes: liveness: enabled: false spec: initialDelaySeconds: 120 failureThreshold: 3 readiness: enabled: false spec: initialDelaySeconds: 120 failureThreshold: 3 startup: enabled: false spec: initialDelaySeconds: 120 failureThreshold: 3 ingress: main: enabled: true annotations: cert-manager.io/cluster-issuer: letsencrypt-prod-rfc2136 traefik.ingress.kubernetes.io/router.entrypoints: websecure hosts: - host: FOO paths: - path: / tls: - secretName: immich-tls hosts: - FOO ``` ### Your .env content ```Shell N/A ``` ### Reproduction steps ```bash run pg16 update from 1.92.1 to 1.94.1 ``` ### Additional information I had a side issue with running pgvector 0.1.13 (updated from 0.1.11 iirc). that was solved by updating to 0.2.0 and running the upgrade instructions. The vectors_amhandler_wrapper error occurred both before and after.
Author
Owner

@prometheanfire commented on GitHub (Feb 1, 2024):

looks like pgvector support was constricted to 0.1.11, so I may be SOL

@prometheanfire commented on GitHub (Feb 1, 2024): looks like pgvector support was constricted to 0.1.11, so I may be SOL
Author
Owner

@mertalev commented on GitHub (Feb 1, 2024):

You might be able to fix it by running DROP EXTENSION vectors and rolling back to 0.1.11

@mertalev commented on GitHub (Feb 1, 2024): You might be able to fix it by running `DROP EXTENSION vectors` and rolling back to 0.1.11
Author
Owner

@prometheanfire commented on GitHub (Feb 1, 2024):

tried that, but 0.1.11 is having issues starting

[2024-02-01T02:45:53Z ERROR vectors::bgworker] Panickied. Info: PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Error("invalid type: sequence, expected a map", line: 1, column: 11)), location: Location { file: "src/utils/file_atomic.rs", line: 51, col: 52 }, can_unwind: true, force_no_backtrace: false }. Backtrace: disabled backtrace.

@prometheanfire commented on GitHub (Feb 1, 2024): tried that, but 0.1.11 is having issues starting ``` [2024-02-01T02:45:53Z ERROR vectors::bgworker] Panickied. Info: PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Error("invalid type: sequence, expected a map", line: 1, column: 11)), location: Location { file: "src/utils/file_atomic.rs", line: 51, col: 52 }, can_unwind: true, force_no_backtrace: false }. Backtrace: disabled backtrace. ```
Author
Owner

@prometheanfire commented on GitHub (Feb 1, 2024):

removing the vectors directory in the postgresql data directory after downgrade seemed to fix it

@prometheanfire commented on GitHub (Feb 1, 2024): removing the vectors directory in the postgresql data directory after downgrade seemed to fix it
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#2100