Issues with machine learning since 1.91.0 #1916

Closed
opened 2026-02-05 04:25:34 +03:00 by OVERLORD · 13 comments
Owner

Originally created by @GlenSatchwell on GitHub (Jan 1, 2024).

The bug

Hey! Absolutely loving Immich so far!

Upgrading to 1.91.0 and also the minor versions (1.91.4) everything started back up fine. I made sure my docker-compose was using the correct changes for "tensorchord/pgvecto-rs:pg14-v0.1.11".

Everything was fine until I noticed that faces were no longer showing newer photos and only existing matched photos.

There's also some weird search bugs happening. I know that the switch from typesense has caused some features to need rebuilding but when searching without metadata the search on both web/mobile it loads forever and eventually causes a 500.

The main bug is relating to the People/Faces, when logging the immich_postgres container, I can see an error below:

ERROR: column faces.embedding does not exist at character 370

Hoping someone can assist with what could have caused this?

Thanks! ❤️

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.91.4

Version of Immich Mobile App

v1.91.4 Build 116

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"
#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#
name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:80cc8518800438c684a53ed829c621c94afd1087aaeb59b0d4343ed3e7bcf6c5
    restart: always
  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
  backup:
    container_name: immich_db_dumper
    image: prodrigestivill/postgres-backup-local
    env_file:
      - .env
    environment:
      POSTGRES_HOST: database
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      SCHEDULE: "@daily"
      BACKUP_DIR: /db_dumps
    volumes: [HIDDEN FOR PRIVACY]
    depends_on:
      - database
    restart: always
volumes:
  pgdata:
  model-cache:

Your .env content

You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
# The location where your uploaded files are stored
UPLOAD_LOCATION=[HIDDEN FOR PRIVACY]
# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
# Connection secrets for postgres and typesense. You should change these to random passwords
DB_PASSWORD=
# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Login to web.
2. Visit Adminstration -> Job Status.
3. Rerun all Recognise Faces.

This will wipe all existing faces which I didn't mind since I can recreate them after. Since rerunning this, no faces are detected and the above error is shown in postgres logs.

Additional information

No response

Originally created by @GlenSatchwell on GitHub (Jan 1, 2024). ### The bug Hey! Absolutely loving Immich so far! Upgrading to 1.91.0 and also the minor versions (1.91.4) everything started back up fine. I made sure my docker-compose was using the correct changes for "tensorchord/pgvecto-rs:pg14-v0.1.11". Everything was fine until I noticed that faces were no longer showing newer photos and only existing matched photos. There's also some weird search bugs happening. I know that the switch from typesense has caused some features to need rebuilding but when searching without metadata the search on both web/mobile it loads forever and eventually causes a 500. The main bug is relating to the People/Faces, when logging the immich_postgres container, I can see an error below: `ERROR: column faces.embedding does not exist at character 370 ` Hoping someone can assist with what could have caused this? Thanks! ❤️ ### The OS that Immich Server is running on Ubuntu 22.04 ### Version of Immich Server v1.91.4 ### Version of Immich Mobile App v1.91.4 Build 116 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 depends_on: - redis - database restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:80cc8518800438c684a53ed829c621c94afd1087aaeb59b0d4343ed3e7bcf6c5 restart: always database: container_name: immich_postgres image: tensorchord/pgvecto-rs:pg14-v0.1.11 env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always backup: container_name: immich_db_dumper image: prodrigestivill/postgres-backup-local env_file: - .env environment: POSTGRES_HOST: database POSTGRES_DB: ${DB_DATABASE_NAME} POSTGRES_USER: ${DB_USERNAME} POSTGRES_PASSWORD: ${DB_PASSWORD} SCHEDULE: "@daily" BACKUP_DIR: /db_dumps volumes: [HIDDEN FOR PRIVACY] depends_on: - database restart: always volumes: pgdata: model-cache: ``` ### Your .env content ```Shell You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=[HIDDEN FOR PRIVACY] # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secrets for postgres and typesense. You should change these to random passwords DB_PASSWORD= # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. Login to web. 2. Visit Adminstration -> Job Status. 3. Rerun all Recognise Faces. This will wipe all existing faces which I didn't mind since I can recreate them after. Since rerunning this, no faces are detected and the above error is shown in postgres logs. ``` ### Additional information _No response_
Author
Owner

@alextran1502 commented on GitHub (Jan 1, 2024):

Hello, which facial recognition model are you running?

@alextran1502 commented on GitHub (Jan 1, 2024): Hello, which facial recognition model are you running?
Author
Owner

@GlenSatchwell commented on GitHub (Jan 1, 2024):

Hello, which facial recognition model are you running?

Hey Alex, Thanks for the quick response.

Server is currently using buffalo_l model which I believe is the default. I swapped this to another one yesterday to see if that made any difference but still the same.

@GlenSatchwell commented on GitHub (Jan 1, 2024): > Hello, which facial recognition model are you running? Hey Alex, Thanks for the quick response. Server is currently using buffalo_l model which I believe is the default. I swapped this to another one yesterday to see if that made any difference but still the same.
Author
Owner

@alextran1502 commented on GitHub (Jan 1, 2024):

Can you help swapping back the default and then capture the screenshot of the machine learning settings on your instance?

@alextran1502 commented on GitHub (Jan 1, 2024): Can you help swapping back the default and then capture the screenshot of the machine learning settings on your instance?
Author
Owner

@GlenSatchwell commented on GitHub (Jan 1, 2024):

No problem, see below my current machine learning settings:

{
  "machineLearning": {
    "enabled": true,
    "url": "http://immich-machine-learning:3003",
    "classification": {
      "enabled": true,
      "modelName": "microsoft/resnet-50",
      "minScore": 0.9
    },
    "clip": {
      "enabled": true,
      "modelName": "ViT-B-32__openai"
    },
    "facialRecognition": {
      "enabled": true,
      "modelName": "buffalo_l",
      "minScore": 0.7,
      "maxDistance": 0.6,
      "minFaces": 2
    }
  }
}
@GlenSatchwell commented on GitHub (Jan 1, 2024): No problem, see below my current machine learning settings: ``` { "machineLearning": { "enabled": true, "url": "http://immich-machine-learning:3003", "classification": { "enabled": true, "modelName": "microsoft/resnet-50", "minScore": 0.9 }, "clip": { "enabled": true, "modelName": "ViT-B-32__openai" }, "facialRecognition": { "enabled": true, "modelName": "buffalo_l", "minScore": 0.7, "maxDistance": 0.6, "minFaces": 2 } } } ```
Author
Owner

@univ95 commented on GitHub (Jan 3, 2024):

Can you check if the column name embedding is misspelled by any chance?

@univ95 commented on GitHub (Jan 3, 2024): Can you check if the column name embedding is misspelled by any chance?
Author
Owner

@GlenSatchwell commented on GitHub (Jan 3, 2024):

See below the current list of tables for the Immich db, does this all look as expected?

Screenshot_2024-01-03-13-22-50-90_61c78dc80ee02b53007c815fefe993e3.jpg

@GlenSatchwell commented on GitHub (Jan 3, 2024): See below the current list of tables for the Immich db, does this all look as expected? ![Screenshot_2024-01-03-13-22-50-90_61c78dc80ee02b53007c815fefe993e3.jpg](https://github.com/immich-app/immich/assets/20623090/257a36b9-710a-4f47-8c89-d2be5a29dbac)
Author
Owner

@GlenSatchwell commented on GitHub (Jan 8, 2024):

To update on this issue, since upgrading to 1.92 the error has slightly changed. See full error message below from immich_server log

[Nest] 6  - 01/08/2024, 7:14:00 PM   ERROR [QueryFailedError: column s.embedding does not exist
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.
js:211:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async SelectQueryBuilder.loadRawResults (/usr/src/app/node_modules/typeorm/query-builder/SelectQu
eryBuilder.js:2183:25)
    at async SelectQueryBuilder.executeEntitiesAndRawResults (/usr/src/app/node_modules/typeorm/query-bu
ilder/SelectQueryBuilder.js:2034:26)
    at async SelectQueryBuilder.getRawAndEntities (/usr/src/app/node_modules/typeorm/query-builder/Selec
tQueryBuilder.js:684:29)
    at async SelectQueryBuilder.getMany (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuil
der.js:750:25)
    at async /usr/src/app/dist/infra/repositories/smart-info.repository.js:57:23
    at async EntityManager.transaction (/usr/src/app/node_modules/typeorm/entity-manager/EntityManager.j
s:73:28)
    at async SmartInfoRepository.searchCLIP (/usr/src/app/dist/infra/repositories/smart-info.repository.
js:54:9)
    at async SearchService.search (/usr/src/app/dist/domain/search/search.service.js:66:26)] Failed to s
earch
@GlenSatchwell commented on GitHub (Jan 8, 2024): To update on this issue, since upgrading to 1.92 the error has slightly changed. See full error message below from immich_server log ``` [Nest] 6 - 01/08/2024, 7:14:00 PM ERROR [QueryFailedError: column s.embedding does not exist at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner. js:211:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async SelectQueryBuilder.loadRawResults (/usr/src/app/node_modules/typeorm/query-builder/SelectQu eryBuilder.js:2183:25) at async SelectQueryBuilder.executeEntitiesAndRawResults (/usr/src/app/node_modules/typeorm/query-bu ilder/SelectQueryBuilder.js:2034:26) at async SelectQueryBuilder.getRawAndEntities (/usr/src/app/node_modules/typeorm/query-builder/Selec tQueryBuilder.js:684:29) at async SelectQueryBuilder.getMany (/usr/src/app/node_modules/typeorm/query-builder/SelectQueryBuil der.js:750:25) at async /usr/src/app/dist/infra/repositories/smart-info.repository.js:57:23 at async EntityManager.transaction (/usr/src/app/node_modules/typeorm/entity-manager/EntityManager.j s:73:28) at async SmartInfoRepository.searchCLIP (/usr/src/app/dist/infra/repositories/smart-info.repository. js:54:9) at async SearchService.search (/usr/src/app/dist/domain/search/search.service.js:66:26)] Failed to s earch ```
Author
Owner

@vktr2b commented on GitHub (Jan 9, 2024):

I'm not exactly sure how much this will help but I've managed to by accident reproduce this exact issue, I've had an unrelated issue with my DB and someone in another thread suggested

DROP EXTENSION vectors CASCADE;

and then adding it again with

CREATE EXTENSION vectors;

which did not solve my original issue which as it turned was unrelated but after these 2 commands I got the exact same error as you

I first tried the command without the cascade option but it seems that it had some table column dependencies and with cascade those embeddings columns got removed from the asset_faces and smart_search table

immich=# DROP EXTENSION vectors;
ERROR:  cannot drop extension vectors because other objects depend on it
DETAIL:  column embedding of table asset_faces depends on type vector
column embedding of table smart_search depends on type vector
HINT:  Use DROP ... CASCADE to drop the dependent objects too.

Have you by any chance dropped the extension and re-added it again?
Can you check if you have the embeding column in asset_faces and smart_search table ?
Unfortunately I cannot provide a solution as I solved it by restoring my backup of the DB from yesterday but hopefully this helps point to the right direction :)

@vktr2b commented on GitHub (Jan 9, 2024): I'm not exactly sure how much this will help but I've managed to by accident reproduce this exact issue, I've had an unrelated issue with my DB and someone in another thread suggested ```sql DROP EXTENSION vectors CASCADE; ``` and then adding it again with ```sql CREATE EXTENSION vectors; ``` which did not solve my original issue which as it turned was unrelated but after these 2 commands I got the exact same error as you I first tried the command without the cascade option but it seems that it had some table column dependencies and with cascade those embeddings columns got removed from the `asset_faces` and `smart_search` table ```sql immich=# DROP EXTENSION vectors; ERROR: cannot drop extension vectors because other objects depend on it DETAIL: column embedding of table asset_faces depends on type vector column embedding of table smart_search depends on type vector HINT: Use DROP ... CASCADE to drop the dependent objects too. ``` Have you by any chance dropped the extension and re-added it again? Can you check if you have the embeding column in `asset_faces` and `smart_search` table ? Unfortunately I cannot provide a solution as I solved it by restoring my backup of the DB from yesterday but hopefully this helps point to the right direction :)
Author
Owner

@GlenSatchwell commented on GitHub (Jan 9, 2024):

@vktr2b Thanks for the advice, I also came across this same fix for a different issue so I had already dropped and recreated the vectors extension before so this maybe the reason why columns are missing 🤔

I've just checked the table for both asset_faces and smart_search, which confirms the embedding column is missing. The smart_search table only includes a assetId column. See below:

Table "public.asset_faces"
    Column     |  Type   | Collation | Nullable |      Default
---------------+---------+-----------+----------+--------------------
 assetId       | uuid    |           | not null |
 personId      | uuid    |           |          |
 imageWidth    | integer |           | not null | 0
 imageHeight   | integer |           | not null | 0
 boundingBoxX1 | integer |           | not null | 0
 boundingBoxY1 | integer |           | not null | 0
 boundingBoxX2 | integer |           | not null | 0
 boundingBoxY2 | integer |           | not null | 0
 id            | uuid    |           | not null | uuid_generate_v4()
Table "public.smart_search"
 Column  | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
---------+------+-----------+----------+---------+---------+-------------+--------------+-------------
 assetId | uuid |           | not null |         | plain   |             |
 |
@GlenSatchwell commented on GitHub (Jan 9, 2024): @vktr2b Thanks for the advice, I also came across this same fix for a different issue so I had already dropped and recreated the vectors extension before so this maybe the reason why columns are missing 🤔 I've just checked the table for both asset_faces and smart_search, which confirms the embedding column is missing. The smart_search table only includes a assetId column. See below: ``` Table "public.asset_faces" Column | Type | Collation | Nullable | Default ---------------+---------+-----------+----------+-------------------- assetId | uuid | | not null | personId | uuid | | | imageWidth | integer | | not null | 0 imageHeight | integer | | not null | 0 boundingBoxX1 | integer | | not null | 0 boundingBoxY1 | integer | | not null | 0 boundingBoxX2 | integer | | not null | 0 boundingBoxY2 | integer | | not null | 0 id | uuid | | not null | uuid_generate_v4() ``` ``` Table "public.smart_search" Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description ---------+------+-----------+----------+---------+---------+-------------+--------------+------------- assetId | uuid | | not null | | plain | | | ```
Author
Owner

@mertalev commented on GitHub (Jan 17, 2024):

If you dropped the embedding columns, you'll need to recreate them and their indices to use smart search or facial recognition. You can do this by running these commands:

START TRANSACTION;

ALTER TABLE smart_search ADD COLUMN embedding vector(512) NOT NULL;

ALTER TABLE asset_faces ADD COLUMN embedding vector(512) NOT NULL;

CREATE INDEX IF NOT EXISTS clip_index ON smart_search
      USING vectors (embedding cosine_ops) WITH (options = $$
      [indexing.hnsw]
      m = 16
      ef_construction = 300
      $$);

CREATE INDEX IF NOT EXISTS face_index ON asset_faces
      USING vectors (embedding cosine_ops) WITH (options = $$
      [indexing.hnsw]
      m = 16
      ef_construction = 300
      $$);

COMMIT;
@mertalev commented on GitHub (Jan 17, 2024): If you dropped the embedding columns, you'll need to recreate them and their indices to use smart search or facial recognition. You can do this by running these commands: ``` START TRANSACTION; ALTER TABLE smart_search ADD COLUMN embedding vector(512) NOT NULL; ALTER TABLE asset_faces ADD COLUMN embedding vector(512) NOT NULL; CREATE INDEX IF NOT EXISTS clip_index ON smart_search USING vectors (embedding cosine_ops) WITH (options = $$ [indexing.hnsw] m = 16 ef_construction = 300 $$); CREATE INDEX IF NOT EXISTS face_index ON asset_faces USING vectors (embedding cosine_ops) WITH (options = $$ [indexing.hnsw] m = 16 ef_construction = 300 $$); COMMIT; ```
Author
Owner

@JshKlsn commented on GitHub (Feb 4, 2024):

If you dropped the embedding columns, you'll need to recreate them and their indices to use smart search or facial recognition. You can do this by running these commands:

START TRANSACTION;

ALTER TABLE smart_search ADD COLUMN embedding vector(512) NOT NULL;

ALTER TABLE asset_faces ADD COLUMN embedding vector(512) NOT NULL;

CREATE INDEX IF NOT EXISTS clip_index ON smart_search
      USING vectors (embedding cosine_ops) WITH (options = $$
      [indexing.hnsw]
      m = 16
      ef_construction = 300
      $$);

CREATE INDEX IF NOT EXISTS face_index ON asset_faces
      USING vectors (embedding cosine_ops) WITH (options = $$
      [indexing.hnsw]
      m = 16
      ef_construction = 300
      $$);

COMMIT;

I get the following errors;

Error in query: ERROR: column "embedding" of relation "smart_search" contains null values

Error in query: ERROR: current transaction is aborted, commands ignored until end of transaction block

Error in query: ERROR: current transaction is aborted, commands ignored until end of transaction block

Error in query: ERROR: current transaction is aborted, commands ignored until end of transaction block

Error in query: 2 3 4 5

Any idea why? Thanks!

@JshKlsn commented on GitHub (Feb 4, 2024): > If you dropped the embedding columns, you'll need to recreate them and their indices to use smart search or facial recognition. You can do this by running these commands: > > ``` > START TRANSACTION; > > ALTER TABLE smart_search ADD COLUMN embedding vector(512) NOT NULL; > > ALTER TABLE asset_faces ADD COLUMN embedding vector(512) NOT NULL; > > CREATE INDEX IF NOT EXISTS clip_index ON smart_search > USING vectors (embedding cosine_ops) WITH (options = $$ > [indexing.hnsw] > m = 16 > ef_construction = 300 > $$); > > CREATE INDEX IF NOT EXISTS face_index ON asset_faces > USING vectors (embedding cosine_ops) WITH (options = $$ > [indexing.hnsw] > m = 16 > ef_construction = 300 > $$); > > COMMIT; > ``` I get the following errors; `Error in query: ERROR: column "embedding" of relation "smart_search" contains null values ` `Error in query: ERROR: current transaction is aborted, commands ignored until end of transaction block` `Error in query: ERROR: current transaction is aborted, commands ignored until end of transaction block ` `Error in query: ERROR: current transaction is aborted, commands ignored until end of transaction block` `Error in query: 2 3 4 5` Any idea why? Thanks!
Author
Owner

@mertalev commented on GitHub (Feb 4, 2024):

Before running ALTER TABLE smart_search ADD COLUMN embedding vector(512) NOT NULL;, run this command: DELETE FROM smart_search WHERE embedding IS NULL;. This is a bizarre error, though.

@mertalev commented on GitHub (Feb 4, 2024): Before running `ALTER TABLE smart_search ADD COLUMN embedding vector(512) NOT NULL;`, run this command: `DELETE FROM smart_search WHERE embedding IS NULL;`. This is a bizarre error, though.
Author
Owner

@JshKlsn commented on GitHub (Feb 4, 2024):

I get the new following error;

Error in query: ERROR: column "embedding" does not exist LINE 1: DELETE FROM smart_search WHERE embedding IS NULL

I think Immich hates me haha.

@JshKlsn commented on GitHub (Feb 4, 2024): I get the new following error; `Error in query: ERROR: column "embedding" does not exist LINE 1: DELETE FROM smart_search WHERE embedding IS NULL` I think Immich hates me haha.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1916