immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 00:30:57 +03:00
Code Issues 11 Packages Projects Releases 10 Wiki Activity

[BUG] After the face recognition task is completely re-run, every face is a single photo? #1596

New Issue
Closed
opened 2026-02-05 02:28:35 +03:00 by OVERLORD · 14 comments
OVERLORD commented 2026-02-05 02:28:35 +03:00
Owner
Copy Link

Originally created by @davidpan on GitHub (Nov 9, 2023).

The bug

It looks like there is no merging of similar/same person faces and the face recognition parameters are using default values.

There are about 20,000 photos and a few hundred videos on the immich.

Is there any way to fix this?

The OS that Immich Server is running on

Ubuntu 22.04.3 LTS

Version of Immich Server

v1.85.0

Version of Immich Mobile App

v1.85.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /media/1T/PhotoPrism:/mnt/media/PhotoPrism:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /media/1T/PhotoPrism:/mnt/media/PhotoPrism:ro
    devices:
      - /dev/dri/renderD128:/dev/dri/renderD128
      - /dev/dri/card0:/dev/dri/card0
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always
    dns:
      - 1.1.1.1

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.25.1
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      #- GLOG_minloglevel=1
    volumes:
      - tsdata1:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata1:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/media/immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=*
DB_PASSWORD=*

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. rebuild all hosts, delete tsdata volume
2. re-run the face recognition task completely
3. check the photos in the PEOPLE interface after the task is complete
...

Additional information

immich_typesense logs:

I20231109 11:16:35.191955   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20231109 11:16:39.852726   166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939
I20231109 11:16:39.852813   210 raft_server.h:60] Peer refresh succeeded!
I20231109 11:16:49.853705   166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939
I20231109 11:16:49.853812   204 raft_server.h:60] Peer refresh succeeded!
I20231109 11:16:59.854790   166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939
I20231109 11:16:59.854955   210 raft_server.h:60] Peer refresh succeeded!
I20231109 11:17:09.855840   166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939
I20231109 11:17:09.855958   204 raft_server.h:60] Peer refresh succeeded!
I20231109 11:17:19.856822   166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939
I20231109 11:17:19.856940   210 raft_server.h:60] Peer refresh succeeded!
I20231109 11:17:29.857842   166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939
I20231109 11:17:29.857949   204 raft_server.h:60] Peer refresh succeeded!
I20231109 11:17:36.197216   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20231109 11:17:39.859115   166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939
I20231109 11:17:39.859218   210 raft_server.h:60] Peer refresh succeeded!
Originally created by @davidpan on GitHub (Nov 9, 2023). ### The bug It looks like there is no merging of similar/same person faces and the face recognition parameters are using default values. There are about 20,000 photos and a few hundred videos on the immich. Is there any way to fix this? ### The OS that Immich Server is running on Ubuntu 22.04.3 LTS ### Version of Immich Server v1.85.0 ### Version of Immich Mobile App v1.85.0 ### Platform with the issue - [X] Server - [X] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML version: "3.8" services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: ["start.sh", "immich"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - /media/1T/PhotoPrism:/mnt/media/PhotoPrism:ro env_file: - .env depends_on: - redis - database - typesense restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: ["start.sh", "microservices"] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - /media/1T/PhotoPrism:/mnt/media/PhotoPrism:ro devices: - /dev/dri/renderD128:/dev/dri/renderD128 - /dev/dri/card0:/dev/dri/card0 env_file: - .env depends_on: - redis - database - typesense restart: always dns: - 1.1.1.1 immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env restart: always immich-web: container_name: immich_web image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release} env_file: - .env restart: always typesense: container_name: immich_typesense image: typesense/typesense:0.25.1 environment: - TYPESENSE_API_KEY=${TYPESENSE_API_KEY} - TYPESENSE_DATA_DIR=/data # remove this to get debug messages #- GLOG_minloglevel=1 volumes: - tsdata1:/data restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3 restart: always database: container_name: immich_postgres image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441 env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always immich-proxy: container_name: immich_proxy image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release} environment: # Make sure these values get passed through from the env file - IMMICH_SERVER_URL - IMMICH_WEB_URL ports: - 2283:8080 depends_on: - immich-server - immich-web restart: always volumes: pgdata: model-cache: tsdata1: ``` ### Your .env content ```Shell # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables # The location where your uploaded files are stored UPLOAD_LOCATION=/media/immich # The Immich version to use. You can pin this to a specific version like "v1.71.0" IMMICH_VERSION=release # Connection secrets for postgres and typesense. You should change these to random passwords TYPESENSE_API_KEY=* DB_PASSWORD=* # The values below this line do not need to be changed ################################################################################### DB_HOSTNAME=immich_postgres DB_USERNAME=postgres DB_DATABASE_NAME=immich REDIS_HOSTNAME=immich_redis ``` ### Reproduction steps ```bash 1. rebuild all hosts, delete tsdata volume 2. re-run the face recognition task completely 3. check the photos in the PEOPLE interface after the task is complete ... ``` ### Additional information immich_typesense logs: ``` I20231109 11:16:35.191955 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20231109 11:16:39.852726 166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939 I20231109 11:16:39.852813 210 raft_server.h:60] Peer refresh succeeded! I20231109 11:16:49.853705 166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939 I20231109 11:16:49.853812 204 raft_server.h:60] Peer refresh succeeded! I20231109 11:16:59.854790 166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939 I20231109 11:16:59.854955 210 raft_server.h:60] Peer refresh succeeded! I20231109 11:17:09.855840 166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939 I20231109 11:17:09.855958 204 raft_server.h:60] Peer refresh succeeded! I20231109 11:17:19.856822 166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939 I20231109 11:17:19.856940 210 raft_server.h:60] Peer refresh succeeded! I20231109 11:17:29.857842 166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939 I20231109 11:17:29.857949 204 raft_server.h:60] Peer refresh succeeded! I20231109 11:17:36.197216 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20231109 11:17:39.859115 166 raft_server.cpp:564] Term: 2, last_index index: 1510, committed_index: 1510, known_applied_index: 1510, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2939 I20231109 11:17:39.859218 210 raft_server.h:60] Peer refresh succeeded! ```
OVERLORD commented 2026-02-05 02:28:41 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Nov 9, 2023):

Can you explain every face is a single photo?

@alextran1502 commented on GitHub (Nov 9, 2023): Can you explain every face is a single photo?
OVERLORD commented 2026-02-05 02:28:43 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 10, 2023):

Can you explain every face is a single photo?

@alextran1502
Enter the people menu item, there are a bunch of face recognition results on the right side, and multiple face recognition results for the same person are visible in this interface. If you click on any face, there is only one photo of that face in this sub-interface.

Clicked on quite a few of them and they all came up with this.

@davidpan commented on GitHub (Nov 10, 2023): > Can you explain every face is a single photo? @alextran1502 Enter the people menu item, there are a bunch of face recognition results on the right side, and multiple face recognition results for the same person are visible in this interface. If you click on any face, there is only one photo of that face in this sub-interface. Clicked on quite a few of them and they all came up with this.
OVERLORD commented 2026-02-05 02:28:47 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Nov 10, 2023):

Did you modify the parameter of the facial recognition machine learning job?

@alextran1502 commented on GitHub (Nov 10, 2023): Did you modify the parameter of the facial recognition machine learning job?
OVERLORD commented 2026-02-05 02:28:54 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 10, 2023):

Did you modify the parameter of the facial recognition machine learning job?

The default parameters are used.

{
  "ffmpeg": {
    "crf": 23,
    "threads": 0,
    "preset": "ultrafast",
    "targetVideoCodec": "h264",
    "targetAudioCodec": "aac",
    "targetResolution": "720",
    "maxBitrate": "0",
    "bframes": -1,
    "refs": 0,
    "gopSize": 0,
    "npl": 0,
    "temporalAQ": false,
    "cqMode": "auto",
    "twoPass": false,
    "transcode": "disabled",
    "tonemap": "hable",
    "accel": "disabled"
  },
  "job": {
    "backgroundTask": {
      "concurrency": 5
    },
    "clipEncoding": {
      "concurrency": 2
    },
    "metadataExtraction": {
      "concurrency": 3
    },
    "objectTagging": {
      "concurrency": 1
    },
    "recognizeFaces": {
      "concurrency": 1
    },
    "search": {
      "concurrency": 5
    },
    "sidecar": {
      "concurrency": 3
    },
    "library": {
      "concurrency": 3
    },
    "storageTemplateMigration": {
      "concurrency": 5
    },
    "migration": {
      "concurrency": 3
    },
    "thumbnailGeneration": {
      "concurrency": 3
    },
    "videoConversion": {
      "concurrency": 1
    }
  },
  "machineLearning": {
    "enabled": true,
    "url": "http://immich-machine-learning:3003",
    "classification": {
      "enabled": true,
      "modelName": "microsoft/resnet-50",
      "minScore": 0.9
    },
    "clip": {
      "enabled": true,
      "modelName": "ViT-B-32__openai"
    },
    "facialRecognition": {
      "enabled": true,
      "modelName": "buffalo_l",
      "minScore": 0.7,
      "maxDistance": 0.6,
      "minFaces": 1
    }
  },
  "map": {
    "enabled": true,
    "tileUrl": "https://tile.openstreetmap.org/{z}/{x}/{y}.png"
  },
  "reverseGeocoding": {
    "enabled": true,
    "citiesFileOverride": "cities500"
  },
  "oauth": {
    "enabled": false,
    "issuerUrl": "",
    "clientId": "",
    "clientSecret": "",
    "mobileOverrideEnabled": false,
    "mobileRedirectUri": "",
    "scope": "openid email profile",
    "storageLabelClaim": "preferred_username",
    "buttonText": "Login with OAuth",
    "autoRegister": true,
    "autoLaunch": false
  },
  "passwordLogin": {
    "enabled": true
  },
  "storageTemplate": {
    "template": "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}"
  },
  "thumbnail": {
    "webpSize": 250,
    "jpegSize": 1440,
    "quality": 80,
    "colorspace": "p3"
  },
  "newVersionCheck": {
    "enabled": true
  },
  "trash": {
    "enabled": true,
    "days": 30
  },
  "theme": {
    "customCss": ""
  },
  "library": {
    "scan": {
      "enabled": true,
      "cronExpression": "0 2 * * *"
    }
  }
}
@davidpan commented on GitHub (Nov 10, 2023): > Did you modify the parameter of the facial recognition machine learning job? The default parameters are used. ```json { "ffmpeg": { "crf": 23, "threads": 0, "preset": "ultrafast", "targetVideoCodec": "h264", "targetAudioCodec": "aac", "targetResolution": "720", "maxBitrate": "0", "bframes": -1, "refs": 0, "gopSize": 0, "npl": 0, "temporalAQ": false, "cqMode": "auto", "twoPass": false, "transcode": "disabled", "tonemap": "hable", "accel": "disabled" }, "job": { "backgroundTask": { "concurrency": 5 }, "clipEncoding": { "concurrency": 2 }, "metadataExtraction": { "concurrency": 3 }, "objectTagging": { "concurrency": 1 }, "recognizeFaces": { "concurrency": 1 }, "search": { "concurrency": 5 }, "sidecar": { "concurrency": 3 }, "library": { "concurrency": 3 }, "storageTemplateMigration": { "concurrency": 5 }, "migration": { "concurrency": 3 }, "thumbnailGeneration": { "concurrency": 3 }, "videoConversion": { "concurrency": 1 } }, "machineLearning": { "enabled": true, "url": "http://immich-machine-learning:3003", "classification": { "enabled": true, "modelName": "microsoft/resnet-50", "minScore": 0.9 }, "clip": { "enabled": true, "modelName": "ViT-B-32__openai" }, "facialRecognition": { "enabled": true, "modelName": "buffalo_l", "minScore": 0.7, "maxDistance": 0.6, "minFaces": 1 } }, "map": { "enabled": true, "tileUrl": "https://tile.openstreetmap.org/{z}/{x}/{y}.png" }, "reverseGeocoding": { "enabled": true, "citiesFileOverride": "cities500" }, "oauth": { "enabled": false, "issuerUrl": "", "clientId": "", "clientSecret": "", "mobileOverrideEnabled": false, "mobileRedirectUri": "", "scope": "openid email profile", "storageLabelClaim": "preferred_username", "buttonText": "Login with OAuth", "autoRegister": true, "autoLaunch": false }, "passwordLogin": { "enabled": true }, "storageTemplate": { "template": "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}" }, "thumbnail": { "webpSize": 250, "jpegSize": 1440, "quality": 80, "colorspace": "p3" }, "newVersionCheck": { "enabled": true }, "trash": { "enabled": true, "days": 30 }, "theme": { "customCss": "" }, "library": { "scan": { "enabled": true, "cronExpression": "0 2 * * *" } } } ```
OVERLORD commented 2026-02-05 02:28:57 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 10, 2023):

Not sure if this helps troubleshoot the issue.

pgsql query.

immich=# select count(*) from person;
 count
-------
 21836
(1 row)

immich=# select count(*) from asset_faces ;
-------
 21836
(1 row)
@davidpan commented on GitHub (Nov 10, 2023): Not sure if this helps troubleshoot the issue. pgsql query. ```sql immich=# select count(*) from person; count ------- 21836 (1 row) immich=# select count(*) from asset_faces ; ------- 21836 (1 row) ```
OVERLORD commented 2026-02-05 02:29:02 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Nov 10, 2023):

Can you take the screenshot of facial recognition settings as well as your typesense log

@alextran1502 commented on GitHub (Nov 10, 2023): Can you take the screenshot of facial recognition settings as well as your typesense log
OVERLORD commented 2026-02-05 02:29:06 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 10, 2023):

Can you take the screenshot of facial recognition settings as well as your typesense log

image The json file for the configuration, I posted in my reply earlier.

typesense log:

I20231109 23:16:26.262069   210 raft_server.h:60] Peer refresh succeeded!
I20231109 23:16:28.850704   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20231109 23:16:36.263206   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:16:36.263381   204 raft_server.h:60] Peer refresh succeeded!
I20231109 23:16:46.264508   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:16:46.264744   210 raft_server.h:60] Peer refresh succeeded!
I20231109 23:16:56.265906   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:16:56.266078   204 raft_server.h:60] Peer refresh succeeded!
I20231109 23:17:06.267251   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:17:06.267427   210 raft_server.h:60] Peer refresh succeeded!
I20231109 23:17:16.268637   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:17:16.268848   204 raft_server.h:60] Peer refresh succeeded!
I20231109 23:17:26.270015   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:17:26.270227   210 raft_server.h:60] Peer refresh succeeded!
I20231109 23:17:29.859565   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20231109 23:17:36.271350   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:17:36.271524   204 raft_server.h:60] Peer refresh succeeded!
I20231109 23:17:46.272728   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:17:46.272954   210 raft_server.h:60] Peer refresh succeeded!
I20231109 23:17:56.274204   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:17:56.274381   204 raft_server.h:60] Peer refresh succeeded!
I20231109 23:18:06.275574   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:18:06.275732   210 raft_server.h:60] Peer refresh succeeded!
I20231109 23:18:16.276924   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:18:16.277107   204 raft_server.h:60] Peer refresh succeeded!
I20231109 23:18:26.278317   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:18:26.278514   210 raft_server.h:60] Peer refresh succeeded!
I20231109 23:18:30.868346   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20231109 23:18:36.279769   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:18:36.279942   204 raft_server.h:60] Peer refresh succeeded!
I20231109 23:18:46.281116   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:18:46.281275   210 raft_server.h:60] Peer refresh succeeded!
I20231109 23:18:56.282516   166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975
I20231109 23:18:56.282683   166 raft_server.cpp:910] Snapshot timer is active, current_ts: 1699571936, last_snapshot_ts: 1699568335
I20231109 23:18:56.282759   204 raft_server.h:60] Peer refresh succeeded!

I just restarted immich_microservices vm and below is the log after the restart.

typesense logs:

I20231110 00:15:27.352088   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20231110 00:15:36.749202   166 raft_server.cpp:564] Term: 2, last_index index: 1523, committed_index: 1523, known_applied_index: 1523, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2978
I20231110 00:15:36.749308   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:15:46.750113   166 raft_server.cpp:564] Term: 2, last_index index: 1523, committed_index: 1523, known_applied_index: 1523, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2978
I20231110 00:15:46.750206   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:15:56.750955   166 raft_server.cpp:564] Term: 2, last_index index: 1526, committed_index: 1526, known_applied_index: 1526, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3060
I20231110 00:15:56.751076   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:16:06.751930   166 raft_server.cpp:564] Term: 2, last_index index: 1526, committed_index: 1526, known_applied_index: 1526, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3060
I20231110 00:16:06.752126   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:16:08.570422   204 log.cpp:536] close a full segment. Current first_index: 1441 last_index: 1528 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001441_00000000000000001528
I20231110 00:16:08.570698   204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001441' to `/data/state/log/log_00000000000000001441_00000000000000001528'
I20231110 00:16:08.590584   204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001529' with fd=26
I20231110 00:16:14.870448   210 log.cpp:536] close a full segment. Current first_index: 1529 last_index: 1534 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001529_00000000000000001534
I20231110 00:16:14.870646   210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001529' to `/data/state/log/log_00000000000000001529_00000000000000001534'
I20231110 00:16:14.870937   210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001535' with fd=24
I20231110 00:16:16.756269   166 raft_server.cpp:564] Term: 2, last_index index: 1537, committed_index: 1537, known_applied_index: 1537, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 12074
I20231110 00:16:16.756393   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:16:18.201920   210 log.cpp:536] close a full segment. Current first_index: 1535 last_index: 1539 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001535_00000000000000001539
I20231110 00:16:18.202150   210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001535' to `/data/state/log/log_00000000000000001535_00000000000000001539'
I20231110 00:16:18.202281   210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001540' with fd=32
I20231110 00:16:21.146152   204 log.cpp:536] close a full segment. Current first_index: 1540 last_index: 1544 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001540_00000000000000001544
I20231110 00:16:21.146365   204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001540' to `/data/state/log/log_00000000000000001540_00000000000000001544'
I20231110 00:16:21.146546   204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001545' with fd=28
I20231110 00:16:26.467864   204 log.cpp:536] close a full segment. Current first_index: 1545 last_index: 1550 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001545_00000000000000001550
I20231110 00:16:26.468016   204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001545' to `/data/state/log/log_00000000000000001545_00000000000000001550'
I20231110 00:16:26.468088   204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001551' with fd=23
I20231110 00:16:26.757413   166 raft_server.cpp:564] Term: 2, last_index index: 1552, committed_index: 1551, known_applied_index: 1551, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 24092
I20231110 00:16:26.757577   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:16:28.357731   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 1
I20231110 00:16:30.246258   210 log.cpp:536] close a full segment. Current first_index: 1551 last_index: 1556 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001551_00000000000000001556
I20231110 00:16:30.246462   210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001551' to `/data/state/log/log_00000000000000001551_00000000000000001556'
I20231110 00:16:30.246601   210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001557' with fd=35
I20231110 00:16:33.785753   204 log.cpp:536] close a full segment. Current first_index: 1557 last_index: 1562 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001557_00000000000000001562
I20231110 00:16:33.785879   204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001557' to `/data/state/log/log_00000000000000001557_00000000000000001562'
I20231110 00:16:33.785961   204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001563' with fd=33
I20231110 00:16:36.758733   166 raft_server.cpp:564] Term: 2, last_index index: 1567, committed_index: 1567, known_applied_index: 1567, applying_index: 0, queued_writes: 4, pending_queue_size: 0, local_sequence: 36112
I20231110 00:16:36.758976   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:16:38.990906   210 log.cpp:536] close a full segment. Current first_index: 1563 last_index: 1569 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001563_00000000000000001569
I20231110 00:16:38.991134   210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001563' to `/data/state/log/log_00000000000000001563_00000000000000001569'
I20231110 00:16:38.991261   210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001570' with fd=25
I20231110 00:16:44.340844   204 log.cpp:536] close a full segment. Current first_index: 1570 last_index: 1575 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001570_00000000000000001575
I20231110 00:16:44.341044   204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001570' to `/data/state/log/log_00000000000000001570_00000000000000001575'
I20231110 00:16:44.341136   204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001576' with fd=29
I20231110 00:16:46.760064   166 raft_server.cpp:564] Term: 2, last_index index: 1579, committed_index: 1579, known_applied_index: 1579, applying_index: 0, queued_writes: 1, pending_queue_size: 0, local_sequence: 47635
I20231110 00:16:46.760186   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:16:47.557982   204 log.cpp:536] close a full segment. Current first_index: 1576 last_index: 1581 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001576_00000000000000001581
I20231110 00:16:47.558235   204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001576' to `/data/state/log/log_00000000000000001576_00000000000000001581'
I20231110 00:16:47.558512   204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001582' with fd=44
I20231110 00:16:51.910176   210 log.cpp:536] close a full segment. Current first_index: 1582 last_index: 1586 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001582_00000000000000001586
I20231110 00:16:51.910302   210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001582' to `/data/state/log/log_00000000000000001582_00000000000000001586'
I20231110 00:16:51.910399   210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001587' with fd=30
I20231110 00:16:55.258899   210 log.cpp:536] close a full segment. Current first_index: 1587 last_index: 1591 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001587_00000000000000001591
I20231110 00:16:55.259092   210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001587' to `/data/state/log/log_00000000000000001587_00000000000000001591'
I20231110 00:16:55.259263   210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001592' with fd=40
I20231110 00:16:56.761041   166 raft_server.cpp:564] Term: 2, last_index index: 1593, committed_index: 1592, known_applied_index: 1592, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 60145
I20231110 00:16:56.761143   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:16:59.810357   204 log.cpp:536] close a full segment. Current first_index: 1592 last_index: 1598 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001592_00000000000000001598
I20231110 00:16:59.810587   204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001592' to `/data/state/log/log_00000000000000001592_00000000000000001598'
I20231110 00:16:59.810781   204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001599' with fd=47
I20231110 00:17:06.762121   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:17:06.762269   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:17:16.763197   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:17:16.763314   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:17:26.764365   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:17:26.764544   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:17:29.365083   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20231110 00:17:36.765491   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:17:36.765650   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:17:46.766604   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:17:46.766760   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:17:56.767726   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:17:56.767877   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:18:06.768841   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:18:06.769001   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:18:16.770224   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:18:16.770326   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:18:26.771477   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:18:26.771674   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:18:30.372797   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20231110 00:18:36.772691   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:18:36.772852   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:18:46.773964   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:18:46.774140   210 raft_server.h:60] Peer refresh succeeded!
I20231110 00:18:56.775189   166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670
I20231110 00:18:56.775316   166 raft_server.cpp:910] Snapshot timer is active, current_ts: 1699575536, last_snapshot_ts: 1699571936
I20231110 00:18:56.775354   204 raft_server.h:60] Peer refresh succeeded!
I20231110 00:18:56.775398   166 node.cpp:943] node default_group:172.18.0.9:8107:8108 starts to do snapshot
I20231110 00:18:56.776059   210 raft_server.cpp:463] on_snapshot_save
I20231110 00:18:56.776129   210 batched_indexer.cpp:397] Serialized 0 in-flight requests for snapshot.
I20231110 00:18:57.137897   210 raft_server.cpp:405] save_snapshot called
I20231110 00:18:58.025563   210 snapshot.cpp:642] Deleting /data/state/snapshot/snapshot_00000000000000001604
I20231110 00:18:58.026091   210 snapshot.cpp:648] Renaming /data/state/snapshot/temp to /data/state/snapshot/snapshot_00000000000000001604
I20231110 00:18:58.026456   210 snapshot.cpp:519] Deleting /data/state/snapshot/snapshot_00000000000000001522
I20231110 00:18:58.028293   210 snapshot_executor.cpp:234] node default_group:172.18.0.9:8107:8108 snapshot_save_done, last_included_index=1604 last_included_term=2
I20231110 00:18:58.028499   202 raft_server.cpp:994] Timed snapshot succeeded!
I20231110 00:18:58.028934   204 raft_server.cpp:776] Dummy write to http://172.18.0.9:8108/health, status = 200, response = {"ok":true}
I20231110 00:18:58.029006   204 raft_server.cpp:456] save_snapshot done
I20231110 00:18:58.117205   207 log.cpp:1150] log save_meta /data/state/log/log_meta first_log_index: 1523 time: 88709
I20231110 00:19:06.776978   166 raft_server.cpp:564] Term: 2, last_index index: 1605, committed_index: 1605, known_applied_index: 1605, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68673
I20231110 00:19:06.777139   210 raft_server.h:60] Peer refresh succeeded!

immich_microservices logs:


[Nest] 7  - 11/10/2023, 12:04:23 AM     LOG [JobService] Updating queue concurrency settings
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +76ms
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +7ms
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 7  - 11/10/2023, 12:15:22 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:23 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +641ms
[Nest] 7  - 11/10/2023, 12:15:23 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 7  - 11/10/2023, 12:15:23 AM     LOG [InstanceLoader] InfraModule dependencies initialized +5ms
[Nest] 7  - 11/10/2023, 12:15:23 AM     LOG [InstanceLoader] DomainModule dependencies initialized +50ms
[Nest] 7  - 11/10/2023, 12:15:23 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +49ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +7ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +220ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] InfraModule dependencies initialized +4ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] DomainModule dependencies initialized +25ms
[Nest] 7  - 11/10/2023, 12:15:37 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [MetadataService] Initialized local reverse geocoder with cities500
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [SearchService] Running bootstrap
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"assets-v10","name":"assets"}]
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":true,"faces":true}
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [NestApplication] Nest application successfully started +390ms
[Nest] 7  - 11/10/2023, 12:15:50 AM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.85.0] [PRODUCTION]
[Nest] 7  - 11/10/2023, 12:15:51 AM     LOG [SearchService] Indexing 25 albums
[Nest] 7  - 11/10/2023, 12:15:52 AM     LOG [TypesenseRepository] Using new schema: (unset) => albums-v2
[Nest] 7  - 11/10/2023, 12:16:04 AM     LOG [SearchService] Indexing 21836 faces
[Nest] 7  - 11/10/2023, 12:17:02 AM     LOG [TypesenseRepository] Using new schema: (unset) => faces-v1
@davidpan commented on GitHub (Nov 10, 2023): > Can you take the screenshot of facial recognition settings as well as your typesense log <img width="635" alt="image" src="https://github.com/immich-app/immich/assets/12668/660baf44-bde6-4b96-b53f-9b1e9aa4727a"> The json file for the configuration, I posted in my reply earlier. typesense log: ``` I20231109 23:16:26.262069 210 raft_server.h:60] Peer refresh succeeded! I20231109 23:16:28.850704 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20231109 23:16:36.263206 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:16:36.263381 204 raft_server.h:60] Peer refresh succeeded! I20231109 23:16:46.264508 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:16:46.264744 210 raft_server.h:60] Peer refresh succeeded! I20231109 23:16:56.265906 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:16:56.266078 204 raft_server.h:60] Peer refresh succeeded! I20231109 23:17:06.267251 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:17:06.267427 210 raft_server.h:60] Peer refresh succeeded! I20231109 23:17:16.268637 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:17:16.268848 204 raft_server.h:60] Peer refresh succeeded! I20231109 23:17:26.270015 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:17:26.270227 210 raft_server.h:60] Peer refresh succeeded! I20231109 23:17:29.859565 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20231109 23:17:36.271350 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:17:36.271524 204 raft_server.h:60] Peer refresh succeeded! I20231109 23:17:46.272728 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:17:46.272954 210 raft_server.h:60] Peer refresh succeeded! I20231109 23:17:56.274204 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:17:56.274381 204 raft_server.h:60] Peer refresh succeeded! I20231109 23:18:06.275574 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:18:06.275732 210 raft_server.h:60] Peer refresh succeeded! I20231109 23:18:16.276924 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:18:16.277107 204 raft_server.h:60] Peer refresh succeeded! I20231109 23:18:26.278317 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:18:26.278514 210 raft_server.h:60] Peer refresh succeeded! I20231109 23:18:30.868346 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20231109 23:18:36.279769 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:18:36.279942 204 raft_server.h:60] Peer refresh succeeded! I20231109 23:18:46.281116 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:18:46.281275 210 raft_server.h:60] Peer refresh succeeded! I20231109 23:18:56.282516 166 raft_server.cpp:564] Term: 2, last_index index: 1522, committed_index: 1522, known_applied_index: 1522, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2975 I20231109 23:18:56.282683 166 raft_server.cpp:910] Snapshot timer is active, current_ts: 1699571936, last_snapshot_ts: 1699568335 I20231109 23:18:56.282759 204 raft_server.h:60] Peer refresh succeeded! ``` I just restarted immich_microservices vm and below is the log after the restart. typesense logs: ``` I20231110 00:15:27.352088 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20231110 00:15:36.749202 166 raft_server.cpp:564] Term: 2, last_index index: 1523, committed_index: 1523, known_applied_index: 1523, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2978 I20231110 00:15:36.749308 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:15:46.750113 166 raft_server.cpp:564] Term: 2, last_index index: 1523, committed_index: 1523, known_applied_index: 1523, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 2978 I20231110 00:15:46.750206 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:15:56.750955 166 raft_server.cpp:564] Term: 2, last_index index: 1526, committed_index: 1526, known_applied_index: 1526, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3060 I20231110 00:15:56.751076 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:16:06.751930 166 raft_server.cpp:564] Term: 2, last_index index: 1526, committed_index: 1526, known_applied_index: 1526, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 3060 I20231110 00:16:06.752126 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:16:08.570422 204 log.cpp:536] close a full segment. Current first_index: 1441 last_index: 1528 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001441_00000000000000001528 I20231110 00:16:08.570698 204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001441' to `/data/state/log/log_00000000000000001441_00000000000000001528' I20231110 00:16:08.590584 204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001529' with fd=26 I20231110 00:16:14.870448 210 log.cpp:536] close a full segment. Current first_index: 1529 last_index: 1534 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001529_00000000000000001534 I20231110 00:16:14.870646 210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001529' to `/data/state/log/log_00000000000000001529_00000000000000001534' I20231110 00:16:14.870937 210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001535' with fd=24 I20231110 00:16:16.756269 166 raft_server.cpp:564] Term: 2, last_index index: 1537, committed_index: 1537, known_applied_index: 1537, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 12074 I20231110 00:16:16.756393 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:16:18.201920 210 log.cpp:536] close a full segment. Current first_index: 1535 last_index: 1539 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001535_00000000000000001539 I20231110 00:16:18.202150 210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001535' to `/data/state/log/log_00000000000000001535_00000000000000001539' I20231110 00:16:18.202281 210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001540' with fd=32 I20231110 00:16:21.146152 204 log.cpp:536] close a full segment. Current first_index: 1540 last_index: 1544 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001540_00000000000000001544 I20231110 00:16:21.146365 204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001540' to `/data/state/log/log_00000000000000001540_00000000000000001544' I20231110 00:16:21.146546 204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001545' with fd=28 I20231110 00:16:26.467864 204 log.cpp:536] close a full segment. Current first_index: 1545 last_index: 1550 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001545_00000000000000001550 I20231110 00:16:26.468016 204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001545' to `/data/state/log/log_00000000000000001545_00000000000000001550' I20231110 00:16:26.468088 204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001551' with fd=23 I20231110 00:16:26.757413 166 raft_server.cpp:564] Term: 2, last_index index: 1552, committed_index: 1551, known_applied_index: 1551, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 24092 I20231110 00:16:26.757577 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:16:28.357731 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 1 I20231110 00:16:30.246258 210 log.cpp:536] close a full segment. Current first_index: 1551 last_index: 1556 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001551_00000000000000001556 I20231110 00:16:30.246462 210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001551' to `/data/state/log/log_00000000000000001551_00000000000000001556' I20231110 00:16:30.246601 210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001557' with fd=35 I20231110 00:16:33.785753 204 log.cpp:536] close a full segment. Current first_index: 1557 last_index: 1562 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001557_00000000000000001562 I20231110 00:16:33.785879 204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001557' to `/data/state/log/log_00000000000000001557_00000000000000001562' I20231110 00:16:33.785961 204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001563' with fd=33 I20231110 00:16:36.758733 166 raft_server.cpp:564] Term: 2, last_index index: 1567, committed_index: 1567, known_applied_index: 1567, applying_index: 0, queued_writes: 4, pending_queue_size: 0, local_sequence: 36112 I20231110 00:16:36.758976 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:16:38.990906 210 log.cpp:536] close a full segment. Current first_index: 1563 last_index: 1569 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001563_00000000000000001569 I20231110 00:16:38.991134 210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001563' to `/data/state/log/log_00000000000000001563_00000000000000001569' I20231110 00:16:38.991261 210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001570' with fd=25 I20231110 00:16:44.340844 204 log.cpp:536] close a full segment. Current first_index: 1570 last_index: 1575 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001570_00000000000000001575 I20231110 00:16:44.341044 204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001570' to `/data/state/log/log_00000000000000001570_00000000000000001575' I20231110 00:16:44.341136 204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001576' with fd=29 I20231110 00:16:46.760064 166 raft_server.cpp:564] Term: 2, last_index index: 1579, committed_index: 1579, known_applied_index: 1579, applying_index: 0, queued_writes: 1, pending_queue_size: 0, local_sequence: 47635 I20231110 00:16:46.760186 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:16:47.557982 204 log.cpp:536] close a full segment. Current first_index: 1576 last_index: 1581 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001576_00000000000000001581 I20231110 00:16:47.558235 204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001576' to `/data/state/log/log_00000000000000001576_00000000000000001581' I20231110 00:16:47.558512 204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001582' with fd=44 I20231110 00:16:51.910176 210 log.cpp:536] close a full segment. Current first_index: 1582 last_index: 1586 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001582_00000000000000001586 I20231110 00:16:51.910302 210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001582' to `/data/state/log/log_00000000000000001582_00000000000000001586' I20231110 00:16:51.910399 210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001587' with fd=30 I20231110 00:16:55.258899 210 log.cpp:536] close a full segment. Current first_index: 1587 last_index: 1591 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001587_00000000000000001591 I20231110 00:16:55.259092 210 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001587' to `/data/state/log/log_00000000000000001587_00000000000000001591' I20231110 00:16:55.259263 210 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001592' with fd=40 I20231110 00:16:56.761041 166 raft_server.cpp:564] Term: 2, last_index index: 1593, committed_index: 1592, known_applied_index: 1592, applying_index: 0, queued_writes: 0, pending_queue_size: 1, local_sequence: 60145 I20231110 00:16:56.761143 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:16:59.810357 204 log.cpp:536] close a full segment. Current first_index: 1592 last_index: 1598 raft_sync_segments: 0 will_sync: 1 path: /data/state/log/log_00000000000000001592_00000000000000001598 I20231110 00:16:59.810587 204 log.cpp:550] Renamed `/data/state/log/log_inprogress_00000000000000001592' to `/data/state/log/log_00000000000000001592_00000000000000001598' I20231110 00:16:59.810781 204 log.cpp:114] Created new segment `/data/state/log/log_inprogress_00000000000000001599' with fd=47 I20231110 00:17:06.762121 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:17:06.762269 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:17:16.763197 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:17:16.763314 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:17:26.764365 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:17:26.764544 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:17:29.365083 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20231110 00:17:36.765491 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:17:36.765650 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:17:46.766604 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:17:46.766760 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:17:56.767726 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:17:56.767877 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:18:06.768841 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:18:06.769001 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:18:16.770224 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:18:16.770326 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:18:26.771477 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:18:26.771674 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:18:30.372797 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20231110 00:18:36.772691 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:18:36.772852 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:18:46.773964 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:18:46.774140 210 raft_server.h:60] Peer refresh succeeded! I20231110 00:18:56.775189 166 raft_server.cpp:564] Term: 2, last_index index: 1604, committed_index: 1604, known_applied_index: 1604, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68670 I20231110 00:18:56.775316 166 raft_server.cpp:910] Snapshot timer is active, current_ts: 1699575536, last_snapshot_ts: 1699571936 I20231110 00:18:56.775354 204 raft_server.h:60] Peer refresh succeeded! I20231110 00:18:56.775398 166 node.cpp:943] node default_group:172.18.0.9:8107:8108 starts to do snapshot I20231110 00:18:56.776059 210 raft_server.cpp:463] on_snapshot_save I20231110 00:18:56.776129 210 batched_indexer.cpp:397] Serialized 0 in-flight requests for snapshot. I20231110 00:18:57.137897 210 raft_server.cpp:405] save_snapshot called I20231110 00:18:58.025563 210 snapshot.cpp:642] Deleting /data/state/snapshot/snapshot_00000000000000001604 I20231110 00:18:58.026091 210 snapshot.cpp:648] Renaming /data/state/snapshot/temp to /data/state/snapshot/snapshot_00000000000000001604 I20231110 00:18:58.026456 210 snapshot.cpp:519] Deleting /data/state/snapshot/snapshot_00000000000000001522 I20231110 00:18:58.028293 210 snapshot_executor.cpp:234] node default_group:172.18.0.9:8107:8108 snapshot_save_done, last_included_index=1604 last_included_term=2 I20231110 00:18:58.028499 202 raft_server.cpp:994] Timed snapshot succeeded! I20231110 00:18:58.028934 204 raft_server.cpp:776] Dummy write to http://172.18.0.9:8108/health, status = 200, response = {"ok":true} I20231110 00:18:58.029006 204 raft_server.cpp:456] save_snapshot done I20231110 00:18:58.117205 207 log.cpp:1150] log save_meta /data/state/log/log_meta first_log_index: 1523 time: 88709 I20231110 00:19:06.776978 166 raft_server.cpp:564] Term: 2, last_index index: 1605, committed_index: 1605, known_applied_index: 1605, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 68673 I20231110 00:19:06.777139 210 raft_server.h:60] Peer refresh succeeded! ``` immich_microservices logs: ``` [Nest] 7 - 11/10/2023, 12:04:23 AM LOG [JobService] Updating queue concurrency settings [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [NestFactory] Starting Nest application... [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +76ms [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [InstanceLoader] ConfigModule dependencies initialized +7ms [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms [Nest] 7 - 11/10/2023, 12:15:22 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:23 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +641ms [Nest] 7 - 11/10/2023, 12:15:23 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms [Nest] 7 - 11/10/2023, 12:15:23 AM LOG [InstanceLoader] InfraModule dependencies initialized +5ms [Nest] 7 - 11/10/2023, 12:15:23 AM LOG [InstanceLoader] DomainModule dependencies initialized +50ms [Nest] 7 - 11/10/2023, 12:15:23 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [NestFactory] Starting Nest application... [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +49ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] ConfigModule dependencies initialized +7ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +220ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] InfraModule dependencies initialized +4ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] DomainModule dependencies initialized +25ms [Nest] 7 - 11/10/2023, 12:15:37 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [MetadataService] Initialized local reverse geocoder with cities500 [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [SearchService] Running bootstrap [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"assets-v10","name":"assets"}] [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":true,"faces":true} [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [NestApplication] Nest application successfully started +390ms [Nest] 7 - 11/10/2023, 12:15:50 AM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.85.0] [PRODUCTION] [Nest] 7 - 11/10/2023, 12:15:51 AM LOG [SearchService] Indexing 25 albums [Nest] 7 - 11/10/2023, 12:15:52 AM LOG [TypesenseRepository] Using new schema: (unset) => albums-v2 [Nest] 7 - 11/10/2023, 12:16:04 AM LOG [SearchService] Indexing 21836 faces [Nest] 7 - 11/10/2023, 12:17:02 AM LOG [TypesenseRepository] Using new schema: (unset) => faces-v1 ```
OVERLORD commented 2026-02-05 02:29:10 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 10, 2023):

Repeat the face recognition task again and don't see the problem at this time.

Could it be that you previously used some of the remote face recognition VM processing?

@davidpan commented on GitHub (Nov 10, 2023): Repeat the face recognition task again and don't see the problem at this time. Could it be that you previously used some of the remote face recognition VM processing?
OVERLORD commented 2026-02-05 02:29:14 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 10, 2023):

psql query :

immich=# select count(*) from asset_faces;
 count
-------
   833
(1 row)

immich=# select count(*) from person;
 count
-------
   244
(1 row)
@davidpan commented on GitHub (Nov 10, 2023): psql query : ``` immich=# select count(*) from asset_faces; count ------- 833 (1 row) immich=# select count(*) from person; count ------- 244 (1 row) ```
OVERLORD commented 2026-02-05 02:29:20 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Nov 10, 2023):

Facial recognition relies on clustering faces together. And that calculation is made in typesense. It is possible that some issue with typesense led to no faces being matched and subsequently new people being created for each one. It's possible the typesense issue resolved itself when you did a restart and then everything worked properly afterwards.

@jrasm91 commented on GitHub (Nov 10, 2023): Facial recognition relies on clustering faces together. And that calculation is made in typesense. It is possible that some issue with typesense led to no faces being matched and subsequently new people being created for each one. It's possible the typesense issue resolved itself when you did a restart and then everything worked properly afterwards.
OVERLORD commented 2026-02-05 02:29:22 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 10, 2023):

Facial recognition relies on clustering faces together. And that calculation is made in typesense. It is possible that some issue with typesense led to no faces being matched and subsequently new people being created for each one. It's possible the typesense issue resolved itself when you did a restart and then everything worked properly afterwards.

There was no typesense restart, only a restart of immich_microservices.

Based on my current understanding and existing phenomena, it appears to be an issue of asynchronous synchronization between the Postgres database and Typesense ?

@davidpan commented on GitHub (Nov 10, 2023): > Facial recognition relies on clustering faces together. And that calculation is made in typesense. It is possible that some issue with typesense led to no faces being matched and subsequently new people being created for each one. It's possible the typesense issue resolved itself when you did a restart and then everything worked properly afterwards. There was no typesense restart, only a restart of immich_microservices. Based on my current understanding and existing phenomena, it appears to be an issue of asynchronous synchronization between the Postgres database and Typesense ?
OVERLORD commented 2026-02-05 02:29:26 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Nov 10, 2023):

Facial recognition relies on clustering faces together. And that calculation is made in typesense. It is possible that some issue with typesense led to no faces being matched and subsequently new people being created for each one. It's possible the typesense issue resolved itself when you did a restart and then everything worked properly afterwards.

There was no typesense restart, only a restart of immich_microservices.

Based on my current understanding and existing phenomena, it appears to be an issue of asynchronous synchronization between the Postgres database and Typesense ?

Yeah, I quickly scanned through the issue, but yes - data synchronizes from postgres to typesense and then is used for closeness queries. A problem with that sync is most likely the issue.

@jrasm91 commented on GitHub (Nov 10, 2023): > > Facial recognition relies on clustering faces together. And that calculation is made in typesense. It is possible that some issue with typesense led to no faces being matched and subsequently new people being created for each one. It's possible the typesense issue resolved itself when you did a restart and then everything worked properly afterwards. > > There was no typesense restart, only a restart of immich_microservices. > > Based on my current understanding and existing phenomena, it appears to be an issue of asynchronous synchronization between the Postgres database and Typesense ? Yeah, I quickly scanned through the issue, but yes - data synchronizes from postgres to typesense and then is used for closeness queries. A problem with that sync is most likely the issue.
OVERLORD commented 2026-02-05 02:29:29 +03:00
Author
Owner
Copy Link

@davidpan commented on GitHub (Nov 10, 2023):

immich_microservices logs:

Request #1699591568273: Sleeping for 4s and then retrying request...
Request #1699591568273: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591568273: Sleeping for 4s and then retrying request...
Request #1699591593270: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591593270: Sleeping for 4s and then retrying request...
Request #1699591598271: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591598271: Sleeping for 4s and then retrying request...
Request #1699591593270: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591593270: Sleeping for 4s and then retrying request...
Request #1699591603260: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591603260: Sleeping for 4s and then retrying request...
Request #1699591608268: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591608268: Sleeping for 4s and then retrying request...
Request #1699591593270: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591593270: Sleeping for 4s and then retrying request...
Request #1699591608268: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591608268: Sleeping for 4s and then retrying request...
Request #1699591593270: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591593270: Sleeping for 4s and then retrying request...
Request #1699591623254: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591623254: Sleeping for 4s and then retrying request...
Request #1699591608268: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1699591608268: Sleeping for 4s and then retrying request...
Request #1699591628253: Request to Node 0 failed due to "ECONNRESET socket hang up"

but Checking typesense service status is normal ?

docker exec -ti immich_microservices /bin/bash
curl http://172.18.0.9:8108/health
{"ok":true}

curl http://immich_typesense:8108/health
{"ok":true}

typesense logs:

I20231110 04:51:49.340808   204 raft_server.h:60] Peer refresh succeeded!
E20231110 04:51:50.340910   166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500
I20231110 04:51:59.342160   166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920
E20231110 04:51:59.355644   166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500
I20231110 04:51:59.355744   210 raft_server.h:60] Peer refresh succeeded!
I20231110 04:52:01.465101   167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
E20231110 04:52:08.356701   166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500
I20231110 04:52:09.356995   166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920
I20231110 04:52:09.357213   204 raft_server.h:60] Peer refresh succeeded!
E20231110 04:52:17.358305   166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500
I20231110 04:52:19.358726   166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920
I20231110 04:52:19.358907   210 raft_server.h:60] Peer refresh succeeded!
E20231110 04:52:26.359800   166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500
I20231110 04:52:29.360343   166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920
I20231110 04:52:29.360559   204 raft_server.h:60] Peer refresh succeeded!
E20231110 04:52:35.361035   166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500
I20231110 04:52:39.361752   166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920
I20231110 04:52:39.361982   210 raft_server.h:60] Peer refresh succeeded!
E20231110 04:52:44.362555   166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500
I20231110 04:52:49.363415   166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920
I20231110 04:52:49.363642   204 raft_server.h:60] Peer refresh succeeded!
E20231110 04:52:53.364151   166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500
@davidpan commented on GitHub (Nov 10, 2023): immich_microservices logs: ``` Request #1699591568273: Sleeping for 4s and then retrying request... Request #1699591568273: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591568273: Sleeping for 4s and then retrying request... Request #1699591593270: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591593270: Sleeping for 4s and then retrying request... Request #1699591598271: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591598271: Sleeping for 4s and then retrying request... Request #1699591593270: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591593270: Sleeping for 4s and then retrying request... Request #1699591603260: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591603260: Sleeping for 4s and then retrying request... Request #1699591608268: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591608268: Sleeping for 4s and then retrying request... Request #1699591593270: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591593270: Sleeping for 4s and then retrying request... Request #1699591608268: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591608268: Sleeping for 4s and then retrying request... Request #1699591593270: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591593270: Sleeping for 4s and then retrying request... Request #1699591623254: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591623254: Sleeping for 4s and then retrying request... Request #1699591608268: Request to Node 0 failed due to "ECONNRESET socket hang up" Request #1699591608268: Sleeping for 4s and then retrying request... Request #1699591628253: Request to Node 0 failed due to "ECONNRESET socket hang up" ``` but Checking typesense service status is normal ? ``` docker exec -ti immich_microservices /bin/bash curl http://172.18.0.9:8108/health {"ok":true} curl http://immich_typesense:8108/health {"ok":true} ``` typesense logs: ``` I20231110 04:51:49.340808 204 raft_server.h:60] Peer refresh succeeded! E20231110 04:51:50.340910 166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500 I20231110 04:51:59.342160 166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920 E20231110 04:51:59.355644 166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500 I20231110 04:51:59.355744 210 raft_server.h:60] Peer refresh succeeded! I20231110 04:52:01.465101 167 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 E20231110 04:52:08.356701 166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500 I20231110 04:52:09.356995 166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920 I20231110 04:52:09.357213 204 raft_server.h:60] Peer refresh succeeded! E20231110 04:52:17.358305 166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500 I20231110 04:52:19.358726 166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920 I20231110 04:52:19.358907 210 raft_server.h:60] Peer refresh succeeded! E20231110 04:52:26.359800 166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500 I20231110 04:52:29.360343 166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920 I20231110 04:52:29.360559 204 raft_server.h:60] Peer refresh succeeded! E20231110 04:52:35.361035 166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500 I20231110 04:52:39.361752 166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920 I20231110 04:52:39.361982 210 raft_server.h:60] Peer refresh succeeded! E20231110 04:52:44.362555 166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500 I20231110 04:52:49.363415 166 raft_server.cpp:564] Term: 2, last_index index: 8516, committed_index: 8516, known_applied_index: 8516, applying_index: 0, queued_writes: 501, pending_queue_size: 0, local_sequence: 198920 I20231110 04:52:49.363642 204 raft_server.h:60] Peer refresh succeeded! E20231110 04:52:53.364151 166 raft_server.cpp:654] 501 queued writes > healthy write lag of 500 ```
OVERLORD commented 2026-02-05 02:29:33 +03:00
Author
Owner
Copy Link

@tsemachh commented on GitHub (Nov 16, 2023):

We had same issue , I uploaded 300K images from all times albums(migrated from G Photos)
I defined 40 Threads for the facial , and this was a mistake because I got the same exact face in different pictures few times.
This is because each Thread of facial recognition runs on isolated transaction to my understanding and when the match is done it's possible other threads with same person also attempts to perform the match , this way we got 4 and 5 same persons as different persons , it's a lot of manual work to merge them , anyway when running with 1 thread all was o.k.
To summarize we need to think how can we make the face recognition process work in a way it will be easy to fix or make the recognition process in typesense synchronized in some way.

@tsemachh commented on GitHub (Nov 16, 2023): We had same issue , I uploaded 300K images from all times albums(migrated from G Photos) I defined 40 Threads for the facial , and this was a mistake because I got the same exact face in different pictures few times. This is because each Thread of facial recognition runs on isolated transaction to my understanding and when the match is done it's possible other threads with same person also attempts to perform the match , this way we got 4 and 5 same persons as different persons , it's a lot of manual work to merge them , anyway when running with 1 thread all was o.k. To summarize we need to think how can we make the face recognition process work in a way it will be easy to fix or make the recognition process in typesense synchronized in some way.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:translation
cli
date-time
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1596
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?