immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-24 19:08:04 +03:00
Code Issues 783 Packages Projects Releases 12 Wiki Activity

[BUG] Typesense is not starting after update to v1.62.1 #976

New Issue
Closed
opened 2026-02-04 23:45:36 +03:00 by OVERLORD · 3 comments
OVERLORD commented 2026-02-04 23:45:36 +03:00
Owner
Copy Link

Originally created by @bbieberle on GitHub (Jun 20, 2023).

The bug

After updating to v1.62.1 my typesense container has a startup problem. I tried to rollback to 1.61.0 but the problem was still there. I assume this is some kind of borked typesense data?

│ I20230620 08:24:22.900575     1 typesense_server_utils.cpp:357] Starting Typesense 0.24.0                                                                                                                                                                                              │
│ I20230620 08:24:22.900625     1 typesense_server_utils.cpp:360] Typesense is using jemalloc.                                                                                                                                                                                           │
│ I20230620 08:24:22.901129     1 typesense_server_utils.cpp:409] Thread pool size: 32                                                                                                                                                                                                   │
│ I20230620 08:24:22.910552     1 store.h:63] Initializing DB by opening state dir: /tsdata/db                                                                                                                                                                                           │
│ I20230620 08:24:22.981109     1 store.h:63] Initializing DB by opening state dir: /tsdata/meta                                                                                                                                                                                         │
│ I20230620 08:24:23.037550     1 ratelimit_manager.cpp:475] Loaded 0 rate limit rules.                                                                                                                                                                                                  │
│ I20230620 08:24:23.037570     1 ratelimit_manager.cpp:476] Loaded 0 rate limit bans.                                                                                                                                                                                                   │
│ I20230620 08:24:23.037634     1 typesense_server_utils.cpp:504] Starting API service...                                                                                                                                                                                                │
│ I20230620 08:24:23.037787     1 http_server.cpp:178] Typesense has started listening on port 8108                                                                                                                                                                                      │
│ I20230620 08:24:23.038038   166 typesense_server_utils.cpp:255] Since no --nodes argument is provided, starting a single node Typesense cluster.                                                                                                                                       │
│ I20230620 08:24:23.038439   167 batched_indexer.cpp:124] Starting batch indexer with 32 threads.                                                                                                                                                                                       │
│ I20230620 08:24:23.046463   166 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107.                                                                                                              │
│ I20230620 08:24:23.046479   166 server.cpp:1048] Check out http://immich-release-typesense-5d8b69fb7f-p5qxw:8107 in web browser.                                                                                                                                                       │
│ I20230620 08:24:23.046694   166 raft_server.cpp:65] Nodes configuration: 10.42.0.213:8107:8108                                                                                                                                                                                         │
│ I20230620 08:24:23.047500   167 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999                                                                                                                                                                                              │
│ I20230620 08:24:23.049089   166 log.cpp:674] Use crc32c as the checksum type of appending entries                                                                                                                                                                                      │
│ I20230620 08:24:23.049530   166 log.cpp:1158] log load_meta /tsdata/state/log/log_meta first_log_index: 51188 time: 424                                                                                                                                                                │
│ I20230620 08:24:23.049635   166 log.cpp:1098] load open segment, path: /tsdata/state/log first_index: 50627                                                                                                                                                                            │
│ I20230620 08:24:23.051201   166 raft_meta.cpp:521] Loaded single stable meta, path /tsdata/state/meta term 10 votedfor 10.42.0.93:8107:8108 time: 238                                                                                                                                  │
│ I20230620 08:24:23.052065   181 raft_server.cpp:511] on_snapshot_load                                                                                                                                                                                                                  │
│ I20230620 08:24:23.087462   181 store.h:298] rm /tsdata/db success                                                                                                                                                                                                                     │
│ I20230620 08:24:23.111184   181 store.h:308] copy snapshot /tsdata/state/snapshot/snapshot_00000000000000051188/db_snapshot to /tsdata/db success                                                                                                                                      │
│ I20230620 08:24:23.111375   181 store.h:63] Initializing DB by opening state dir: /tsdata/db                                                                                                                                                                                           │
│ I20230620 08:24:23.168947   181 store.h:322] DB open success!                                                                                                                                                                                                                          │
│ I20230620 08:24:23.168963   181 raft_server.cpp:490] Loading collections from disk...                                                                                                                                                                                                  │
│ I20230620 08:24:23.168977   181 collection_manager.cpp:159] CollectionManager::load()                                                                                                                                                                                                  │
│ I20230620 08:24:23.169215   181 auth_manager.cpp:34] Indexing 0 API key(s) found on disk.                                                                                                                                                                                              │
│ I20230620 08:24:23.169247   181 collection_manager.cpp:179] Loading upto 16 collections in parallel, 1000 documents at a time.                                                                                                                                                         │
│ I20230620 08:24:23.169284   181 collection_manager.cpp:188] Found 3 collection(s) on disk.                                                                                                                                                                                             │
│ I20230620 08:24:23.169960   226 collection_manager.cpp:109] Found collection albums-v1 with 4 memory shards.                                                                                                                                                                           │
│ I20230620 08:24:23.170112   228 collection_manager.cpp:109] Found collection assets-v7 with 4 memory shards.                                                                                                                                                                           │
│ I20230620 08:24:23.170147   226 collection_manager.cpp:1203] Loading collection albums-v1                                                                                                                                                                                              │
│ I20230620 08:24:23.170411   227 collection_manager.cpp:109] Found collection faces-v1 with 4 memory shards.                                                                                                                                                                            │
│ I20230620 08:24:23.172364   227 collection_manager.cpp:1203] Loading collection faces-v1                                                                                                                                                                                               │
│ I20230620 08:24:23.172947   228 collection_manager.cpp:1203] Loading collection assets-v7                                                                                                                                                                                              │
│ I20230620 08:24:23.173628   226 collection_manager.cpp:1320] Indexed 22/22 documents into collection albums-v1                                                                                                                                                                         │
│ I20230620 08:24:23.173655   226 collection_manager.cpp:228] Loaded 1 collection(s) so far                                                                                                                                                                                              │
│ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [127.0627000, 68.4030000]                                                                                                             │
│ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [12.1337000, 565.0590000]                                                                                                             │
│ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [12.1376000, 572.7020000]                                                                                                             │
│ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [12.1369000, 567.8900000]                                                                                                             │
│ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [126.9228000, 29.9310000]  
I20230620 08:24:52.554098     1 typesense_server_utils.cpp:48] Stopping Typesense server...                                                                                                                                                                                            │
│ I20230620 08:24:52.555671   228 collection_manager.cpp:1320] Indexed 20000/20901 documents into collection assets-v7                                                                                                                                                                   │
│ I20230620 08:24:52.705886   228 collection_manager.cpp:228] Loaded 2 collection(s) so far                                                                                                                                                                                              │
│ I20230620 08:24:52.852463   227 collection_manager.cpp:1320] Indexed 33000/33000 documents into collection faces-v1                                                                                                                                                                    │
│ I20230620 08:24:52.852645   227 collection_manager.cpp:228] Loaded 3 collection(s) so far                                                                                                                                                                                              │
│ I20230620 08:24:52.852782   181 collection_manager.cpp:278] Loaded 3 collection(s).                                                                                                                                                                                                    │
│ I20230620 08:24:52.853505   181 collection_manager.cpp:282] Initializing batched indexer from snapshot state...                                                                                                                                                                        │
│ I20230620 08:24:52.853554   181 batched_indexer.cpp:441] Restored 0 in-flight requests from snapshot.                                                                                                                                                                                  │
│ I20230620 08:24:52.853565   181 raft_server.cpp:497] Finished loading collections from disk.                                                                                                                                                                                           │
│ I20230620 08:24:52.853577   181 raft_server.h:273] Configuration of this group is 10.42.0.93:8107:8108                                                                                                                                                                                 │
│ I20230620 08:24:52.853605   181 snapshot_executor.cpp:264] node default_group:10.42.0.213:8107:8108 snapshot_load_done, last_included_index: 51188 last_included_term: 10 peers: "10.42.0.93:8107:8108"                                                                                │
│ I20230620 08:24:52.853750   166 raft_meta.cpp:521] Loaded single stable meta, path /tsdata/state/meta term 10 votedfor 10.42.0.93:8107:8108 time: 77                                                                                                                                   │
│ I20230620 08:24:52.853767   166 node.cpp:576] node default_group:10.42.0.213:8107:8108 init, term: 10 last_log_id: (index=51189,term=10) conf: 10.42.0.93:8107:8108 old_conf:                                                                                                          │
│ I20230620 08:24:52.853806   166 raft_server.cpp:129] Node last_index: 51189                                                                                                                                                                                                            │
│ I20230620 08:24:52.853814   166 typesense_server_utils.cpp:306] Typesense peering service is running on 10.42.0.213:8107                                                                                                                                                               │
│ I20230620 08:24:52.853819   166 typesense_server_utils.cpp:307] Snapshot interval configured as: 3600s                                                                                                                                                                                 │
│ I20230620 08:24:52.853823   166 typesense_server_utils.cpp:308] Snapshot max byte count configured as: 4194304                                                                                                                                                                         │
│ W20230620 08:24:52.853830   166 controller.cpp:1454] SIGINT was installed with 1                                                                                                                                                                                                       │
│ I20230620 08:24:52.853835   166 typesense_server_utils.cpp:340] Typesense peering service is going to quit.                                                                                                                                                                            │
│ I20230620 08:24:52.853840   166 raft_server.cpp:781] Set shutting_down = true                                                                                                                                                                                                          │
│ I20230620 08:24:52.853844   166 raft_server.cpp:785] Waiting for in-flight writes to finish...                                                                                                                                                                                         │
│ I20230620 08:24:52.853848   166 raft_server.cpp:791] Replication state shutdown, store sequence: 616392                                                                                                                                                                                │
│ I20230620 08:24:52.853853   166 raft_server.cpp:795] node->shutdown                                                                                                                                                                                                                    │
│ I20230620 08:24:52.853857   166 node.cpp:929] node default_group:10.42.0.213:8107:8108 shutdown, current_term 10 state FOLLOWER                                                                                                                                                        │
│ I20230620 08:24:52.853874   166 raft_server.cpp:799] node->join                                                                                                                                                                                                                        │
│ I20230620 08:24:52.853888   181 raft_server.h:265] This node is down                                                                                                                                                                                                                   │
│ I20230620 08:24:52.853909   166 node.cpp:929] node default_group:10.42.0.213:8107:8108 shutdown, current_term 10 state SHUTDOWN                                                                                                                                                        │
│ I20230620 08:24:52.854068   166 typesense_server_utils.cpp:345] raft_server.stop()                                                                                                                                                                                                     │
│ I20230620 08:24:52.854076   166 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit                                                                                                                      │
│ I20230620 08:24:52.854205   166 typesense_server_utils.cpp:348] raft_server.join()                                                                                                                                                                                                     │
│ I20230620 08:24:52.854233   166 typesense_server_utils.cpp:351] Typesense peering service has quit.                                                                                                                                                                                    │
│ I20230620 08:24:52.854403   166 typesense_server_utils.cpp:483] Shutting down batch indexer...                                                                                                                                                                                         │
│ I20230620 08:24:52.854411   166 typesense_server_utils.cpp:486] Waiting for batch indexing thread to be done...                                                                                                                                                                        │
│ I20230620 08:24:53.097316   167 batched_indexer.cpp:311] Notifying batch indexer threads about shutdown...                                                                                                                                                                             │
│ I20230620 08:24:53.097445   167 batched_indexer.cpp:319] Batched indexer threadpool shutdown...                                                                                                                                                                                        │
│ I20230620 08:24:53.098897   166 typesense_server_utils.cpp:489] Shutting down server_thread_pool                                                                                                                                                                                       │
│ I20230620 08:24:53.100102   166 typesense_server_utils.cpp:493] Shutting down app_thread_pool.                                                                                                                                                                                         │
│ I20230620 08:24:53.101214   166 typesense_server_utils.cpp:497] Shutting down replication_thread_pool.                                                                                                                                                                                 │
│ I20230620 08:24:53.103644     1 typesense_server_utils.cpp:511] Typesense API service has quit.                                                                                                                                                                                        │
│ I20230620 08:24:53.103727     1 typesense_server_utils.cpp:515] Deleting batch indexer                                                                                                                                                                                                 │
│ I20230620 08:24:53.103745     1 typesense_server_utils.cpp:519] CURL clean up       
│ I20230620 08:24:53.103751     1 typesense_server_utils.cpp:523] Deleting server                                                                                                                                                                                                        │
│ I20230620 08:24:53.104840     1 typesense_server_utils.cpp:527] CollectionManager dispose, this might take some time...                                                                                                                                                                │
│ I20230620 08:24:53.284337     1 typesense_server_utils.cpp:531] Bye.

The OS that Immich Server is running on

Ubuntu 23.04, Kubernetes (K3s)

Version of Immich Server

v1.62.1

Version of Immich Mobile App

v1.62.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

see https://artifacthub.io/packages/helm/immich/immich

my helmrelease:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: immich-data
  namespace: immich
  labels:
    app.kubernetes.io/instance: immich-release
spec:
  storageClassName: localnvme
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi


---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: immich-release
  namespace: flux-system
spec:
  interval: 1m
  targetNamespace: immich
  releaseName: immich-release
  chart:
    spec:
      version: 0.1.1
      chart: immich 
      sourceRef:
        kind: HelmRepository
        name: immich
        namespace: flux-system
      interval: 1m
  valuesFrom:
    - kind: Secret
      name: immich-secret
      valuesKey: jwtSecret
      targetPath: common_env.JWT_SECRET
    - kind: Secret
      name: immich-secret
      valuesKey: dbPassword
      targetPath: common_env.DB_PASSWORD
    - kind: Secret
      name: immich-secret
      valuesKey: dbPassword
      targetPath: postgresql.global.postgresql.auth.password
  values:
    image:
      tag: v1.62.1
    common_env:
        REDIS_HOSTNAME: '{{ printf "%s-redis-master" (include "common.names.fullname" .) }}'
        DB_HOSTNAME: '{{ include "common.names.name" . }}-postgresql'
        DB_USERNAME: immich
        DB_DATABASE_NAME: immich
        DISABLE_REVERSE_GEOCODING: true
        ENABLE_MAPBOX: false
    postgresql:
      enabled: true
      global:
        postgresql:
          auth:
            username: immich
            database: immich
      primary:
        resources:
          requests:
            cpu: 20m
    redis:
      enabled: true

    immich:
      persistence:
        library:
          existingClaim: immich-data
    
    typesense:
      enabled: true
      persistence:
        tsdata:
          enabled: true  
          storageClass: localnvme

    proxy:
      ingress:
        main:
          enabled: true
          primary: enabled
          hosts:
            - host: example.com
              paths:
                - path: "/"
          tls:
            - hosts:
                - example.com
              secretName: tls-immich-ingress
          annotations:
            cert-manager.io/cluster-issuer: letsencrypt-issuer
            traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
            traefik.ingress.kubernetes.io/router.middlewares: traefik-https-redirect@kubernetescrd, traefik-security-headers@kubernetescrd



### Your .env content

```Shell
DB_DATABASE_NAME:             immich                                                                                                                                                                                                                                             DB_HOSTNAME:                  immich-release-postgresql                                                                                                                                                                                                                          DB_PASSWORD:                  XXXX                                                                                                                                                                                                          DB_USERNAME:                  immich                                                                                                                                                                                                                                             IMMICH_MACHINE_LEARNING_URL:  http://immich-release-machine-learning:3003                                                                                                                                                                                                        IMMICH_SERVER_URL:            http://immich-release-server:3001                                                                                                                                                                                                                  IMMICH_WEB_URL:               http://immich-release-web:3000                                                                                                                                                                                                                     REDIS_HOSTNAME:               immich-release-redis-master                                                                                                                                                                                                                        TYPESENSE_API_KEY:            XXXX
TYPESENSE_DATA_DIR:           /tsdata                                                                                                                                                                                                                                            TYPESENSE_ENABLED:            true                                                                                                                                                                                                                                               TYPESENSE_HOST:               immich-release-typesense

Reproduction steps

I updated the helmrelease to v1.62.1

Additional information

No response

Originally created by @bbieberle on GitHub (Jun 20, 2023). ### The bug After updating to v1.62.1 my typesense container has a startup problem. I tried to rollback to 1.61.0 but the problem was still there. I assume this is some kind of borked typesense data? ``` │ I20230620 08:24:22.900575 1 typesense_server_utils.cpp:357] Starting Typesense 0.24.0 │ │ I20230620 08:24:22.900625 1 typesense_server_utils.cpp:360] Typesense is using jemalloc. │ │ I20230620 08:24:22.901129 1 typesense_server_utils.cpp:409] Thread pool size: 32 │ │ I20230620 08:24:22.910552 1 store.h:63] Initializing DB by opening state dir: /tsdata/db │ │ I20230620 08:24:22.981109 1 store.h:63] Initializing DB by opening state dir: /tsdata/meta │ │ I20230620 08:24:23.037550 1 ratelimit_manager.cpp:475] Loaded 0 rate limit rules. │ │ I20230620 08:24:23.037570 1 ratelimit_manager.cpp:476] Loaded 0 rate limit bans. │ │ I20230620 08:24:23.037634 1 typesense_server_utils.cpp:504] Starting API service... │ │ I20230620 08:24:23.037787 1 http_server.cpp:178] Typesense has started listening on port 8108 │ │ I20230620 08:24:23.038038 166 typesense_server_utils.cpp:255] Since no --nodes argument is provided, starting a single node Typesense cluster. │ │ I20230620 08:24:23.038439 167 batched_indexer.cpp:124] Starting batch indexer with 32 threads. │ │ I20230620 08:24:23.046463 166 server.cpp:1045] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is serving on port=8107. │ │ I20230620 08:24:23.046479 166 server.cpp:1048] Check out http://immich-release-typesense-5d8b69fb7f-p5qxw:8107 in web browser. │ │ I20230620 08:24:23.046694 166 raft_server.cpp:65] Nodes configuration: 10.42.0.213:8107:8108 │ │ I20230620 08:24:23.047500 167 batched_indexer.cpp:129] BatchedIndexer skip_index: -9999 │ │ I20230620 08:24:23.049089 166 log.cpp:674] Use crc32c as the checksum type of appending entries │ │ I20230620 08:24:23.049530 166 log.cpp:1158] log load_meta /tsdata/state/log/log_meta first_log_index: 51188 time: 424 │ │ I20230620 08:24:23.049635 166 log.cpp:1098] load open segment, path: /tsdata/state/log first_index: 50627 │ │ I20230620 08:24:23.051201 166 raft_meta.cpp:521] Loaded single stable meta, path /tsdata/state/meta term 10 votedfor 10.42.0.93:8107:8108 time: 238 │ │ I20230620 08:24:23.052065 181 raft_server.cpp:511] on_snapshot_load │ │ I20230620 08:24:23.087462 181 store.h:298] rm /tsdata/db success │ │ I20230620 08:24:23.111184 181 store.h:308] copy snapshot /tsdata/state/snapshot/snapshot_00000000000000051188/db_snapshot to /tsdata/db success │ │ I20230620 08:24:23.111375 181 store.h:63] Initializing DB by opening state dir: /tsdata/db │ │ I20230620 08:24:23.168947 181 store.h:322] DB open success! │ │ I20230620 08:24:23.168963 181 raft_server.cpp:490] Loading collections from disk... │ │ I20230620 08:24:23.168977 181 collection_manager.cpp:159] CollectionManager::load() │ │ I20230620 08:24:23.169215 181 auth_manager.cpp:34] Indexing 0 API key(s) found on disk. │ │ I20230620 08:24:23.169247 181 collection_manager.cpp:179] Loading upto 16 collections in parallel, 1000 documents at a time. │ │ I20230620 08:24:23.169284 181 collection_manager.cpp:188] Found 3 collection(s) on disk. │ │ I20230620 08:24:23.169960 226 collection_manager.cpp:109] Found collection albums-v1 with 4 memory shards. │ │ I20230620 08:24:23.170112 228 collection_manager.cpp:109] Found collection assets-v7 with 4 memory shards. │ │ I20230620 08:24:23.170147 226 collection_manager.cpp:1203] Loading collection albums-v1 │ │ I20230620 08:24:23.170411 227 collection_manager.cpp:109] Found collection faces-v1 with 4 memory shards. │ │ I20230620 08:24:23.172364 227 collection_manager.cpp:1203] Loading collection faces-v1 │ │ I20230620 08:24:23.172947 228 collection_manager.cpp:1203] Loading collection assets-v7 │ │ I20230620 08:24:23.173628 226 collection_manager.cpp:1320] Indexed 22/22 documents into collection albums-v1 │ │ I20230620 08:24:23.173655 226 collection_manager.cpp:228] Loaded 1 collection(s) so far │ │ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [127.0627000, 68.4030000] │ │ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [12.1337000, 565.0590000] │ │ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [12.1376000, 572.7020000] │ │ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [12.1369000, 567.8900000] │ │ /typesense/external-Linux/s2geometry-efb124d8eaf3433323d3e877dedd5e94a63339a3/src/s2/s2latlng.cc:37 ERROR Invalid S2LatLng in S2LatLng::ToPoint: [126.9228000, 29.9310000] I20230620 08:24:52.554098 1 typesense_server_utils.cpp:48] Stopping Typesense server... │ │ I20230620 08:24:52.555671 228 collection_manager.cpp:1320] Indexed 20000/20901 documents into collection assets-v7 │ │ I20230620 08:24:52.705886 228 collection_manager.cpp:228] Loaded 2 collection(s) so far │ │ I20230620 08:24:52.852463 227 collection_manager.cpp:1320] Indexed 33000/33000 documents into collection faces-v1 │ │ I20230620 08:24:52.852645 227 collection_manager.cpp:228] Loaded 3 collection(s) so far │ │ I20230620 08:24:52.852782 181 collection_manager.cpp:278] Loaded 3 collection(s). │ │ I20230620 08:24:52.853505 181 collection_manager.cpp:282] Initializing batched indexer from snapshot state... │ │ I20230620 08:24:52.853554 181 batched_indexer.cpp:441] Restored 0 in-flight requests from snapshot. │ │ I20230620 08:24:52.853565 181 raft_server.cpp:497] Finished loading collections from disk. │ │ I20230620 08:24:52.853577 181 raft_server.h:273] Configuration of this group is 10.42.0.93:8107:8108 │ │ I20230620 08:24:52.853605 181 snapshot_executor.cpp:264] node default_group:10.42.0.213:8107:8108 snapshot_load_done, last_included_index: 51188 last_included_term: 10 peers: "10.42.0.93:8107:8108" │ │ I20230620 08:24:52.853750 166 raft_meta.cpp:521] Loaded single stable meta, path /tsdata/state/meta term 10 votedfor 10.42.0.93:8107:8108 time: 77 │ │ I20230620 08:24:52.853767 166 node.cpp:576] node default_group:10.42.0.213:8107:8108 init, term: 10 last_log_id: (index=51189,term=10) conf: 10.42.0.93:8107:8108 old_conf: │ │ I20230620 08:24:52.853806 166 raft_server.cpp:129] Node last_index: 51189 │ │ I20230620 08:24:52.853814 166 typesense_server_utils.cpp:306] Typesense peering service is running on 10.42.0.213:8107 │ │ I20230620 08:24:52.853819 166 typesense_server_utils.cpp:307] Snapshot interval configured as: 3600s │ │ I20230620 08:24:52.853823 166 typesense_server_utils.cpp:308] Snapshot max byte count configured as: 4194304 │ │ W20230620 08:24:52.853830 166 controller.cpp:1454] SIGINT was installed with 1 │ │ I20230620 08:24:52.853835 166 typesense_server_utils.cpp:340] Typesense peering service is going to quit. │ │ I20230620 08:24:52.853840 166 raft_server.cpp:781] Set shutting_down = true │ │ I20230620 08:24:52.853844 166 raft_server.cpp:785] Waiting for in-flight writes to finish... │ │ I20230620 08:24:52.853848 166 raft_server.cpp:791] Replication state shutdown, store sequence: 616392 │ │ I20230620 08:24:52.853853 166 raft_server.cpp:795] node->shutdown │ │ I20230620 08:24:52.853857 166 node.cpp:929] node default_group:10.42.0.213:8107:8108 shutdown, current_term 10 state FOLLOWER │ │ I20230620 08:24:52.853874 166 raft_server.cpp:799] node->join │ │ I20230620 08:24:52.853888 181 raft_server.h:265] This node is down │ │ I20230620 08:24:52.853909 166 node.cpp:929] node default_group:10.42.0.213:8107:8108 shutdown, current_term 10 state SHUTDOWN │ │ I20230620 08:24:52.854068 166 typesense_server_utils.cpp:345] raft_server.stop() │ │ I20230620 08:24:52.854076 166 server.cpp:1100] Server[braft::RaftStatImpl+braft::FileServiceImpl+braft::RaftServiceImpl+braft::CliServiceImpl] is going to quit │ │ I20230620 08:24:52.854205 166 typesense_server_utils.cpp:348] raft_server.join() │ │ I20230620 08:24:52.854233 166 typesense_server_utils.cpp:351] Typesense peering service has quit. │ │ I20230620 08:24:52.854403 166 typesense_server_utils.cpp:483] Shutting down batch indexer... │ │ I20230620 08:24:52.854411 166 typesense_server_utils.cpp:486] Waiting for batch indexing thread to be done... │ │ I20230620 08:24:53.097316 167 batched_indexer.cpp:311] Notifying batch indexer threads about shutdown... │ │ I20230620 08:24:53.097445 167 batched_indexer.cpp:319] Batched indexer threadpool shutdown... │ │ I20230620 08:24:53.098897 166 typesense_server_utils.cpp:489] Shutting down server_thread_pool │ │ I20230620 08:24:53.100102 166 typesense_server_utils.cpp:493] Shutting down app_thread_pool. │ │ I20230620 08:24:53.101214 166 typesense_server_utils.cpp:497] Shutting down replication_thread_pool. │ │ I20230620 08:24:53.103644 1 typesense_server_utils.cpp:511] Typesense API service has quit. │ │ I20230620 08:24:53.103727 1 typesense_server_utils.cpp:515] Deleting batch indexer │ │ I20230620 08:24:53.103745 1 typesense_server_utils.cpp:519] CURL clean up │ I20230620 08:24:53.103751 1 typesense_server_utils.cpp:523] Deleting server │ │ I20230620 08:24:53.104840 1 typesense_server_utils.cpp:527] CollectionManager dispose, this might take some time... │ │ I20230620 08:24:53.284337 1 typesense_server_utils.cpp:531] Bye. ``` ### The OS that Immich Server is running on Ubuntu 23.04, Kubernetes (K3s) ### Version of Immich Server v1.62.1 ### Version of Immich Mobile App v1.62.0 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML see https://artifacthub.io/packages/helm/immich/immich my helmrelease: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: immich-data namespace: immich labels: app.kubernetes.io/instance: immich-release spec: storageClassName: localnvme accessModes: - ReadWriteMany resources: requests: storage: 500Gi --- apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: immich-release namespace: flux-system spec: interval: 1m targetNamespace: immich releaseName: immich-release chart: spec: version: 0.1.1 chart: immich sourceRef: kind: HelmRepository name: immich namespace: flux-system interval: 1m valuesFrom: - kind: Secret name: immich-secret valuesKey: jwtSecret targetPath: common_env.JWT_SECRET - kind: Secret name: immich-secret valuesKey: dbPassword targetPath: common_env.DB_PASSWORD - kind: Secret name: immich-secret valuesKey: dbPassword targetPath: postgresql.global.postgresql.auth.password values: image: tag: v1.62.1 common_env: REDIS_HOSTNAME: '{{ printf "%s-redis-master" (include "common.names.fullname" .) }}' DB_HOSTNAME: '{{ include "common.names.name" . }}-postgresql' DB_USERNAME: immich DB_DATABASE_NAME: immich DISABLE_REVERSE_GEOCODING: true ENABLE_MAPBOX: false postgresql: enabled: true global: postgresql: auth: username: immich database: immich primary: resources: requests: cpu: 20m redis: enabled: true immich: persistence: library: existingClaim: immich-data typesense: enabled: true persistence: tsdata: enabled: true storageClass: localnvme proxy: ingress: main: enabled: true primary: enabled hosts: - host: example.com paths: - path: "/" tls: - hosts: - example.com secretName: tls-immich-ingress annotations: cert-manager.io/cluster-issuer: letsencrypt-issuer traefik.ingress.kubernetes.io/router.entrypoints: web,websecure traefik.ingress.kubernetes.io/router.middlewares: traefik-https-redirect@kubernetescrd, traefik-security-headers@kubernetescrd ``` ``` ### Your .env content ```Shell DB_DATABASE_NAME: immich DB_HOSTNAME: immich-release-postgresql DB_PASSWORD: XXXX DB_USERNAME: immich IMMICH_MACHINE_LEARNING_URL: http://immich-release-machine-learning:3003 IMMICH_SERVER_URL: http://immich-release-server:3001 IMMICH_WEB_URL: http://immich-release-web:3000 REDIS_HOSTNAME: immich-release-redis-master TYPESENSE_API_KEY: XXXX TYPESENSE_DATA_DIR: /tsdata TYPESENSE_ENABLED: true TYPESENSE_HOST: immich-release-typesense ``` ### Reproduction steps ```bash I updated the helmrelease to v1.62.1 ``` ### Additional information _No response_
OVERLORD commented 2026-02-04 23:45:50 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Jun 20, 2023):

Can you try remove the typesense volume and try again? also we are using 1.24.1 for typesense

@alextran1502 commented on GitHub (Jun 20, 2023): Can you try remove the typesense volume and try again? also we are using 1.24.1 for typesense
OVERLORD commented 2026-02-04 23:45:58 +03:00
Author
Owner
Copy Link

@bbieberle commented on GitHub (Jun 20, 2023):

I was unsure, if deleting the typesense PVC would be of concern. But i guess it worked without hickup.
I will create a pull request in the helm chart repository to represent the version change to 0.24.1 as there is still the old version in the default values.
Thanks for the quick help. Love your work <3

@bbieberle commented on GitHub (Jun 20, 2023): I was unsure, if deleting the typesense PVC would be of concern. But i guess it worked without hickup. I will create a pull request in the helm chart repository to represent the version change to 0.24.1 as there is still the old version in the default values. Thanks for the quick help. Love your work <3
OVERLORD commented 2026-02-04 23:46:00 +03:00
Author
Owner
Copy Link

@NiklasRosenstein commented on GitHub (Sep 21, 2023):

Is deleting the type sense volume the recommended solution for this issue? This is the second time I'm experiencing it, and it happened after the K8s node that typesense was running on had died.

@NiklasRosenstein commented on GitHub (Sep 21, 2023): Is deleting the type sense volume the recommended solution for this issue? This is the second time I'm experiencing it, and it happened after the K8s node that typesense was running on had died.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#976
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?