immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-25 14:00:45 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Redis crashing while scanning external library on Kubernetes #4280

New Issue
Closed
opened 2026-02-05 09:59:15 +03:00 by OVERLORD · 7 comments
OVERLORD commented 2026-02-05 09:59:15 +03:00
Owner
Copy Link

Originally created by @Echelon101 on GitHub (Sep 9, 2024).

The bug

shortly after starting initial scan of my external libraries redis restarts and immich server restarts shortly after

External Library contains about 100k assets

The OS that Immich Server is running on

Ubuntu 24.04 LTS

Version of Immich Server

v1.114.0

Version of Immich Mobile App

v1.114.0

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

Deployed via Helm! Here is my modified server.yaml
{{- define "immich.server.hardcodedValues" -}}
global:
  nameOverride: server

env:
  {{ if .Values.immich.metrics.enabled }}
      IMMICH_METRICS: true
  {{ end }}
  {{- if .Values.immich.configuration }}
      IMMICH_CONFIG_FILE: /config/immich-config.yaml
  {{- end }}

{{- if .Values.immich.configuration }}
podAnnotations:
  checksum/config: {{ .Values.immich.configuration | toYaml | sha256sum }}
{{- end }}

service:
  main:
    enabled: true
    primary: true
    type: ClusterIP
    ports:
      http:
        enabled: true
        primary: true
        port: 3001
        protocol: HTTP
      metrics-api:
        enabled: {{ .Values.immich.metrics.enabled }}
        port: 8081
        protocol: HTTP
      metrics-ms:
        enabled: {{ .Values.immich.metrics.enabled }}
        port: 8082
        protocol: HTTP


serviceMonitor:
  main:
    enabled: {{ .Values.immich.metrics.enabled }}
    endpoints:
      - port: metrics-api
        scheme: http
      - port: metrics-ms
        scheme: http

probes:
  liveness: &probes
    enabled: true
    custom: true
    spec:
      httpGet:
        path: /api/server-info/ping
        port: http
      initialDelaySeconds: 0
      periodSeconds: 10
      timeoutSeconds: 1
      failureThreshold: 3
  readiness: *probes
  startup:
    enabled: true
    custom: true
    spec:
      httpGet:
        path: /api/server-info/ping
        port: http
      initialDelaySeconds: 0
      periodSeconds: 10
      timeoutSeconds: 1
      failureThreshold: 30

persistence:
{{- if .Values.immich.configuration }}
  config:
    enabled: true
    type: configMap
    name: {{ .Release.Name }}-immich-config
{{- end }}
  library:
    enabled: true
    mountPath: /usr/src/app/upload
    existingClaim: {{ .Values.immich.persistence.library.existingClaim }}
  family:
    enabled: true
    mountPath: /mnt/family
    existingClaim: photo-family
  storage:
    enabled: true
    mountPath: /mnt/storage
    existingClaim: photo-storage
  other:
    enabled: true
    mountPath: /mnt/other
    existingClaim: photo-other
{{- end }}
#  {{- range $.Values.immich.persistence.external }}
#  {{ .name }}
#    enabled: true
#    mountPath: {{ .mountPath }}
#    existingClaim: {{ .existingClaim }}
#  {{- end}}
#
{{ if .Values.server.enabled }}
{{- $ctx := deepCopy . -}}
{{- $_ := get .Values "server" | mergeOverwrite $ctx.Values -}}
{{- $_ = include "immich.server.hardcodedValues" . | fromYaml | merge $ctx.Values -}}
{{- include "bjw-s.common.loader.all" $ctx }}
{{ end }}

Your .env content

No special env set. Set by Helmchart

Reproduction steps

  1. Deploy Helm Chart
  2. Add External Library
  3. Start Library Refresh
  4. Wait for fail
    ...

Relevant log output

Immich Server Log:
[Nest] 7  - 09/09/2024, 6:14:06 PM     LOG [Microservices:Bootstrap] Immich Microservices is running [v1.114.0] [PRODUCTION] 
[Nest] 7  - 09/09/2024, 6:16:11 PM     LOG [Microservices:LibraryService] Refreshing library 5ee1f56a-bdab-450c-adff-d80d59bfb8bb
Error: read ECONNRESET
    at TCP.onStreamRead (node:internal/stream_base_commons:218:20) {
  errno: -104,
  code: 'ECONNRESET',
  syscall: 'read'
}
[Nest] 7  - 09/09/2024, 6:16:34 PM   ERROR [Microservices:JobService] Unable to run job handler (library/library-refresh): AbortError: Command aborted due to connection close
[Nest] 7  - 09/09/2024, 6:16:34 PM   ERROR [Microservices:JobService] AbortError: Command aborted due to connection close
    at abortError (/usr/src/app/node_modules/ioredis/built/redis/event_handler.js:81:17)
    at abortIncompletePipelines (/usr/src/app/node_modules/ioredis/built/redis/event_handler.js:105:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/event_handler.js:140:13)
    at Object.onceWrapper (node:events:634:26)
    at Socket.emit (node:events:519:28)
    at TCP.<anonymous> (node:net:339:12)
[Nest] 7  - 09/09/2024, 6:16:34 PM   ERROR [Microservices:JobService] Object:
{
  "id": "5ee1f56a-bdab-450c-adff-d80d59bfb8bb",
  "refreshModifiedFiles": false,
  "refreshAllFiles": false
}

missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
missing 'error' handler on this Redis client
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}
Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0)
    at internalConnect (node:net:1094:16)
    at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18)
    at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8)
    at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  errno: -1,
  code: 'EPERM',
  syscall: 'connect',
  address: '10.43.91.65',
  port: 6379
}

REDIS Log:
1:C 09 Sep 2024 18:16:35.446 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 09 Sep 2024 18:16:35.446 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 09 Sep 2024 18:16:35.446 * Configuration loaded
1:M 09 Sep 2024 18:16:35.447 * monotonic clock: POSIX clock_gettime
1:M 09 Sep 2024 18:16:35.449 * Running mode=standalone, port=6379.
1:M 09 Sep 2024 18:16:35.451 * Server initialized
1:M 09 Sep 2024 18:16:35.467 * Reading RDB base file on AOF loading...
1:M 09 Sep 2024 18:16:35.467 * Loading RDB produced by version 7.2.5
1:M 09 Sep 2024 18:16:35.467 * RDB age 272 seconds
1:M 09 Sep 2024 18:16:35.467 * RDB memory usage when created 0.86 Mb
1:M 09 Sep 2024 18:16:35.467 * RDB is base AOF
1:M 09 Sep 2024 18:16:35.467 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 09 Sep 2024 18:16:35.467 * DB loaded from base file appendonly.aof.1.base.rdb: 0.005 seconds
1:M 09 Sep 2024 18:16:35.539 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.071 seconds
1:M 09 Sep 2024 18:16:35.539 * DB loaded from append only file: 0.077 seconds
1:M 09 Sep 2024 18:16:35.539 * Opening AOF incr file appendonly.aof.1.incr.aof on server start
1:M 09 Sep 2024 18:16:35.539 * Ready to accept connections tcp

Additional information

Helm Chart with tiny modifications for to directly add mounts for external libraries

Originally created by @Echelon101 on GitHub (Sep 9, 2024). ### The bug shortly after starting initial scan of my external libraries redis restarts and immich server restarts shortly after External Library contains about 100k assets ### The OS that Immich Server is running on Ubuntu 24.04 LTS ### Version of Immich Server v1.114.0 ### Version of Immich Mobile App v1.114.0 ### Platform with the issue - [X] Server - [ ] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML Deployed via Helm! Here is my modified server.yaml {{- define "immich.server.hardcodedValues" -}} global: nameOverride: server env: {{ if .Values.immich.metrics.enabled }} IMMICH_METRICS: true {{ end }} {{- if .Values.immich.configuration }} IMMICH_CONFIG_FILE: /config/immich-config.yaml {{- end }} {{- if .Values.immich.configuration }} podAnnotations: checksum/config: {{ .Values.immich.configuration | toYaml | sha256sum }} {{- end }} service: main: enabled: true primary: true type: ClusterIP ports: http: enabled: true primary: true port: 3001 protocol: HTTP metrics-api: enabled: {{ .Values.immich.metrics.enabled }} port: 8081 protocol: HTTP metrics-ms: enabled: {{ .Values.immich.metrics.enabled }} port: 8082 protocol: HTTP serviceMonitor: main: enabled: {{ .Values.immich.metrics.enabled }} endpoints: - port: metrics-api scheme: http - port: metrics-ms scheme: http probes: liveness: &probes enabled: true custom: true spec: httpGet: path: /api/server-info/ping port: http initialDelaySeconds: 0 periodSeconds: 10 timeoutSeconds: 1 failureThreshold: 3 readiness: *probes startup: enabled: true custom: true spec: httpGet: path: /api/server-info/ping port: http initialDelaySeconds: 0 periodSeconds: 10 timeoutSeconds: 1 failureThreshold: 30 persistence: {{- if .Values.immich.configuration }} config: enabled: true type: configMap name: {{ .Release.Name }}-immich-config {{- end }} library: enabled: true mountPath: /usr/src/app/upload existingClaim: {{ .Values.immich.persistence.library.existingClaim }} family: enabled: true mountPath: /mnt/family existingClaim: photo-family storage: enabled: true mountPath: /mnt/storage existingClaim: photo-storage other: enabled: true mountPath: /mnt/other existingClaim: photo-other {{- end }} # {{- range $.Values.immich.persistence.external }} # {{ .name }} # enabled: true # mountPath: {{ .mountPath }} # existingClaim: {{ .existingClaim }} # {{- end}} # {{ if .Values.server.enabled }} {{- $ctx := deepCopy . -}} {{- $_ := get .Values "server" | mergeOverwrite $ctx.Values -}} {{- $_ = include "immich.server.hardcodedValues" . | fromYaml | merge $ctx.Values -}} {{- include "bjw-s.common.loader.all" $ctx }} {{ end }} ``` ### Your .env content ```Shell No special env set. Set by Helmchart ``` ### Reproduction steps 1. Deploy Helm Chart 2. Add External Library 3. Start Library Refresh 4. Wait for fail ... ### Relevant log output ```shell Immich Server Log: [Nest] 7 - 09/09/2024, 6:14:06 PM  LOG [Microservices:Bootstrap] Immich Microservices is running [v1.114.0] [PRODUCTION]  [Nest] 7 - 09/09/2024, 6:16:11 PM  LOG [Microservices:LibraryService] Refreshing library 5ee1f56a-bdab-450c-adff-d80d59bfb8bb Error: read ECONNRESET at TCP.onStreamRead (node:internal/stream_base_commons:218:20) { errno: -104, code: 'ECONNRESET', syscall: 'read' } [Nest] 7 - 09/09/2024, 6:16:34 PM  ERROR [Microservices:JobService] Unable to run job handler (library/library-refresh): AbortError: Command aborted due to connection close [Nest] 7 - 09/09/2024, 6:16:34 PM  ERROR [Microservices:JobService] AbortError: Command aborted due to connection close at abortError (/usr/src/app/node_modules/ioredis/built/redis/event_handler.js:81:17) at abortIncompletePipelines (/usr/src/app/node_modules/ioredis/built/redis/event_handler.js:105:28) at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/redis/event_handler.js:140:13) at Object.onceWrapper (node:events:634:26) at Socket.emit (node:events:519:28) at TCP.<anonymous> (node:net:339:12) [Nest] 7 - 09/09/2024, 6:16:34 PM  ERROR [Microservices:JobService] Object: { "id": "5ee1f56a-bdab-450c-adff-d80d59bfb8bb", "refreshModifiedFiles": false, "refreshAllFiles": false } missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client missing 'error' handler on this Redis client Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } Error: connect EPERM 10.43.91.65:6379 - Local (0.0.0.0:0) at internalConnect (node:net:1094:16) at defaultTriggerAsyncIdScope (node:internal/async_hooks:464:18) at GetAddrInfoReqWrap.emitLookup [as callback] (node:net:1400:9) at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:111:8) at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:130:17) { errno: -1, code: 'EPERM', syscall: 'connect', address: '10.43.91.65', port: 6379 } REDIS Log: 1:C 09 Sep 2024 18:16:35.446 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1:C 09 Sep 2024 18:16:35.446 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started 1:C 09 Sep 2024 18:16:35.446 * Configuration loaded 1:M 09 Sep 2024 18:16:35.447 * monotonic clock: POSIX clock_gettime 1:M 09 Sep 2024 18:16:35.449 * Running mode=standalone, port=6379. 1:M 09 Sep 2024 18:16:35.451 * Server initialized 1:M 09 Sep 2024 18:16:35.467 * Reading RDB base file on AOF loading... 1:M 09 Sep 2024 18:16:35.467 * Loading RDB produced by version 7.2.5 1:M 09 Sep 2024 18:16:35.467 * RDB age 272 seconds 1:M 09 Sep 2024 18:16:35.467 * RDB memory usage when created 0.86 Mb 1:M 09 Sep 2024 18:16:35.467 * RDB is base AOF 1:M 09 Sep 2024 18:16:35.467 * Done loading RDB, keys loaded: 0, keys expired: 0. 1:M 09 Sep 2024 18:16:35.467 * DB loaded from base file appendonly.aof.1.base.rdb: 0.005 seconds 1:M 09 Sep 2024 18:16:35.539 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.071 seconds 1:M 09 Sep 2024 18:16:35.539 * DB loaded from append only file: 0.077 seconds 1:M 09 Sep 2024 18:16:35.539 * Opening AOF incr file appendonly.aof.1.incr.aof on server start 1:M 09 Sep 2024 18:16:35.539 * Ready to accept connections tcp ``` ### Additional information Helm Chart with tiny modifications for to directly add mounts for external libraries
OVERLORD commented 2026-02-05 09:59:18 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Sep 10, 2024):

@etnoy do you remember if we've seen previous cases of the library scan taking out redis?

@bo0tzz commented on GitHub (Sep 10, 2024): @etnoy do you remember if we've seen previous cases of the library scan taking out redis?
OVERLORD commented 2026-02-05 09:59:19 +03:00
Author
Owner
Copy Link

@etnoy commented on GitHub (Sep 10, 2024):

Yes, I've seen this before. @Echelon101 what is the system specs, especially ram? Can you provide redis logs?

@etnoy commented on GitHub (Sep 10, 2024): Yes, I've seen this before. @Echelon101 what is the system specs, especially ram? Can you provide redis logs?
OVERLORD commented 2026-02-05 09:59:21 +03:00
Author
Owner
Copy Link

@Echelon101 commented on GitHub (Sep 10, 2024):

I'll have a look at my ram / CPU limits in my cluster. Overall the nodes shouldn't have an issue with CPU/ram usage because a node has 8c / 32gb. All redis logs that I was able to grab are at the end of the relevant logs field in this issue.

@Echelon101 commented on GitHub (Sep 10, 2024): I'll have a look at my ram / CPU limits in my cluster. Overall the nodes shouldn't have an issue with CPU/ram usage because a node has 8c / 32gb. All redis logs that I was able to grab are at the end of the relevant logs field in this issue.
OVERLORD commented 2026-02-05 09:59:21 +03:00
Author
Owner
Copy Link

@Echelon101 commented on GitHub (Sep 10, 2024):

I've changed the limits for redis that the container is able to get up to 2gi of ram. For currently unknown reasons the readiness / liveness probes are failing and kubernetes restarts the pod.

here the latest log of redis

2024-09-10T12:14:22.694396014Z 1:C 10 Sep 2024 12:14:22.694 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
2024-09-10T12:14:22.694403306Z 1:C 10 Sep 2024 12:14:22.694 * Configuration loaded
2024-09-10T12:14:22.694787457Z 1:M 10 Sep 2024 12:14:22.694 * monotonic clock: POSIX clock_gettime
2024-09-10T12:14:22.695559842Z 1:M 10 Sep 2024 12:14:22.695 * Running mode=standalone, port=6379.
2024-09-10T12:14:22.696380063Z 1:M 10 Sep 2024 12:14:22.696 * Server initialized
2024-09-10T12:14:22.706857723Z 1:M 10 Sep 2024 12:14:22.706 * Reading RDB base file on AOF loading...
2024-09-10T12:14:22.706942895Z 1:M 10 Sep 2024 12:14:22.706 * Loading RDB produced by version 7.2.5
2024-09-10T12:14:22.706958354Z 1:M 10 Sep 2024 12:14:22.706 * RDB age 64939 seconds
2024-09-10T12:14:22.706980231Z 1:M 10 Sep 2024 12:14:22.706 * RDB memory usage when created 0.86 Mb
2024-09-10T12:14:22.706995107Z 1:M 10 Sep 2024 12:14:22.706 * RDB is base AOF
2024-09-10T12:14:22.707009691Z 1:M 10 Sep 2024 12:14:22.706 * Done loading RDB, keys loaded: 0, keys expired: 0.
2024-09-10T12:14:22.707049652Z 1:M 10 Sep 2024 12:14:22.706 * DB loaded from base file appendonly.aof.1.base.rdb: 0.005 seconds
2024-09-10T12:14:23.495938847Z 1:M 10 Sep 2024 12:14:23.495 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.789 seconds
2024-09-10T12:14:23.496003018Z 1:M 10 Sep 2024 12:14:23.495 * DB loaded from append only file: 0.794 seconds
2024-09-10T12:14:23.496007977Z 1:M 10 Sep 2024 12:14:23.495 * Opening AOF incr file appendonly.aof.1.incr.aof on server start
2024-09-10T12:14:23.496012644Z 1:M 10 Sep 2024 12:14:23.495 * Ready to accept connections tcp
2024-09-10T12:16:04.291359969Z 1:signal-handler (1725970564) Received SIGTERM scheduling shutdown...
2024-09-10T12:16:11.996308990Z 1:M 10 Sep 2024 12:16:11.996 * User requested shutdown...
2024-09-10T12:16:11.996362660Z 1:M 10 Sep 2024 12:16:11.996 * Calling fsync() on the AOF file.
2024-09-10T12:16:14.767383501Z 1:M 10 Sep 2024 12:16:14.767 # Redis is now ready to exit, bye bye...
@Echelon101 commented on GitHub (Sep 10, 2024): I've changed the limits for redis that the container is able to get up to 2gi of ram. For currently unknown reasons the readiness / liveness probes are failing and kubernetes restarts the pod. here the latest log of redis ```shell 2024-09-10T12:14:22.694396014Z 1:C 10 Sep 2024 12:14:22.694 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started 2024-09-10T12:14:22.694403306Z 1:C 10 Sep 2024 12:14:22.694 * Configuration loaded 2024-09-10T12:14:22.694787457Z 1:M 10 Sep 2024 12:14:22.694 * monotonic clock: POSIX clock_gettime 2024-09-10T12:14:22.695559842Z 1:M 10 Sep 2024 12:14:22.695 * Running mode=standalone, port=6379. 2024-09-10T12:14:22.696380063Z 1:M 10 Sep 2024 12:14:22.696 * Server initialized 2024-09-10T12:14:22.706857723Z 1:M 10 Sep 2024 12:14:22.706 * Reading RDB base file on AOF loading... 2024-09-10T12:14:22.706942895Z 1:M 10 Sep 2024 12:14:22.706 * Loading RDB produced by version 7.2.5 2024-09-10T12:14:22.706958354Z 1:M 10 Sep 2024 12:14:22.706 * RDB age 64939 seconds 2024-09-10T12:14:22.706980231Z 1:M 10 Sep 2024 12:14:22.706 * RDB memory usage when created 0.86 Mb 2024-09-10T12:14:22.706995107Z 1:M 10 Sep 2024 12:14:22.706 * RDB is base AOF 2024-09-10T12:14:22.707009691Z 1:M 10 Sep 2024 12:14:22.706 * Done loading RDB, keys loaded: 0, keys expired: 0. 2024-09-10T12:14:22.707049652Z 1:M 10 Sep 2024 12:14:22.706 * DB loaded from base file appendonly.aof.1.base.rdb: 0.005 seconds 2024-09-10T12:14:23.495938847Z 1:M 10 Sep 2024 12:14:23.495 * DB loaded from incr file appendonly.aof.1.incr.aof: 0.789 seconds 2024-09-10T12:14:23.496003018Z 1:M 10 Sep 2024 12:14:23.495 * DB loaded from append only file: 0.794 seconds 2024-09-10T12:14:23.496007977Z 1:M 10 Sep 2024 12:14:23.495 * Opening AOF incr file appendonly.aof.1.incr.aof on server start 2024-09-10T12:14:23.496012644Z 1:M 10 Sep 2024 12:14:23.495 * Ready to accept connections tcp 2024-09-10T12:16:04.291359969Z 1:signal-handler (1725970564) Received SIGTERM scheduling shutdown... 2024-09-10T12:16:11.996308990Z 1:M 10 Sep 2024 12:16:11.996 * User requested shutdown... 2024-09-10T12:16:11.996362660Z 1:M 10 Sep 2024 12:16:11.996 * Calling fsync() on the AOF file. 2024-09-10T12:16:14.767383501Z 1:M 10 Sep 2024 12:16:14.767 # Redis is now ready to exit, bye bye... ```
OVERLORD commented 2026-02-05 09:59:22 +03:00
Author
Owner
Copy Link

@etnoy commented on GitHub (Sep 10, 2024):

Thanks for pointing me to the logs. The issue I've seen is when you go above 1M assets with less memory than the 32g you have. In that case, our bullmq library makes very expensive calls to redis which causes it to go haywire. This, however, looks quite different, so I'm not sure it's the same issue.

I find it strange that it's an EPERM error which sounds like permissions.

@etnoy commented on GitHub (Sep 10, 2024): Thanks for pointing me to the logs. The issue I've seen is when you go above 1M assets with less memory than the 32g you have. In that case, our bullmq library makes very expensive calls to redis which causes it to go haywire. This, however, looks quite different, so I'm not sure it's the same issue. I find it strange that it's an EPERM error which sounds like permissions.
OVERLORD commented 2026-02-05 09:59:23 +03:00
Author
Owner
Copy Link

@Echelon101 commented on GitHub (Sep 10, 2024):

With the latest try immich is now able to queue items and is starting to work through the queue. I'll have a look how it behaves with subsequent scans. But at least immich was able to start tasks even though redis was taken out initially

@Echelon101 commented on GitHub (Sep 10, 2024): With the latest try immich is now able to queue items and is starting to work through the queue. I'll have a look how it behaves with subsequent scans. But at least immich was able to start tasks even though redis was taken out initially
OVERLORD commented 2026-02-05 09:59:24 +03:00
Author
Owner
Copy Link

@Echelon101 commented on GitHub (Sep 13, 2024):

@etnoy after incresing all resource limits redis isn't going down anymore if i scan again. thank you for your support

@Echelon101 commented on GitHub (Sep 13, 2024): @etnoy after incresing all resource limits redis isn't going down anymore if i scan again. thank you for your support
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#4280
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?