immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-02-05 08:41:28 +03:00
Code Issues 69 Packages Projects Releases 10 Wiki Activity

[BUG] 504 errors importing videos with CLI using --import #1311

New Issue
Closed
opened 2026-02-05 01:13:47 +03:00 by OVERLORD · 13 comments
OVERLORD commented 2026-02-05 01:13:47 +03:00
Owner
Copy Link

Originally created by @yodatak on GitHub (Sep 7, 2023).

Originally assigned to: @jrasm91 on GitHub.

The bug

in immich-microservices

2023-09-07T16:46:34.760277344+02:00 [Nest] 8  - 09/07/2023, 4:46:34 PM   ERROR [JobService] Unable to run job handler: TypeError: Cannot read properties of null (reading 'to')
2023-09-07T16:46:34.760442799+02:00 [Nest] 8  - 09/07/2023, 4:46:34 PM   ERROR [JobService] TypeError: Cannot read properties of null (reading 'to')
2023-09-07T16:46:34.760450096+02:00     at CommunicationRepository.send (/usr/src/app/dist/infra/repositories/communication.repository.js:20:24)
2023-09-07T16:46:34.760453473+02:00     at JobService.onDone (/usr/src/app/dist/domain/job/job.service.js:164:50)
2023-09-07T16:46:34.760456347+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:46:34.760459978+02:00     at async /usr/src/app/dist/domain/job/job.service.js:107:25
2023-09-07T16:46:34.760462728+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-09-07T16:46:34.760465157+02:00     at async Work
[Nest] 8  - 09/07/2023, 4:48:44 PM   ERROR [JobService] Unable to run job handler: Error: Connection terminated due to connection timeout
[Nest] 8  - 09/07/2023, 4:48:44 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
2023-09-07T16:48:44.470777854+02:00     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
2023-09-07T16:48:44.470781573+02:00     at Object.onceWrapper (node:events:628:28)
2023-09-07T16:48:44.470784056+02:00     at Connection.emit (node:events:514:28)
2023-09-07T16:48:44.470786605+02:00     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-09-07T16:48:44.470789076+02:00     at Socket.emit (node:events:514:28)
2023-09-07T16:48:44.470791493+02:00     at TCP.<anonymous> (node:net:323:12)
[Nest] 8  - 09/07/2023, 4:48:44 PM   ERROR [JobService] Object:
2023-09-07T16:48:44.471332649+02:00 {
2023-09-07T16:48:44.471335870+02:00   "id": "1810065b-7f56-418b-ae57-3162ecdb3f33",
2023-09-07T16:48:44.471338484+02:00   "source": "upload"
2023-09-07T16:48:44.471341241+02:00 }
2023-09-07T16:48:44.471343509+02:00 
[Nest] 8  - 09/07/2023, 4:48:45 PM   ERROR [JobService] Unable to run job handler: Error: Connection terminated due to connection timeout
2023-09-07T16:48:45.551578848+02:00 [Nest] 8  - 09/07/2023, 4:48:45 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
2023-09-07T16:48:45.551586947+02:00     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
2023-09-07T16:48:45.551589740+02:00     at Object.onceWrapper (node:events:628:28)
2023-09-07T16:48:45.551592245+02:00     at Connection.emit (node:events:514:28)
2023-09-07T16:48:45.551594918+02:00     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-09-07T16:48:45.551597387+02:00     at Socket.emit (node:events:514:28)
2023-09-07T16:48:45.551599834+02:00     at TCP.<anonymous> (node:net:323:12)
2023-09-07T16:48:45.551602460+02:00 [Nest] 8  - 09/07/2023, 4:48:45 PM   ERROR [JobService] Object:
2023-09-07T16:48:45.551604977+02:00 {
2023-09-07T16:48:45.551607547+02:00   "id": "ef3ed216-dea7-429e-910f-35ec4eb9fd20",
2023-09-07T16:48:45.551610294+02:00   "source": "upload"
2023-09-07T16:48:45.551612738+02:00 }
2023-09-07T16:48:45.551615079+02:00 
[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] Unable to run job handler: TypeError: fetch failed
[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] TypeError: fetch failed
   at Object.fetch (node:internal/deps/undici/undici:11576:11)
   at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
   at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
   at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
   at async /usr/src/app/dist/domain/job/job.service.js:105:37
   at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
   at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] Object:
{
 "id": "56e660e8-238a-4349-af42-2f75521648dc",
 "source": "upload"
}

[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] Unable to run job handler: TypeError: fetch failed
[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] TypeError: fetch failed
   at Object.fetch (node:internal/deps/undici/undici:11576:11)
   at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
   at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
   at async FacialRecognitionService.handleRecognizeFaces (/usr/src/app/dist/domain/facial-recognition/facial-recognition.services.js:105:23)
2023-09-07T16:48:47.267395051+02:00     at async /usr/src/app/dist/domain/job/job.service.js:105:37
2023-09-07T16:48:47.267405270+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-09-07T16:48:47.267429326+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-09-07T16:48:47.267439507+02:00 [Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] Object:
2023-09-07T16:48:47.267448981+02:00 {
2023-09-07T16:48:47.267458765+02:00   "id": "1691264c-94dd-4d3f-b25d-ce29a91a2a1e",
2023-09-07T16:48:47.267468597+02:00   "source": "upload"
2023-09-07T16:48:47.267478003+02:00 }
2023-09-07T16:48:47.267486760+02:00 
[Nest] 8  - 09/07/2023, 4:48:50 PM     LOG [MediaService] Successfully generated image thumbnail 5b9612f8-b89d-4526-8cd0-17787a48866b
[Nest] 8  - 09/07/2023, 4:48:50 PM   ERROR [JobService] Unable to run job handler: TypeError: Cannot read properties of null (reading 'to')
[Nest] 8  - 09/07/2023, 4:48:50 PM   ERROR [JobService] TypeError: Cannot read properties of null (reading 'to')
2023-09-07T16:48:50.857432731+02:00     at CommunicationRepository.send (/usr/src/app/dist/infra/repositories/communication.repository.js:20:24)
2023-09-07T16:48:50.857436489+02:00     at JobService.onDone (/usr/src/app/dist/domain/job/job.service.js:164:50)
2023-09-07T16:48:50.857438951+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:50.857441504+02:00     at async /usr/src/app/dist/domain/job/job.service.js:107:25
2023-09-07T16:48:50.857443889+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-09-07T16:48:50.857446167+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)

I also goterror in search tab in web 500 error

in immich server

2023-09-07T16:45:49.881940765+02:00 [Nest] 7  - 09/07/2023, 4:45:49 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:49.882261150+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:49.882270180+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:45:49.882274754+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:45:49.882277310+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:45:49.882279386+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:45:49.882281386+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:45:49.882283935+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:49.882285941+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:45:49.882288266+02:00     at Object.next (/usr/src/ap
p/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:49.882290528+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:45:49.882292452+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:45:49.991596213+02:00 [Nest] 7  - 09/07/2023, 4:45:49 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:49.991687056+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:49.991693086+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:45:49.991695052+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:45:49.991696661+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:45:49.991698067+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:45:49.991699431+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:45:49.991701057+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:49.991702431+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
    at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
    at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[Nest] 7  - 09/07/2023, 4:45:55 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:55.015817553+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:45:55.015820320+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:45:55.015823095+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:45:55.015825871+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:45:55.015828015+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:45:55.015830852+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:55.015833377+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:45:55.015836364+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:55.015838592+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:45:55.015841214+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:46:00.458230458+02:00 [Nest] 7  - 09/07/2023, 4:46:00 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:00.458252527+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:00.458255335+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:46:00.458257644+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-
07T16:46:00.458259687+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:46:00.458261842+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:46:00.458264165+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:46:00.458266391+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:00.458268392+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:46:00.458270328+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:00.458272424+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:46:00.458274496+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:46:12.218016163+02:00 [Nest] 7  - 09/07/2023, 4:46:12 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:12.218047522+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:12.218050928+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:46:12.218053146+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:46:12.218057615+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:46:12.218059798+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:46:12.218061658+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:46:12.218063630+02:00     at Object.next (/usr/src/app/node_
modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:12.218065509+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:46:12.218067335+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:12.218069140+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:46:12.218070975+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:46:12.599054790+02:00 [Nest] 7  - 09/07/2023, 4:46:12 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:12.599076786+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:12.599079762+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:46:12.599082027+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:46:12.599083892+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:46:12.599085793+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:46:12.599087629+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:46:12.599089901+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:12.599091948+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:46:12.599094087+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:12.599096188+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:46:12.59909
8065+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:35.701930921+02:00 [Nest] 7  - 09/07/2023, 4:48:35 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:35.701962834+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:35.701965546+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:48:35.701967771+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:35.701974384+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:48:35.701984479+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:35.701986839+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:48:35.701992892+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:35.701995201+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:35.701997417+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:35.701999483+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:35.702002124+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:37.783875036+02:00 [Nest] 7  - 09/07/2023, 4:48:37 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:37.783899772+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:37.783903284+
02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:48:37.783906232+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:37.783910871+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:48:37.783913160+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:37.783915242+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:48:37.783917783+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:37.783920207+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:37.783922453+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:37.783924333+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:37.783926257+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:37.934863402+02:00 [Nest] 7  - 09/07/2023, 4:48:37 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:37.937045849+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:37.937062731+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:48:37.937065249+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:37.937067365+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-0
9-07T16:48:37.937069747+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:37.937071821+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:48:37.937074207+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:37.937076433+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:37.937078538+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:37.937080707+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:37.937090936+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[Nest] 7  - 09/07/2023, 4:48:38 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
    at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:38.048257787+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
    at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:38.048261662+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
    at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:38.048268787+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:38.048270735+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:38.048272606+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:38.048274870+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:38.185808219+02:00 [Nest] 7  - 09/07/2023, 4:48:38 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:38.185883567+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:38.185890507+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:48:38.185892978+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:38.185895142+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:48:38.185897145+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:38.185899122+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:48:38.185901158+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:38.185902987+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:38.185904874+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:38.185906913+02:00     at fulfilled (/usr/src/app/node_
modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:38.185908934+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

The OS that Immich Server is running on

Fedora k3s

Version of Immich Server

v1.76.1

Version of Immich Mobile App

non used

Platform with the issue

  • Server
  • Web
  • Mobile

Your docker-compose.yml content

i use k3s

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: immich
  namespace: immich
spec:
  chart:
    spec:
      chart: immich
      interval: 5m
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: truecharts
        namespace: flux-system
      version: 8.0.8
  interval: 5m
  values:
    TZ: Europe/Paris

    operator:
      register: false
      verify:
        enabled: false
        failOnError: false
    containerOptions:
      NVIDIA_CAPS:
        - all


    # -- Allow specifying a runtimeClassName other than the default one (ie: nvidia)
    runtimeClassName: nvidia


    securityContext:
      container:
        PUID: 1000
        UMASK: '0022'
        runAsGroup: 100
        runAsUser: 1000
      pod:
        fsGroup: 100
        fsGroupChangePolicy: OnRootMismatch

    immich:
      enable_ml: true
      enable_typesense: true
      public_login_page_message: Immich
      disable_reverse_geocoding: false
      reverse_geocoding_precision: 3
      log_level: log

    persistence:
      library:
        enabled: true
        mountPath: /usr/src/app/upload/library
        hostPath: /mnt/XXXX/immich/library
        hostPathType: Directory
        type: hostPath
      uploads:
        enabled: true
        mountPath: /usr/src/app/upload/upload
        hostPath: /mnt/XXXX/immich/upload
        hostPathType: Directory
        type: hostPath
      thumbs:
        enabled: true
        mountPath: /usr/src/app/upload/thumbs
        hostPath: /mnt/XXXX/immich/thumbs
        hostPathType: Directory
        type: hostPath
      profile:
        enabled: true
        mountPath: /usr/src/app/upload/profile
        hostPath: /mnt/XXXX/immich/profile
        hostPathType: Directory
        type: hostPath
      video:
        enabled: true
        mountPath: /usr/src/app/upload/encoded-video
        hostPath: /mnt/XXXX/immich/encoded-video
        hostPathType: Directory
        type: hostPath
      originals-photos:
        enabled: true
        hostPath: /mnt/XXXXX/Photos Videos
        hostPathType: Directory
        mountPath: /mnt/XXXX/Photos Videos
        readOnly: true
        targetSelectAll: true
        type: hostPath

    ingress:
      main:
        enabled: true
        ingressClassName: traefik
        enableFixedMiddlewares: false
        hosts:
          - host: XXXX
            paths:
              - path: /
                pathType: Prefix
        tls:
          - hosts:
              - XXXX
            secretName: photos-tls-secret

Your .env content

none

Reproduction steps

1. i do a big importing of existing readonly galery
2. i some time latter i got this

Additional information

No response

Originally created by @yodatak on GitHub (Sep 7, 2023). Originally assigned to: @jrasm91 on GitHub. ### The bug in immich-microservices ``` 2023-09-07T16:46:34.760277344+02:00 [Nest] 8 - 09/07/2023, 4:46:34 PM ERROR [JobService] Unable to run job handler: TypeError: Cannot read properties of null (reading 'to') 2023-09-07T16:46:34.760442799+02:00 [Nest] 8 - 09/07/2023, 4:46:34 PM ERROR [JobService] TypeError: Cannot read properties of null (reading 'to') 2023-09-07T16:46:34.760450096+02:00 at CommunicationRepository.send (/usr/src/app/dist/infra/repositories/communication.repository.js:20:24) 2023-09-07T16:46:34.760453473+02:00 at JobService.onDone (/usr/src/app/dist/domain/job/job.service.js:164:50) 2023-09-07T16:46:34.760456347+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:46:34.760459978+02:00 at async /usr/src/app/dist/domain/job/job.service.js:107:25 2023-09-07T16:46:34.760462728+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-09-07T16:46:34.760465157+02:00 at async Work [Nest] 8 - 09/07/2023, 4:48:44 PM ERROR [JobService] Unable to run job handler: Error: Connection terminated due to connection timeout [Nest] 8 - 09/07/2023, 4:48:44 PM ERROR [JobService] Error: Connection terminated due to connection timeout 2023-09-07T16:48:44.470777854+02:00 at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) 2023-09-07T16:48:44.470781573+02:00 at Object.onceWrapper (node:events:628:28) 2023-09-07T16:48:44.470784056+02:00 at Connection.emit (node:events:514:28) 2023-09-07T16:48:44.470786605+02:00 at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) 2023-09-07T16:48:44.470789076+02:00 at Socket.emit (node:events:514:28) 2023-09-07T16:48:44.470791493+02:00 at TCP.<anonymous> (node:net:323:12) [Nest] 8 - 09/07/2023, 4:48:44 PM ERROR [JobService] Object: 2023-09-07T16:48:44.471332649+02:00 { 2023-09-07T16:48:44.471335870+02:00 "id": "1810065b-7f56-418b-ae57-3162ecdb3f33", 2023-09-07T16:48:44.471338484+02:00 "source": "upload" 2023-09-07T16:48:44.471341241+02:00 } 2023-09-07T16:48:44.471343509+02:00 [Nest] 8 - 09/07/2023, 4:48:45 PM ERROR [JobService] Unable to run job handler: Error: Connection terminated due to connection timeout 2023-09-07T16:48:45.551578848+02:00 [Nest] 8 - 09/07/2023, 4:48:45 PM ERROR [JobService] Error: Connection terminated due to connection timeout 2023-09-07T16:48:45.551586947+02:00 at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) 2023-09-07T16:48:45.551589740+02:00 at Object.onceWrapper (node:events:628:28) 2023-09-07T16:48:45.551592245+02:00 at Connection.emit (node:events:514:28) 2023-09-07T16:48:45.551594918+02:00 at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) 2023-09-07T16:48:45.551597387+02:00 at Socket.emit (node:events:514:28) 2023-09-07T16:48:45.551599834+02:00 at TCP.<anonymous> (node:net:323:12) 2023-09-07T16:48:45.551602460+02:00 [Nest] 8 - 09/07/2023, 4:48:45 PM ERROR [JobService] Object: 2023-09-07T16:48:45.551604977+02:00 { 2023-09-07T16:48:45.551607547+02:00 "id": "ef3ed216-dea7-429e-910f-35ec4eb9fd20", 2023-09-07T16:48:45.551610294+02:00 "source": "upload" 2023-09-07T16:48:45.551612738+02:00 } 2023-09-07T16:48:45.551615079+02:00 [Nest] 8 - 09/07/2023, 4:48:47 PM ERROR [JobService] Unable to run job handler: TypeError: fetch failed [Nest] 8 - 09/07/2023, 4:48:47 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11576:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22) at async /usr/src/app/dist/domain/job/job.service.js:105:37 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) [Nest] 8 - 09/07/2023, 4:48:47 PM ERROR [JobService] Object: { "id": "56e660e8-238a-4349-af42-2f75521648dc", "source": "upload" } [Nest] 8 - 09/07/2023, 4:48:47 PM ERROR [JobService] Unable to run job handler: TypeError: fetch failed [Nest] 8 - 09/07/2023, 4:48:47 PM ERROR [JobService] TypeError: fetch failed at Object.fetch (node:internal/deps/undici/undici:11576:11) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21) at async FacialRecognitionService.handleRecognizeFaces (/usr/src/app/dist/domain/facial-recognition/facial-recognition.services.js:105:23) 2023-09-07T16:48:47.267395051+02:00 at async /usr/src/app/dist/domain/job/job.service.js:105:37 2023-09-07T16:48:47.267405270+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-09-07T16:48:47.267429326+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) 2023-09-07T16:48:47.267439507+02:00 [Nest] 8 - 09/07/2023, 4:48:47 PM ERROR [JobService] Object: 2023-09-07T16:48:47.267448981+02:00 { 2023-09-07T16:48:47.267458765+02:00 "id": "1691264c-94dd-4d3f-b25d-ce29a91a2a1e", 2023-09-07T16:48:47.267468597+02:00 "source": "upload" 2023-09-07T16:48:47.267478003+02:00 } 2023-09-07T16:48:47.267486760+02:00 [Nest] 8 - 09/07/2023, 4:48:50 PM LOG [MediaService] Successfully generated image thumbnail 5b9612f8-b89d-4526-8cd0-17787a48866b [Nest] 8 - 09/07/2023, 4:48:50 PM ERROR [JobService] Unable to run job handler: TypeError: Cannot read properties of null (reading 'to') [Nest] 8 - 09/07/2023, 4:48:50 PM ERROR [JobService] TypeError: Cannot read properties of null (reading 'to') 2023-09-07T16:48:50.857432731+02:00 at CommunicationRepository.send (/usr/src/app/dist/infra/repositories/communication.repository.js:20:24) 2023-09-07T16:48:50.857436489+02:00 at JobService.onDone (/usr/src/app/dist/domain/job/job.service.js:164:50) 2023-09-07T16:48:50.857438951+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:48:50.857441504+02:00 at async /usr/src/app/dist/domain/job/job.service.js:107:25 2023-09-07T16:48:50.857443889+02:00 at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28) 2023-09-07T16:48:50.857446167+02:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24) ``` I also goterror in search tab in web 500 error in immich server ``` 2023-09-07T16:45:49.881940765+02:00 [Nest] 7 - 09/07/2023, 4:45:49 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:45:49.882261150+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:45:49.882270180+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:45:49.882274754+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:45:49.882277310+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:45:49.882279386+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:45:49.882281386+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:45:49.882283935+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:45:49.882285941+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:45:49.882288266+02:00 at Object.next (/usr/src/ap p/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:45:49.882290528+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:45:49.882292452+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:45:49.991596213+02:00 [Nest] 7 - 09/07/2023, 4:45:49 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:45:49.991687056+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:45:49.991693086+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:45:49.991695052+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:45:49.991696661+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:45:49.991698067+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:45:49.991699431+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:45:49.991701057+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:45:49.991702431+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [Nest] 7 - 09/07/2023, 4:45:55 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:45:55.015817553+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:45:55.015820320+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:45:55.015823095+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:45:55.015825871+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:45:55.015828015+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:45:55.015830852+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:45:55.015833377+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:45:55.015836364+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:45:55.015838592+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:45:55.015841214+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:46:00.458230458+02:00 [Nest] 7 - 09/07/2023, 4:46:00 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:46:00.458252527+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:46:00.458255335+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:46:00.458257644+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09- 07T16:46:00.458259687+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:46:00.458261842+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:46:00.458264165+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:46:00.458266391+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:46:00.458268392+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:46:00.458270328+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:46:00.458272424+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:46:00.458274496+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:46:12.218016163+02:00 [Nest] 7 - 09/07/2023, 4:46:12 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:46:12.218047522+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:46:12.218050928+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:46:12.218053146+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:46:12.218057615+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:46:12.218059798+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:46:12.218061658+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:46:12.218063630+02:00 at Object.next (/usr/src/app/node_ modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:46:12.218065509+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:46:12.218067335+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:46:12.218069140+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:46:12.218070975+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:46:12.599054790+02:00 [Nest] 7 - 09/07/2023, 4:46:12 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:46:12.599076786+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:46:12.599079762+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:46:12.599082027+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:46:12.599083892+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:46:12.599085793+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:46:12.599087629+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:46:12.599089901+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:46:12.599091948+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:46:12.599094087+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:46:12.599096188+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:46:12.59909 8065+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:48:35.701930921+02:00 [Nest] 7 - 09/07/2023, 4:48:35 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:48:35.701962834+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:48:35.701965546+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:48:35.701967771+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:48:35.701974384+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:48:35.701984479+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:48:35.701986839+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:48:35.701992892+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:35.701995201+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:48:35.701997417+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:35.701999483+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:48:35.702002124+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:48:37.783875036+02:00 [Nest] 7 - 09/07/2023, 4:48:37 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:48:37.783899772+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:48:37.783903284+ 02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:48:37.783906232+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:48:37.783910871+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:48:37.783913160+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:48:37.783915242+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:48:37.783917783+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:37.783920207+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:48:37.783922453+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:37.783924333+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:48:37.783926257+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:48:37.934863402+02:00 [Nest] 7 - 09/07/2023, 4:48:37 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:48:37.937045849+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:48:37.937062731+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:48:37.937065249+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:48:37.937067365+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-0 9-07T16:48:37.937069747+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:48:37.937071821+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:48:37.937074207+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:37.937076433+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:48:37.937078538+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:37.937080707+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:48:37.937090936+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [Nest] 7 - 09/07/2023, 4:48:38 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:48:38.048257787+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:48:38.048261662+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:38.048268787+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:48:38.048270735+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:38.048272606+02:00 at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:48:38.048274870+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-09-07T16:48:38.185808219+02:00 [Nest] 7 - 09/07/2023, 4:48:38 PM ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:48:38.185883567+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found. 2023-09-07T16:48:38.185890507+02:00 at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28) 2023-09-07T16:48:38.185892978+02:00 at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42) 2023-09-07T16:48:38.185895142+02:00 at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21) 2023-09-07T16:48:38.185897145+02:00 at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98 2023-09-07T16:48:38.185899122+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23) 2023-09-07T16:48:38.185901158+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:38.185902987+02:00 at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139) 2023-09-07T16:48:38.185904874+02:00 at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53) 2023-09-07T16:48:38.185906913+02:00 at fulfilled (/usr/src/app/node_ modules/typesense/lib/Typesense/ApiCall.js:5:58) 2023-09-07T16:48:38.185908934+02:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) ``` ### The OS that Immich Server is running on Fedora k3s ### Version of Immich Server v1.76.1 ### Version of Immich Mobile App non used ### Platform with the issue - [X] Server - [X] Web - [ ] Mobile ### Your docker-compose.yml content ```YAML i use k3s --- apiVersion: helm.toolkit.fluxcd.io/v2beta1 kind: HelmRelease metadata: name: immich namespace: immich spec: chart: spec: chart: immich interval: 5m reconcileStrategy: ChartVersion sourceRef: kind: HelmRepository name: truecharts namespace: flux-system version: 8.0.8 interval: 5m values: TZ: Europe/Paris operator: register: false verify: enabled: false failOnError: false containerOptions: NVIDIA_CAPS: - all # -- Allow specifying a runtimeClassName other than the default one (ie: nvidia) runtimeClassName: nvidia securityContext: container: PUID: 1000 UMASK: '0022' runAsGroup: 100 runAsUser: 1000 pod: fsGroup: 100 fsGroupChangePolicy: OnRootMismatch immich: enable_ml: true enable_typesense: true public_login_page_message: Immich disable_reverse_geocoding: false reverse_geocoding_precision: 3 log_level: log persistence: library: enabled: true mountPath: /usr/src/app/upload/library hostPath: /mnt/XXXX/immich/library hostPathType: Directory type: hostPath uploads: enabled: true mountPath: /usr/src/app/upload/upload hostPath: /mnt/XXXX/immich/upload hostPathType: Directory type: hostPath thumbs: enabled: true mountPath: /usr/src/app/upload/thumbs hostPath: /mnt/XXXX/immich/thumbs hostPathType: Directory type: hostPath profile: enabled: true mountPath: /usr/src/app/upload/profile hostPath: /mnt/XXXX/immich/profile hostPathType: Directory type: hostPath video: enabled: true mountPath: /usr/src/app/upload/encoded-video hostPath: /mnt/XXXX/immich/encoded-video hostPathType: Directory type: hostPath originals-photos: enabled: true hostPath: /mnt/XXXXX/Photos Videos hostPathType: Directory mountPath: /mnt/XXXX/Photos Videos readOnly: true targetSelectAll: true type: hostPath ingress: main: enabled: true ingressClassName: traefik enableFixedMiddlewares: false hosts: - host: XXXX paths: - path: / pathType: Prefix tls: - hosts: - XXXX secretName: photos-tls-secret ``` ### Your .env content ```Shell none ``` ### Reproduction steps ```bash 1. i do a big importing of existing readonly galery 2. i some time latter i got this ``` ### Additional information _No response_
OVERLORD commented 2026-02-05 01:13:55 +03:00
Author
Owner
Copy Link

@bo0tzz commented on GitHub (Sep 7, 2023):

What do the logs of your typesense container say? It seems like that might be overloaded.

@bo0tzz commented on GitHub (Sep 7, 2023): What do the logs of your typesense container say? It seems like that might be overloaded.
OVERLORD commented 2026-02-05 01:13:58 +03:00
Author
Owner
Copy Link

@yodatak commented on GitHub (Sep 7, 2023):

Thanks for help some more logs

{
  "job": {
    "thumbnailGeneration": {
      "concurrency": 8
    },
    "metadataExtraction": {
      "concurrency": 8
    },
    "objectTagging": {
      "concurrency": 4
    },
    "recognizeFaces": {
      "concurrency": 4
    },
    "storageTemplateMigration": {
      "concurrency": 8
    },
    "sidecar": {
      "concurrency": 8
    },
    "backgroundTask": {
      "concurrency": 8
    },
    "clipEncoding": {
      "concurrency": 2
    },
    "search": {
      "concurrency": 8
    },
    "videoConversion": {
      "concurrency": 2
    }
  },
  "ffmpeg": {
    "crf": 23,
    "threads": 0,
    "preset": "ultrafast",
    "targetVideoCodec": "h264",
    "targetAudioCodec": "aac",
    "targetResolution": "720",
    "maxBitrate": "0",
    "twoPass": false,
    "transcode": "required",
    "tonemap": "hable",
    "accel": "disabled"
  },
  "machineLearning": {
    "enabled": true,
    "url": "http://immich-machinelearning:10003",
    "classification": {
      "enabled": true,
      "modelName": "microsoft/resnet-50",
      "minScore": 0.9
    },
    "clip": {
      "enabled": true,
      "modelName": "ViT-B-32::openai"
    },
    "facialRecognition": {
      "enabled": true,
      "modelName": "buffalo_l",
      "minScore": 0.7,
      "maxDistance": 0.6
    }
  },
  "oauth": {
    "enabled": false,
    "issuerUrl": "",
    "clientId": "",
    "clientSecret": "",
    "mobileOverrideEnabled": false,
    "mobileRedirectUri": "",
    "scope": "openid email profile",
    "storageLabelClaim": "preferred_username",
    "buttonText": "Login with OAuth",
    "autoRegister": true,
    "autoLaunch": false
  },
  "passwordLogin": {
    "enabled": true
  },
  "storageTemplate": {
    "template": "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}"
  },
  "thumbnail": {
    "webpSize": 250,
    "jpegSize": 1440
  }
}

the log i got from immich import

node@immich-688d755b49-wr8jc:/usr/src/app$ immich upload --key XXXX --server XXX  "/mnt/XXXXX/XXXX" -y --recursive -t 10 --import
Checking connectivity with Immich instance...
Server status: OK
Checking credentials...
Login status: OK
Successful authentication for user XXXXX
Indexing local assets...
Indexing complete, found 327529 local assets
Comparing local assets with those on the Immich instance...
A total of 38675 assets will be uploaded to the server
Start uploading...
Upload Progress | █████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 11% || 38675/327529 || Current file [/mnt/XX 2/XXX.CR2]
Error asset:  {
  file: '/mnt/XXX/2022/XXXXX.mp4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"XXXXX.mp4-1933807498","deviceId":"CLI","fileCreatedAt":"2022-04-22T16:20:52.000Z","fileModifiedAt":"2022-04-22T16:20:52.000Z","isFavorite":"false","fileExtension":".mp4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/2022/Baptiste Solidaires Informatique Urkaine.mp4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 362,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 362\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/Montage/pizza pino/canon eos/MVI_0538.MOV',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"MVI_0538.MOV-2716295569","deviceId":"CLI","fileCreatedAt":"2018-06-02T05:21:26.000Z","fileModifiedAt":"2018-06-02T05:21:26.000Z","isFavorite":"false","fileExtension":".MOV","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/Montage/pizza pino/canon eos/MVI_0538.MOV"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 325,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 325\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/XXXX/C0012.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"C0012.MP4-3674651378","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:12:16.000Z","fileModifiedAt":"2021-07-15T12:12:16.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0012.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/XXXX/C0013.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"C0013.MP4-2936365794","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:22:20.000Z","fileModifiedAt":"2021-07-15T12:22:20.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0013.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/XXXX/C0014.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"C0014.MP4-2978313974","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:31:35.000Z","fileModifiedAt":"2021-07-15T12:31:35.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0014.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/XXXX/C0015.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"C0015.MP4-2646923586","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:35:37.000Z","fileModifiedAt":"2021-07-15T12:35:37.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0015.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}

and typesence

I20230907 16:29:49.203409   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:29:49.203681   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213184129+02:00 I20230907 16:29:52.136260   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:31:59.213185598+02:00 I20230907 16:29:59.204329   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:29:59.204372   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:09.205035   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213189908+02:00 I20230907 16:30:09.205098   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:19.205725   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213192823+02:00 I20230907 16:30:19.205758   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:29.206576   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:30:29.206611   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:39.207240   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213198554+02:00 I20230907 16:30:39.207276   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213199977+02:00 I20230907 16:30:49.207926   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213201404+02:00 I20230907 16:30:49.207965   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:53.143424   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:31:59.213209880+02:00 I20230907 16:30:59.208699   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:30:59.208735   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:31:09.209383   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:31:09.209440   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:31:19.210069   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:31:19.210103   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:31:29.210781   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:31:29.210815   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213221532+02:00 I20230907 16:31:39.211509   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213222980+02:00 I20230907 16:31:39.211549   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213224400+02:00 I20230907 16:31:49.212189   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213225833+02:00 I20230907 16:31:49.212226   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213231839+02:00 I20230907 16:31:54.147421   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:31:59.212867   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:31:59.213135   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:09.213796   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:34:09.223465706+02:00 I20230907 16:32:09.213843   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:19.214514   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:32:19.214550   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:34:09.223471535+02:00 I20230907 16:32:29.215185   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:32:29.215224   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:39.215909   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:32:39.215966   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:49.216591   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:34:09.223486831+02:00 I20230907 16:32:49.216634   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:55.153040   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:34:09.223490478+02:00 I20230907 16:32:59.217676   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:32:59.217723   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:33:09.218418   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:09.218482   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:34:09.223497465+02:00 I20230907 16:33:19.219158   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:19.219189   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:33:29.219866   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:34:09.223506681+02:00 I20230907 16:33:29.219909   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:33:39.220940   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:39.220988   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:34:09.223511900+02:00 I20230907 16:33:49.221670   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:49.221740   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:34:09.223515383+02:00 I20230907 16:33:56.156810   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:33:59.222398   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:59.222434   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266247834+02:00 I20230907 16:34:09.223111   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:09.223410   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:19.224054   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:19.224094   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:29.224741   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:29.224776   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:39.225435   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266601816+02:00 I20230907 16:34:39.225471   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:49.226186   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:49.226222   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:57.163491   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:34:59.227104   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:59.227280   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:35:09.244202   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266663543+02:00 I20230907 16:35:09.245168   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:35:19.246631   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:35:19.246680   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:35:29.249236   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:35:29.249329   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266704149+02:00 I20230907 16:35:39.253510   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266712299+02:00 I20230907 16:35:39.253561   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266720864+02:00 I20230907 16:35:49.256667   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266731455+02:00 I20230907 16:35:49.256719   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266741910+02:00 I20230907 16:35:58.187074   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:36:19.266759612+02:00 I20230907 16:35:59.260445   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266770748+02:00 I20230907 16:35:59.260546   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266781059+02:00 I20230907 16:36:09.263536   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266791757+02:00 I20230907 16:36:09.263577   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318818337+02:00 I20230907 16:36:19.265493   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318836701+02:00 I20230907 16:36:19.266094   299 raft_ser
ver.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318839551+02:00 I20230907 16:36:29.271066   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318841793+02:00 I20230907 16:36:29.271200   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318843941+02:00 I20230907 16:36:39.279420   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318846077+02:00 I20230907 16:36:39.279484   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318848339+02:00 I20230907 16:36:49.290627   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318850467+02:00 I20230907 16:36:49.290772   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318852808+02:00 I20230907 16:36:59.197636   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:38:29.318857698+02:00 I20230907 16:36:59.297367   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318859887+02:00 I20230907 16:36:59.297494   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318862011+02:00 I20230907 16:37:09.298318   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318865014+02:00 I20230907 16:37:09.298417   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318867987+02:00 I20230
907 16:37:19.301503   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318870852+02:00 I20230907 16:37:19.301607   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318873536+02:00 I20230907 16:37:29.304710   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318876457+02:00 I20230907 16:37:29.304750   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318879261+02:00 I20230907 16:37:39.309211   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318887078+02:00 I20230907 16:37:39.309265   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318890386+02:00 I20230907 16:37:49.312556   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318892575+02:00 I20230907 16:37:49.312669   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318894697+02:00 I20230907 16:37:59.313561   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318896859+02:00 I20230907 16:37:59.313596   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318900790+02:00 I20230907 16:38:00.220654   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:38:29.318902972+02:00 I20230907 16:38:09.314311   262 raft_server.cpp:564] Term: 2, last_index index: 14
48, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318905118+02:00 I20230907 16:38:09.314344   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318907229+02:00 I20230907 16:38:19.315969   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318909397+02:00 I20230907 16:38:19.316007   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318923037+02:00 I20230907 16:38:29.318449   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337822037+02:00 I20230907 16:38:29.318509   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337868825+02:00 I20230907 16:38:39.319967   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337872685+02:00 I20230907 16:38:39.320060   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337875300+02:00 I20230907 16:38:49.321424   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337878042+02:00 I20230907 16:38:49.321511   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337880540+02:00 I20230907 16:38:59.322511   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337883079+02:00 I20230907 16:38:59.322556   299 raft_server.
h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337886215+02:00 I20230907 16:39:01.241528   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:40:49.337890293+02:00 I20230907 16:39:09.324499   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337894400+02:00 I20230907 16:39:09.324645   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337904709+02:00 I20230907 16:39:19.325690   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337909459+02:00 I20230907 16:39:19.325826   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337912058+02:00 I20230907 16:39:29.326601   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337914554+02:00 I20230907 16:39:29.326664   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337917030+02:00 I20230907 16:39:39.327740   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337919697+02:00 I20230907 16:39:39.327777   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337922183+02:00 I20230907 16:39:49.328608   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337924687+02:00 I20230907 16:39:49.328646   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337927182+02:00 I20230907 
16:39:59.329387   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337929678+02:00 I20230907 16:39:59.329454   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337932170+02:00 I20230907 16:40:02.255044   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:40:49.337934697+02:00 I20230907 16:40:09.330129   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337937468+02:00 I20230907 16:40:09.330201   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337941170+02:00 I20230907 16:40:19.332188   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337945343+02:00 I20230907 16:40:19.332222   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337949409+02:00 I20230907 16:40:29.333328   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337953706+02:00 I20230907 16:40:29.333364   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337957895+02:00 I20230907 16:40:39.335991   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337975805+02:00 I20230907 16:40:39.336083   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377697696+02:00 I20230907 16:40:49.337464   262 raft_server.cpp:564] Term: 2, last_index index: 1448, 
committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377714719+02:00 I20230907 16:40:49.337800   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377717017+02:00 I20230907 16:40:59.342478   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377725003+02:00 I20230907 16:40:59.342556   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377727057+02:00 I20230907 16:41:03.264240   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:42:59.377728802+02:00 I20230907 16:41:09.343191   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377730522+02:00 I20230907 16:41:09.343227   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377732455+02:00 I20230907 16:41:19.345552   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377734324+02:00 I20230907 16:41:19.345598   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377736057+02:00 I20230907 16:41:29.346921   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377737773+02:00 I20230907 16:41:29.347064   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377739471+02:00 I20230907 16:41:39.351914   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_
writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377741188+02:00 I20230907 16:41:39.352062   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377742893+02:00 I20230907 16:41:49.352993   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377744614+02:00 I20230907 16:41:49.353029   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377746320+02:00 I20230907 16:41:59.359923   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377748089+02:00 I20230907 16:41:59.359957   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377749813+02:00 I20230907 16:42:04.283854   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:42:59.377751534+02:00 I20230907 16:42:09.364686   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377753936+02:00 I20230907 16:42:09.365665   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377756294+02:00 I20230907 16:42:19.368320   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377758467+02:00 I20230907 16:42:19.368355   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377760745+02:00 I20230907 16:42:29.369127   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:5
9.377763059+02:00 I20230907 16:42:29.369194   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377765245+02:00 I20230907 16:42:39.370123   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377768449+02:00 I20230907 16:42:39.370209   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377770181+02:00 I20230907 16:42:49.370857   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377771904+02:00 I20230907 16:42:49.370903   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406766861+02:00 I20230907 16:42:59.377444   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406826922+02:00 I20230907 16:42:59.377674   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406847676+02:00 I20230907 16:43:05.301301   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:45:09.406856688+02:00 I20230907 16:43:09.380699   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406864071+02:00 I20230907 16:43:09.380733   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406871361+02:00 I20230907 16:43:19.381362   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406878624+02:00 I20230907 16:43:19.381409   299 raft_server.h:60] Peer ref
resh succeeded!
2023-09-07T18:45:09.406886953+02:00 I20230907 16:43:29.384903   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406896817+02:00 I20230907 16:43:29.384936   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406906368+02:00 I20230907 16:43:39.385644   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406920303+02:00 I20230907 16:43:39.385673   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406930428+02:00 I20230907 16:43:49.386977   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406940005+02:00 I20230907 16:43:49.387043   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406950790+02:00 I20230907 16:43:59.388796   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406958537+02:00 I20230907 16:43:59.388839   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406966513+02:00 I20230907 16:44:06.319248   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:45:09.406973859+02:00 I20230907 16:44:09.390421   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406981215+02:00 I20230907 16:44:09.390461   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406988674+02:00 I20230907 16:44:19.39280
2   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407007241+02:00 I20230907 16:44:19.392920   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407014721+02:00 I20230907 16:44:29.394214   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407022017+02:00 I20230907 16:44:29.394250   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407029277+02:00 I20230907 16:44:39.396353   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407036554+02:00 I20230907 16:44:39.396409   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407043846+02:00 I20230907 16:44:49.398945   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407051136+02:00 I20230907 16:44:49.398979   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407058499+02:00 I20230907 16:44:59.400401   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407065746+02:00 I20230907 16:44:59.400434   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407097088+02:00 I20230907 16:45:07.340353   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:47:19.445969365+02:00 I20230907 16:45:09.405902   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_inde
x: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.445990304+02:00 I20230907 16:45:09.406706   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:47:19.445992623+02:00 I20230907 16:45:19.409269   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.445994606+02:00 I20230907 16:45:19.409313   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:47:19.445996605+02:00 I20230907 16:45:29.412350   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.445998353+02:00 I20230907 16:45:29.412381   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:47:19.446000078+02:00 I20230907 16:45:39.413200   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.446001818+02:00 I20230907 16:45:39.413240   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:45:49.414548   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:45:49.414584   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:47:19.446007029+02:00 I20230907 16:45:59.415594   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.446013002+02:00 I20230907 16:45:59.415688   292 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:08.349647   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:46:09.416549   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:46:09.416652   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:19.420742   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.446024520+02:00 I20230907 16:46:19.420869   292 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:29.422032   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.446029833+02:00 I20230907 16:46:29.422075   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:39.426769   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:46:39.426831   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:49.432679   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:46:49.432727   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:59.436492   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:46:59.436576   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:47:09.367693   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:47:09.444370   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:47:09.444492   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472058114+02:00 I20230907 16:47:19.445700   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:49:29.472104443+02:00 I20230907 16:47:19.445942   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472110891+02:00 I20230907 16:47:29.447484   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:47:29.447587   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472123594+02:00 I20230907 16:47:39.449258   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:49:29.472144613+02:00 I20230907 16:47:39.449947   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:47:49.451269   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:47:49.451321   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472205154+02:00 I20230907 16:47:59.452847   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:49:29.472209676+02:00 I20230907 16:47:59.452962   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472214101+02:00 I20230907 16:48:09.455587   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:49:29.472218838+02:00 I20230907 16:48:09.456355   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:48:10.378072   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:48:19.459204   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:19.459300   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472237254+02:00 I20230907 16:48:29.463717   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:29.463759   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:48:39.464480   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:39.464511   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:48:49.465201   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:49.465250   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:48:59.466076   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:59.466217   305 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472274026+02:00 I20230907 16:49:09.466974   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:49:09.467108   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472282979+02:00 I20230907 16:49:11.394212   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:49:19.470341   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:49:19.470397   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:49:29.471253   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506833626+02:00 I20230907 16:49:29.471889   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:49:39.472606   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506838513+02:00 I20230907 16:49:39.472642   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:49:49.473484   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506842750+02:00 I20230907 16:49:49.473608   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:49:59.478327   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506847253+02:00 I20230907 16:49:59.478507   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506849308+02:00 I20230907 16:50:09.479187   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:50:09.479246   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:50:12.402351   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:51:39.506855669+02:00 I20230907 16:50:19.480391   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506857777+02:00 I20230907 16:50:19.480535   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:50:29.481473   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506861868+02:00 I20230907 16:50:29.481530   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:50:39.482787   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506866126+02:00 I20230907 16:50:39.482834   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506868371+02:00 I20230907 16:50:49.485591   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:50:49.485635   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506872714+02:00 I20230907 16:50:59.490762   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506879323+02:00 I20230907 16:50:59.491313   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506881606+02:00 I20230907 16:51:09.503599   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506883677+02:00 I20230907 16:51:09.503719   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506885957+02:00 I20230907 16:51:13.412426   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:51:39.506888093+02:00 I20230907 16:51:19.504729   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506890200+02:00 I20230907 16:51:19.504766   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506892458+02:00 I20230907 16:51:29.505527   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506894517+02:00 I20230907 16:51:29.505563   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506902531+02:00 I20230907 16:51:39.506449   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:51:39.506489   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519409613+02:00 I20230907 16:51:49.507345   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519418209+02:00 I20230907 16:51:49.507408   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519425139+02:00 I20230907 16:51:59.508363   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519432892+02:00 I20230907 16:51:59.508404   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519440954+02:00 I20230907 16:52:09.509209   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519448509+02:00 I20230907 16:52:09.509264   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:52:14.417016   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:53:49.519462411+02:00 I20230907 16:52:19.510066   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:52:19.510102   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:52:29.511366   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:52:29.511407   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:52:39.512140   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:52:39.512214   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:52:49.513013   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:52:49.513049   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519520164+02:00 I20230907 16:52:59.513851   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519529132+02:00 I20230907 16:52:59.513895   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519538057+02:00 I20230907 16:53:09.514724   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519543827+02:00 I20230907 16:53:09.514762   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519549488+02:00 I20230907 16:53:15.422190   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:53:49.519555152+02:00 I20230907 16:53:19.515636   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519561307+02:00 I20230907 16:53:19.515671   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:53:29.516753   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519577233+02:00 I20230907 16:53:29.516788   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519584966+02:00 I20230907 16:53:39.517629   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:53:39.517669   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:53:49.518606   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532519966+02:00 I20230907 16:53:49.518702   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532535674+02:00 I20230907 16:53:59.519462   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:53:59.519497   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:54:09.520290   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532562445+02:00 I20230907 16:54:09.520351   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532567609+02:00 I20230907 16:54:16.427273   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:56:09.532570241+02:00 I20230907 16:54:19.522101   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532579192+02:00 I20230907 16:54:19.522168   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532581307+02:00 I20230907 16:54:29.522970   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532583035+02:00 I20230907 16:54:29.523031   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:54:39.523911   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532586577+02:00 I20230907 16:54:39.523945   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532588307+02:00 I20230907 16:54:49.524821   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532590030+02:00 I20230907 16:54:49.524860   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:54:59.525784   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:54:59.525847   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532595385+02:00 I20230907 16:55:09.526659   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532597092+02:00 I20230907 16:55:09.526722   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:55:17.432358   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:56:09.532600564+02:00 I20230907 16:55:19.527577   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532602275+02:00 I20230907 16:55:19.527614   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:55:29.528379   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532605705+02:00 I20230907 16:55:29.528414   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:55:39.529318   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532609135+02:00 I20230907 16:55:39.529350   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532610944+02:00 I20230907 16:55:49.530324   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:55:49.530357   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532614423+02:00 I20230907 16:55:59.531132   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:55:59.531173   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563906133+02:00 I20230907 16:56:09.532176   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:56:09.532493   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:56:18.441802   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:56:19.533317   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:56:19.533390   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:56:29.534129   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563940058+02:00 I20230907 16:56:29.534178   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:56:39.542182   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563944291+02:00 I20230907 16:56:39.544174   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:56:49.550580   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563948527+02:00 I20230907 16:56:49.550616   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563950653+02:00 I20230907 16:56:59.553225   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563952761+02:00 I20230907 16:56:59.553280   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563954878+02:00 I20230907 16:57:09.554186   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:57:09.554231   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563959081+02:00 I20230907 16:57:19.452324   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:57:19.557760   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563963373+02:00 I20230907 16:57:19.557806   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:57:29.558519   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563967681+02:00 I20230907 16:57:29.558558   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563969994+02:00 I20230907 16:57:39.559978   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:57:39.560016   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563978480+02:00 I20230907 16:57:49.561477   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563980603+02:00 I20230907 16:57:49.561517   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563982731+02:00 I20230907 16:57:59.562211   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:57:59.562254   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563986932+02:00 I20230907 16:58:09.562929   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563989164+02:00 I20230907 16:58:09.562968   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:58:19.563656   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580843277+02:00 I20230907 16:58:19.563899   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:58:20.459970   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T19:00:29.580854976+02:00 I20230907 16:58:29.564522   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:58:29.564555   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:58:39.565266   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:58:39.565328   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580867508+02:00 I20230907 16:58:49.566066   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580870828+02:00 I20230907 16:58:49.566104   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:58:59.566893   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580876885+02:00 I20230907 16:58:59.566926   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:09.567569   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580883090+02:00 I20230907 16:59:09.567603   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:19.570607   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:59:19.570647   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:21.463987   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:59:29.571308   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580904231+02:00 I20230907 16:59:29.571343   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:39.572181   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580910195+02:00 I20230907 16:59:39.572216   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580913147+02:00 I20230907 16:59:49.573062   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:59:49.573096   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:59.574018   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580922240+02:00 I20230907 16:59:59.574054   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580925305+02:00 I20230907 17:00:09.578420   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 17:00:09.578545   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580931264+02:00 I20230907 17:00:19.579238   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 17:00:19.579300   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580951212+02:00 I20230907 17:00:22.468595   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
@yodatak commented on GitHub (Sep 7, 2023): Thanks for help some more logs ```json { "job": { "thumbnailGeneration": { "concurrency": 8 }, "metadataExtraction": { "concurrency": 8 }, "objectTagging": { "concurrency": 4 }, "recognizeFaces": { "concurrency": 4 }, "storageTemplateMigration": { "concurrency": 8 }, "sidecar": { "concurrency": 8 }, "backgroundTask": { "concurrency": 8 }, "clipEncoding": { "concurrency": 2 }, "search": { "concurrency": 8 }, "videoConversion": { "concurrency": 2 } }, "ffmpeg": { "crf": 23, "threads": 0, "preset": "ultrafast", "targetVideoCodec": "h264", "targetAudioCodec": "aac", "targetResolution": "720", "maxBitrate": "0", "twoPass": false, "transcode": "required", "tonemap": "hable", "accel": "disabled" }, "machineLearning": { "enabled": true, "url": "http://immich-machinelearning:10003", "classification": { "enabled": true, "modelName": "microsoft/resnet-50", "minScore": 0.9 }, "clip": { "enabled": true, "modelName": "ViT-B-32::openai" }, "facialRecognition": { "enabled": true, "modelName": "buffalo_l", "minScore": 0.7, "maxDistance": 0.6 } }, "oauth": { "enabled": false, "issuerUrl": "", "clientId": "", "clientSecret": "", "mobileOverrideEnabled": false, "mobileRedirectUri": "", "scope": "openid email profile", "storageLabelClaim": "preferred_username", "buttonText": "Login with OAuth", "autoRegister": true, "autoLaunch": false }, "passwordLogin": { "enabled": true }, "storageTemplate": { "template": "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}" }, "thumbnail": { "webpSize": 250, "jpegSize": 1440 } } ``` the log i got from immich import ``` node@immich-688d755b49-wr8jc:/usr/src/app$ immich upload --key XXXX --server XXX "/mnt/XXXXX/XXXX" -y --recursive -t 10 --import Checking connectivity with Immich instance... Server status: OK Checking credentials... Login status: OK Successful authentication for user XXXXX Indexing local assets... Indexing complete, found 327529 local assets Comparing local assets with those on the Immich instance... A total of 38675 assets will be uploaded to the server Start uploading... Upload Progress | █████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 11% || 38675/327529 || Current file [/mnt/XX 2/XXX.CR2] Error asset: { file: '/mnt/XXX/2022/XXXXX.mp4', reason: Error: Request failed with status code 504 at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15) at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11) at IncomingMessage.emit (node:events:526:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: Infinity, maxBodyLength: Infinity, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', maxRedirects: 0, url: 'https://immich.XXX/api/asset/import', data: '{"deviceAssetId":"XXXXX.mp4-1933807498","deviceId":"CLI","fileCreatedAt":"2022-04-22T16:20:52.000Z","fileModifiedAt":"2022-04-22T16:20:52.000Z","isFavorite":"false","fileExtension":".mp4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/2022/Baptiste Solidaires Informatique Urkaine.mp4"}' }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 362, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: true, socket: [TLSSocket], _header: 'POST /api/asset/import HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' + 'User-Agent: axios/0.26.1\r\n' + 'Content-Length: 362\r\n' + 'Host: XXXX.immich.com\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/api/asset/import', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'XXXX.immich.com', protocol: 'https:', [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(errored)]: null, [Symbol(kHighWaterMark)]: 16384, [Symbol(kRejectNonStandardBodyWrites)]: false, [Symbol(kUniqueHeaders)]: null }, response: { status: 504, statusText: 'Gateway Timeout', headers: [Object], config: [Object], request: [ClientRequest], data: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' }, isAxiosError: true, toJSON: [Function: toJSON] }, response: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' } Error asset: { file: '/mnt/XXX/Montage/pizza pino/canon eos/MVI_0538.MOV', reason: Error: Request failed with status code 504 at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15) at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11) at IncomingMessage.emit (node:events:526:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: Infinity, maxBodyLength: Infinity, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', maxRedirects: 0, url: 'https://immich.XXX/api/asset/import', data: '{"deviceAssetId":"MVI_0538.MOV-2716295569","deviceId":"CLI","fileCreatedAt":"2018-06-02T05:21:26.000Z","fileModifiedAt":"2018-06-02T05:21:26.000Z","isFavorite":"false","fileExtension":".MOV","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/Montage/pizza pino/canon eos/MVI_0538.MOV"}' }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 325, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: true, socket: [TLSSocket], _header: 'POST /api/asset/import HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' + 'User-Agent: axios/0.26.1\r\n' + 'Content-Length: 325\r\n' + 'Host: XXXX.immich.com\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/api/asset/import', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'XXXX.immich.com', protocol: 'https:', [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(errored)]: null, [Symbol(kHighWaterMark)]: 16384, [Symbol(kRejectNonStandardBodyWrites)]: false, [Symbol(kUniqueHeaders)]: null }, response: { status: 504, statusText: 'Gateway Timeout', headers: [Object], config: [Object], request: [ClientRequest], data: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' }, isAxiosError: true, toJSON: [Function: toJSON] }, response: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' } Error asset: { file: '/mnt/XXX/XXXX/C0012.MP4', reason: Error: Request failed with status code 504 at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15) at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11) at IncomingMessage.emit (node:events:526:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: Infinity, maxBodyLength: Infinity, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', maxRedirects: 0, url: 'https://immich.XXX/api/asset/import', data: '{"deviceAssetId":"C0012.MP4-3674651378","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:12:16.000Z","fileModifiedAt":"2021-07-15T12:12:16.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0012.MP4"}' }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 322, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: true, socket: [TLSSocket], _header: 'POST /api/asset/import HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' + 'User-Agent: axios/0.26.1\r\n' + 'Content-Length: 322\r\n' + 'Host: XXXX.immich.com\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/api/asset/import', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'XXXX.immich.com', protocol: 'https:', [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(errored)]: null, [Symbol(kHighWaterMark)]: 16384, [Symbol(kRejectNonStandardBodyWrites)]: false, [Symbol(kUniqueHeaders)]: null }, response: { status: 504, statusText: 'Gateway Timeout', headers: [Object], config: [Object], request: [ClientRequest], data: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' }, isAxiosError: true, toJSON: [Function: toJSON] }, response: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' } Error asset: { file: '/mnt/XXX/XXXX/C0013.MP4', reason: Error: Request failed with status code 504 at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15) at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11) at IncomingMessage.emit (node:events:526:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: Infinity, maxBodyLength: Infinity, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', maxRedirects: 0, url: 'https://immich.XXX/api/asset/import', data: '{"deviceAssetId":"C0013.MP4-2936365794","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:22:20.000Z","fileModifiedAt":"2021-07-15T12:22:20.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0013.MP4"}' }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 322, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: true, socket: [TLSSocket], _header: 'POST /api/asset/import HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' + 'User-Agent: axios/0.26.1\r\n' + 'Content-Length: 322\r\n' + 'Host: XXXX.immich.com\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/api/asset/import', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'XXXX.immich.com', protocol: 'https:', [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(errored)]: null, [Symbol(kHighWaterMark)]: 16384, [Symbol(kRejectNonStandardBodyWrites)]: false, [Symbol(kUniqueHeaders)]: null }, response: { status: 504, statusText: 'Gateway Timeout', headers: [Object], config: [Object], request: [ClientRequest], data: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' }, isAxiosError: true, toJSON: [Function: toJSON] }, response: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' } Error asset: { file: '/mnt/XXX/XXXX/C0014.MP4', reason: Error: Request failed with status code 504 at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15) at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11) at IncomingMessage.emit (node:events:526:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: Infinity, maxBodyLength: Infinity, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', maxRedirects: 0, url: 'https://immich.XXX/api/asset/import', data: '{"deviceAssetId":"C0014.MP4-2978313974","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:31:35.000Z","fileModifiedAt":"2021-07-15T12:31:35.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0014.MP4"}' }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 322, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: true, socket: [TLSSocket], _header: 'POST /api/asset/import HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' + 'User-Agent: axios/0.26.1\r\n' + 'Content-Length: 322\r\n' + 'Host: XXXX.immich.com\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/api/asset/import', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'XXXX.immich.com', protocol: 'https:', [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(errored)]: null, [Symbol(kHighWaterMark)]: 16384, [Symbol(kRejectNonStandardBodyWrites)]: false, [Symbol(kUniqueHeaders)]: null }, response: { status: 504, statusText: 'Gateway Timeout', headers: [Object], config: [Object], request: [ClientRequest], data: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' }, isAxiosError: true, toJSON: [Function: toJSON] }, response: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' } Error asset: { file: '/mnt/XXX/XXXX/C0015.MP4', reason: Error: Request failed with status code 504 at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15) at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11) at IncomingMessage.emit (node:events:526:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: Infinity, maxBodyLength: Infinity, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', maxRedirects: 0, url: 'https://immich.XXX/api/asset/import', data: '{"deviceAssetId":"C0015.MP4-2646923586","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:35:37.000Z","fileModifiedAt":"2021-07-15T12:35:37.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0015.MP4"}' }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 322, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: true, socket: [TLSSocket], _header: 'POST /api/asset/import HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' + 'User-Agent: axios/0.26.1\r\n' + 'Content-Length: 322\r\n' + 'Host: XXXX.immich.com\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/api/asset/import', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'XXXX.immich.com', protocol: 'https:', [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(errored)]: null, [Symbol(kHighWaterMark)]: 16384, [Symbol(kRejectNonStandardBodyWrites)]: false, [Symbol(kUniqueHeaders)]: null }, response: { status: 504, statusText: 'Gateway Timeout', headers: [Object], config: [Object], request: [ClientRequest], data: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' }, isAxiosError: true, toJSON: [Function: toJSON] }, response: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' } ``` and typesence ``` I20230907 16:29:49.203409 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:29:49.203681 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:31:59.213184129+02:00 I20230907 16:29:52.136260 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:31:59.213185598+02:00 I20230907 16:29:59.204329 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:29:59.204372 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:30:09.205035 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:31:59.213189908+02:00 I20230907 16:30:09.205098 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:30:19.205725 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:31:59.213192823+02:00 I20230907 16:30:19.205758 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:30:29.206576 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:30:29.206611 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:30:39.207240 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:31:59.213198554+02:00 I20230907 16:30:39.207276 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:31:59.213199977+02:00 I20230907 16:30:49.207926 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:31:59.213201404+02:00 I20230907 16:30:49.207965 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:30:53.143424 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:31:59.213209880+02:00 I20230907 16:30:59.208699 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:30:59.208735 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:31:09.209383 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:31:09.209440 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:31:19.210069 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:31:19.210103 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:31:29.210781 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:31:29.210815 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:31:59.213221532+02:00 I20230907 16:31:39.211509 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:31:59.213222980+02:00 I20230907 16:31:39.211549 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:31:59.213224400+02:00 I20230907 16:31:49.212189 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:31:59.213225833+02:00 I20230907 16:31:49.212226 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:31:59.213231839+02:00 I20230907 16:31:54.147421 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:31:59.212867 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:31:59.213135 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:32:09.213796 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:34:09.223465706+02:00 I20230907 16:32:09.213843 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:32:19.214514 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:32:19.214550 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:34:09.223471535+02:00 I20230907 16:32:29.215185 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:32:29.215224 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:32:39.215909 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:32:39.215966 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:32:49.216591 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:34:09.223486831+02:00 I20230907 16:32:49.216634 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:32:55.153040 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:34:09.223490478+02:00 I20230907 16:32:59.217676 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:32:59.217723 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:33:09.218418 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:33:09.218482 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:34:09.223497465+02:00 I20230907 16:33:19.219158 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:33:19.219189 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:33:29.219866 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:34:09.223506681+02:00 I20230907 16:33:29.219909 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:33:39.220940 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:33:39.220988 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:34:09.223511900+02:00 I20230907 16:33:49.221670 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:33:49.221740 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:34:09.223515383+02:00 I20230907 16:33:56.156810 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:33:59.222398 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:33:59.222434 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:36:19.266247834+02:00 I20230907 16:34:09.223111 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:34:09.223410 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:34:19.224054 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:34:19.224094 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:34:29.224741 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:34:29.224776 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:34:39.225435 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:36:19.266601816+02:00 I20230907 16:34:39.225471 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:34:49.226186 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:34:49.226222 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:34:57.163491 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:34:59.227104 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:34:59.227280 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:35:09.244202 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:36:19.266663543+02:00 I20230907 16:35:09.245168 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:35:19.246631 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:35:19.246680 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:35:29.249236 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:35:29.249329 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:36:19.266704149+02:00 I20230907 16:35:39.253510 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:36:19.266712299+02:00 I20230907 16:35:39.253561 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:36:19.266720864+02:00 I20230907 16:35:49.256667 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:36:19.266731455+02:00 I20230907 16:35:49.256719 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:36:19.266741910+02:00 I20230907 16:35:58.187074 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:36:19.266759612+02:00 I20230907 16:35:59.260445 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:36:19.266770748+02:00 I20230907 16:35:59.260546 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:36:19.266781059+02:00 I20230907 16:36:09.263536 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:36:19.266791757+02:00 I20230907 16:36:09.263577 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318818337+02:00 I20230907 16:36:19.265493 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318836701+02:00 I20230907 16:36:19.266094 299 raft_ser ver.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318839551+02:00 I20230907 16:36:29.271066 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318841793+02:00 I20230907 16:36:29.271200 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318843941+02:00 I20230907 16:36:39.279420 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318846077+02:00 I20230907 16:36:39.279484 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318848339+02:00 I20230907 16:36:49.290627 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318850467+02:00 I20230907 16:36:49.290772 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318852808+02:00 I20230907 16:36:59.197636 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:38:29.318857698+02:00 I20230907 16:36:59.297367 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318859887+02:00 I20230907 16:36:59.297494 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318862011+02:00 I20230907 16:37:09.298318 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318865014+02:00 I20230907 16:37:09.298417 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318867987+02:00 I20230 907 16:37:19.301503 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318870852+02:00 I20230907 16:37:19.301607 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318873536+02:00 I20230907 16:37:29.304710 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318876457+02:00 I20230907 16:37:29.304750 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318879261+02:00 I20230907 16:37:39.309211 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318887078+02:00 I20230907 16:37:39.309265 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318890386+02:00 I20230907 16:37:49.312556 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318892575+02:00 I20230907 16:37:49.312669 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318894697+02:00 I20230907 16:37:59.313561 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318896859+02:00 I20230907 16:37:59.313596 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318900790+02:00 I20230907 16:38:00.220654 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:38:29.318902972+02:00 I20230907 16:38:09.314311 262 raft_server.cpp:564] Term: 2, last_index index: 14 48, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318905118+02:00 I20230907 16:38:09.314344 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318907229+02:00 I20230907 16:38:19.315969 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:38:29.318909397+02:00 I20230907 16:38:19.316007 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:38:29.318923037+02:00 I20230907 16:38:29.318449 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337822037+02:00 I20230907 16:38:29.318509 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337868825+02:00 I20230907 16:38:39.319967 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337872685+02:00 I20230907 16:38:39.320060 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337875300+02:00 I20230907 16:38:49.321424 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337878042+02:00 I20230907 16:38:49.321511 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337880540+02:00 I20230907 16:38:59.322511 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337883079+02:00 I20230907 16:38:59.322556 299 raft_server. h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337886215+02:00 I20230907 16:39:01.241528 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:40:49.337890293+02:00 I20230907 16:39:09.324499 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337894400+02:00 I20230907 16:39:09.324645 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337904709+02:00 I20230907 16:39:19.325690 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337909459+02:00 I20230907 16:39:19.325826 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337912058+02:00 I20230907 16:39:29.326601 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337914554+02:00 I20230907 16:39:29.326664 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337917030+02:00 I20230907 16:39:39.327740 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337919697+02:00 I20230907 16:39:39.327777 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337922183+02:00 I20230907 16:39:49.328608 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337924687+02:00 I20230907 16:39:49.328646 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337927182+02:00 I20230907 16:39:59.329387 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337929678+02:00 I20230907 16:39:59.329454 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337932170+02:00 I20230907 16:40:02.255044 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:40:49.337934697+02:00 I20230907 16:40:09.330129 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337937468+02:00 I20230907 16:40:09.330201 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337941170+02:00 I20230907 16:40:19.332188 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337945343+02:00 I20230907 16:40:19.332222 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337949409+02:00 I20230907 16:40:29.333328 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337953706+02:00 I20230907 16:40:29.333364 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:40:49.337957895+02:00 I20230907 16:40:39.335991 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:40:49.337975805+02:00 I20230907 16:40:39.336083 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377697696+02:00 I20230907 16:40:49.337464 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377714719+02:00 I20230907 16:40:49.337800 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377717017+02:00 I20230907 16:40:59.342478 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377725003+02:00 I20230907 16:40:59.342556 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377727057+02:00 I20230907 16:41:03.264240 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:42:59.377728802+02:00 I20230907 16:41:09.343191 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377730522+02:00 I20230907 16:41:09.343227 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377732455+02:00 I20230907 16:41:19.345552 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377734324+02:00 I20230907 16:41:19.345598 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377736057+02:00 I20230907 16:41:29.346921 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377737773+02:00 I20230907 16:41:29.347064 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377739471+02:00 I20230907 16:41:39.351914 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_ writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377741188+02:00 I20230907 16:41:39.352062 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377742893+02:00 I20230907 16:41:49.352993 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377744614+02:00 I20230907 16:41:49.353029 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377746320+02:00 I20230907 16:41:59.359923 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377748089+02:00 I20230907 16:41:59.359957 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377749813+02:00 I20230907 16:42:04.283854 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:42:59.377751534+02:00 I20230907 16:42:09.364686 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377753936+02:00 I20230907 16:42:09.365665 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377756294+02:00 I20230907 16:42:19.368320 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377758467+02:00 I20230907 16:42:19.368355 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377760745+02:00 I20230907 16:42:29.369127 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:5 9.377763059+02:00 I20230907 16:42:29.369194 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377765245+02:00 I20230907 16:42:39.370123 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377768449+02:00 I20230907 16:42:39.370209 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:42:59.377770181+02:00 I20230907 16:42:49.370857 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:42:59.377771904+02:00 I20230907 16:42:49.370903 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.406766861+02:00 I20230907 16:42:59.377444 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.406826922+02:00 I20230907 16:42:59.377674 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.406847676+02:00 I20230907 16:43:05.301301 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:45:09.406856688+02:00 I20230907 16:43:09.380699 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.406864071+02:00 I20230907 16:43:09.380733 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.406871361+02:00 I20230907 16:43:19.381362 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.406878624+02:00 I20230907 16:43:19.381409 299 raft_server.h:60] Peer ref resh succeeded! 2023-09-07T18:45:09.406886953+02:00 I20230907 16:43:29.384903 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.406896817+02:00 I20230907 16:43:29.384936 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.406906368+02:00 I20230907 16:43:39.385644 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.406920303+02:00 I20230907 16:43:39.385673 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.406930428+02:00 I20230907 16:43:49.386977 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.406940005+02:00 I20230907 16:43:49.387043 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.406950790+02:00 I20230907 16:43:59.388796 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.406958537+02:00 I20230907 16:43:59.388839 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.406966513+02:00 I20230907 16:44:06.319248 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:45:09.406973859+02:00 I20230907 16:44:09.390421 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.406981215+02:00 I20230907 16:44:09.390461 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.406988674+02:00 I20230907 16:44:19.39280 2 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.407007241+02:00 I20230907 16:44:19.392920 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.407014721+02:00 I20230907 16:44:29.394214 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.407022017+02:00 I20230907 16:44:29.394250 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.407029277+02:00 I20230907 16:44:39.396353 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.407036554+02:00 I20230907 16:44:39.396409 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.407043846+02:00 I20230907 16:44:49.398945 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.407051136+02:00 I20230907 16:44:49.398979 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.407058499+02:00 I20230907 16:44:59.400401 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:45:09.407065746+02:00 I20230907 16:44:59.400434 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:45:09.407097088+02:00 I20230907 16:45:07.340353 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:47:19.445969365+02:00 I20230907 16:45:09.405902 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_inde x: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:47:19.445990304+02:00 I20230907 16:45:09.406706 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:47:19.445992623+02:00 I20230907 16:45:19.409269 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:47:19.445994606+02:00 I20230907 16:45:19.409313 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:47:19.445996605+02:00 I20230907 16:45:29.412350 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:47:19.445998353+02:00 I20230907 16:45:29.412381 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:47:19.446000078+02:00 I20230907 16:45:39.413200 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:47:19.446001818+02:00 I20230907 16:45:39.413240 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:45:49.414548 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:45:49.414584 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:47:19.446007029+02:00 I20230907 16:45:59.415594 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:47:19.446013002+02:00 I20230907 16:45:59.415688 292 raft_server.h:60] Peer refresh succeeded! I20230907 16:46:08.349647 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:46:09.416549 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:46:09.416652 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:46:19.420742 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:47:19.446024520+02:00 I20230907 16:46:19.420869 292 raft_server.h:60] Peer refresh succeeded! I20230907 16:46:29.422032 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:47:19.446029833+02:00 I20230907 16:46:29.422075 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:46:39.426769 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:46:39.426831 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:46:49.432679 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:46:49.432727 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:46:59.436492 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:46:59.436576 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:47:09.367693 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:47:09.444370 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:47:09.444492 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:49:29.472058114+02:00 I20230907 16:47:19.445700 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:49:29.472104443+02:00 I20230907 16:47:19.445942 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:49:29.472110891+02:00 I20230907 16:47:29.447484 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:47:29.447587 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:49:29.472123594+02:00 I20230907 16:47:39.449258 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:49:29.472144613+02:00 I20230907 16:47:39.449947 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:47:49.451269 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:47:49.451321 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:49:29.472205154+02:00 I20230907 16:47:59.452847 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:49:29.472209676+02:00 I20230907 16:47:59.452962 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:49:29.472214101+02:00 I20230907 16:48:09.455587 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:49:29.472218838+02:00 I20230907 16:48:09.456355 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:48:10.378072 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:48:19.459204 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:48:19.459300 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:49:29.472237254+02:00 I20230907 16:48:29.463717 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:48:29.463759 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:48:39.464480 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:48:39.464511 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:48:49.465201 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:48:49.465250 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:48:59.466076 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:48:59.466217 305 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:49:29.472274026+02:00 I20230907 16:49:09.466974 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:49:09.467108 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:49:29.472282979+02:00 I20230907 16:49:11.394212 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:49:19.470341 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:49:19.470397 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:49:29.471253 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506833626+02:00 I20230907 16:49:29.471889 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:49:39.472606 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506838513+02:00 I20230907 16:49:39.472642 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:49:49.473484 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506842750+02:00 I20230907 16:49:49.473608 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:49:59.478327 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506847253+02:00 I20230907 16:49:59.478507 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:51:39.506849308+02:00 I20230907 16:50:09.479187 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:50:09.479246 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:50:12.402351 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:51:39.506855669+02:00 I20230907 16:50:19.480391 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506857777+02:00 I20230907 16:50:19.480535 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:50:29.481473 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506861868+02:00 I20230907 16:50:29.481530 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:50:39.482787 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506866126+02:00 I20230907 16:50:39.482834 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:51:39.506868371+02:00 I20230907 16:50:49.485591 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:50:49.485635 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:51:39.506872714+02:00 I20230907 16:50:59.490762 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506879323+02:00 I20230907 16:50:59.491313 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:51:39.506881606+02:00 I20230907 16:51:09.503599 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506883677+02:00 I20230907 16:51:09.503719 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:51:39.506885957+02:00 I20230907 16:51:13.412426 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:51:39.506888093+02:00 I20230907 16:51:19.504729 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506890200+02:00 I20230907 16:51:19.504766 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:51:39.506892458+02:00 I20230907 16:51:29.505527 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:51:39.506894517+02:00 I20230907 16:51:29.505563 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:51:39.506902531+02:00 I20230907 16:51:39.506449 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:51:39.506489 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:53:49.519409613+02:00 I20230907 16:51:49.507345 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:53:49.519418209+02:00 I20230907 16:51:49.507408 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:53:49.519425139+02:00 I20230907 16:51:59.508363 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:53:49.519432892+02:00 I20230907 16:51:59.508404 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:53:49.519440954+02:00 I20230907 16:52:09.509209 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:53:49.519448509+02:00 I20230907 16:52:09.509264 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:52:14.417016 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:53:49.519462411+02:00 I20230907 16:52:19.510066 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:52:19.510102 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:52:29.511366 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:52:29.511407 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:52:39.512140 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:52:39.512214 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:52:49.513013 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:52:49.513049 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:53:49.519520164+02:00 I20230907 16:52:59.513851 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:53:49.519529132+02:00 I20230907 16:52:59.513895 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:53:49.519538057+02:00 I20230907 16:53:09.514724 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:53:49.519543827+02:00 I20230907 16:53:09.514762 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:53:49.519549488+02:00 I20230907 16:53:15.422190 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:53:49.519555152+02:00 I20230907 16:53:19.515636 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:53:49.519561307+02:00 I20230907 16:53:19.515671 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:53:29.516753 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:53:49.519577233+02:00 I20230907 16:53:29.516788 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:53:49.519584966+02:00 I20230907 16:53:39.517629 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:53:39.517669 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:53:49.518606 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532519966+02:00 I20230907 16:53:49.518702 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:56:09.532535674+02:00 I20230907 16:53:59.519462 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:53:59.519497 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:54:09.520290 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532562445+02:00 I20230907 16:54:09.520351 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:56:09.532567609+02:00 I20230907 16:54:16.427273 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:56:09.532570241+02:00 I20230907 16:54:19.522101 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532579192+02:00 I20230907 16:54:19.522168 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:56:09.532581307+02:00 I20230907 16:54:29.522970 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532583035+02:00 I20230907 16:54:29.523031 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:54:39.523911 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532586577+02:00 I20230907 16:54:39.523945 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:56:09.532588307+02:00 I20230907 16:54:49.524821 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532590030+02:00 I20230907 16:54:49.524860 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:54:59.525784 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:54:59.525847 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:56:09.532595385+02:00 I20230907 16:55:09.526659 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532597092+02:00 I20230907 16:55:09.526722 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:55:17.432358 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T18:56:09.532600564+02:00 I20230907 16:55:19.527577 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532602275+02:00 I20230907 16:55:19.527614 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:55:29.528379 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532605705+02:00 I20230907 16:55:29.528414 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:55:39.529318 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:56:09.532609135+02:00 I20230907 16:55:39.529350 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:56:09.532610944+02:00 I20230907 16:55:49.530324 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:55:49.530357 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:56:09.532614423+02:00 I20230907 16:55:59.531132 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:55:59.531173 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:58:19.563906133+02:00 I20230907 16:56:09.532176 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:56:09.532493 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:56:18.441802 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:56:19.533317 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:56:19.533390 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:56:29.534129 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:58:19.563940058+02:00 I20230907 16:56:29.534178 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:56:39.542182 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:58:19.563944291+02:00 I20230907 16:56:39.544174 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:56:49.550580 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:58:19.563948527+02:00 I20230907 16:56:49.550616 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:58:19.563950653+02:00 I20230907 16:56:59.553225 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:58:19.563952761+02:00 I20230907 16:56:59.553280 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:58:19.563954878+02:00 I20230907 16:57:09.554186 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:57:09.554231 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:58:19.563959081+02:00 I20230907 16:57:19.452324 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:57:19.557760 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:58:19.563963373+02:00 I20230907 16:57:19.557806 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:57:29.558519 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:58:19.563967681+02:00 I20230907 16:57:29.558558 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:58:19.563969994+02:00 I20230907 16:57:39.559978 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:57:39.560016 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:58:19.563978480+02:00 I20230907 16:57:49.561477 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:58:19.563980603+02:00 I20230907 16:57:49.561517 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:58:19.563982731+02:00 I20230907 16:57:59.562211 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:57:59.562254 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T18:58:19.563986932+02:00 I20230907 16:58:09.562929 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T18:58:19.563989164+02:00 I20230907 16:58:09.562968 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:58:19.563656 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T19:00:29.580843277+02:00 I20230907 16:58:19.563899 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:58:20.459970 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 2023-09-07T19:00:29.580854976+02:00 I20230907 16:58:29.564522 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:58:29.564555 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:58:39.565266 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:58:39.565328 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T19:00:29.580867508+02:00 I20230907 16:58:49.566066 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T19:00:29.580870828+02:00 I20230907 16:58:49.566104 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:58:59.566893 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T19:00:29.580876885+02:00 I20230907 16:58:59.566926 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:59:09.567569 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T19:00:29.580883090+02:00 I20230907 16:59:09.567603 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:59:19.570607 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:59:19.570647 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:59:21.463987 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 I20230907 16:59:29.571308 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T19:00:29.580904231+02:00 I20230907 16:59:29.571343 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:59:39.572181 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T19:00:29.580910195+02:00 I20230907 16:59:39.572216 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T19:00:29.580913147+02:00 I20230907 16:59:49.573062 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 16:59:49.573096 299 raft_server.h:60] Peer refresh succeeded! I20230907 16:59:59.574018 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 2023-09-07T19:00:29.580922240+02:00 I20230907 16:59:59.574054 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T19:00:29.580925305+02:00 I20230907 17:00:09.578420 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 17:00:09.578545 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T19:00:29.580931264+02:00 I20230907 17:00:19.579238 262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829 I20230907 17:00:19.579300 299 raft_server.h:60] Peer refresh succeeded! 2023-09-07T19:00:29.580951212+02:00 I20230907 17:00:22.468595 263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0 ```
OVERLORD commented 2026-02-05 01:14:01 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 7, 2023):

I mean there are all types of messages and errors here. If you restart the system is there a persistent issue you are running into?

@jrasm91 commented on GitHub (Sep 7, 2023): I mean there are all types of messages and errors here. If you restart the system is there a persistent issue you are running into?
OVERLORD commented 2026-02-05 01:14:04 +03:00
Author
Owner
Copy Link

@yodatak commented on GitHub (Sep 7, 2023):

in mass importing some file i got this error again i use http3 if it can change somethink


Upload Progress | █████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 11% || 38675/327529 || Current file [/mnt/XXXXXX/IMG_4162.CR2]
Error asset:  {
  file: '/mnt/rXXXXXX/C0012.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://XXXXXX/api/asset/import',
      data: '{"deviceAssetId":"C0012.MP4-3674651378","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:12:16.000Z","fileModifiedAt":"2021-07-15T12:12:16.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXXXXX/C0012.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXX',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
@yodatak commented on GitHub (Sep 7, 2023): in mass importing some file i got this error again i use http3 if it can change somethink ```json Upload Progress | █████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 11% || 38675/327529 || Current file [/mnt/XXXXXX/IMG_4162.CR2] Error asset: { file: '/mnt/rXXXXXX/C0012.MP4', reason: Error: Request failed with status code 504 at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15) at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12) at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11) at IncomingMessage.emit (node:events:526:35) at endReadableNT (node:internal/streams/readable:1359:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: Infinity, maxBodyLength: Infinity, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', maxRedirects: 0, url: 'https://XXXXXX/api/asset/import', data: '{"deviceAssetId":"C0012.MP4-3674651378","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:12:16.000Z","fileModifiedAt":"2021-07-15T12:12:16.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXXXXX/C0012.MP4"}' }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: true, chunkedEncoding: false, shouldKeepAlive: false, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: 322, _hasBody: true, _trailer: '', finished: true, _headerSent: true, _closed: true, socket: [TLSSocket], _header: 'POST /api/asset/import HTTP/1.1\r\n' + 'Accept: application/json, text/plain, */*\r\n' + 'Content-Type: application/json\r\n' + 'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' + 'User-Agent: axios/0.26.1\r\n' + 'Content-Length: 322\r\n' + 'Host: XXXX\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/api/asset/import', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'XXX', protocol: 'https:', [Symbol(kCapture)]: false, [Symbol(kBytesWritten)]: 0, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: [Object: null prototype], [Symbol(errored)]: null, [Symbol(kHighWaterMark)]: 16384, [Symbol(kRejectNonStandardBodyWrites)]: false, [Symbol(kUniqueHeaders)]: null }, response: { status: 504, statusText: 'Gateway Timeout', headers: [Object], config: [Object], request: [ClientRequest], data: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' }, isAxiosError: true, toJSON: [Function: toJSON] }, response: '<html>\r\n' + '<head><title>504 Gateway Time-out</title></head>\r\n' + '<body>\r\n' + '<center><h1>504 Gateway Time-out</h1></center>\r\n' + '<hr><center>nginx/1.25.0</center>\r\n' + '</body>\r\n' + '</html>\r\n' } ```
OVERLORD commented 2026-02-05 01:14:07 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 7, 2023):

504 is a gateway timeout which can happen with mp4s and --import. The server takes too long to hash the file and nginx throws an error. They are successfully imported though.

@jrasm91 commented on GitHub (Sep 7, 2023): 504 is a gateway timeout which can happen with mp4s and --import. The server takes too long to hash the file and nginx throws an error. They are successfully imported though.
OVERLORD commented 2026-02-05 01:14:09 +03:00
Author
Owner
Copy Link

@yodatak commented on GitHub (Sep 7, 2023):

Intersting it failled again on the same file , how could i fix it with nginx ?

@yodatak commented on GitHub (Sep 7, 2023): Intersting it failled again on the same file , how could i fix it with nginx ?
OVERLORD commented 2026-02-05 01:14:11 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 7, 2023):

You need to add this in the nginx.conf to file in the /api location block

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

@jrasm91 commented on GitHub (Sep 7, 2023): You need to add this in the nginx.conf to file in the /api location block http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
OVERLORD commented 2026-02-05 01:14:15 +03:00
Author
Owner
Copy Link

@yodatak commented on GitHub (Sep 7, 2023):

In immich proxy i got this error that goes effectily to timeout issues ^^
Deployment: immich-proxy
Container: immich

2023/09/07 21:24:28 [error] 43#43: *103492 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.42.0.78, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://1XXXX:10001/asset/import", host: "XXXXX"
2023-09-07T21:24:28.262313247+02:00 2023/09/07 21:24:28 [error] 43#43: *103155 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 1XXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://XXXX:10001/asset/import", host: "XXXXXX"
2023-09-07T21:24:28.317295066+02:00 2023/09/07 21:24:28 [error] 45#45: *103025 upstream timed out (110: Operation timed out) while reading response header from upstream, client: XXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://XXXX:10001/asset/import", host: "XXXX"
2023/09/07 21:24:28 [error] 43#43: *103079 upstream timed out (110: Operation timed out) while reading response header from upstream, client: XXXXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://10XXXXXX:10001/asset/import", host: "XXXXX"
@yodatak commented on GitHub (Sep 7, 2023): In immich proxy i got this error that goes effectily to timeout issues ^^ Deployment: immich-proxy Container: immich ``` 2023/09/07 21:24:28 [error] 43#43: *103492 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.42.0.78, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://1XXXX:10001/asset/import", host: "XXXXX" 2023-09-07T21:24:28.262313247+02:00 2023/09/07 21:24:28 [error] 43#43: *103155 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 1XXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://XXXX:10001/asset/import", host: "XXXXXX" 2023-09-07T21:24:28.317295066+02:00 2023/09/07 21:24:28 [error] 45#45: *103025 upstream timed out (110: Operation timed out) while reading response header from upstream, client: XXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://XXXX:10001/asset/import", host: "XXXX" 2023/09/07 21:24:28 [error] 43#43: *103079 upstream timed out (110: Operation timed out) while reading response header from upstream, client: XXXXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://10XXXXXX:10001/asset/import", host: "XXXXX" ```
OVERLORD commented 2026-02-05 01:14:17 +03:00
Author
Owner
Copy Link

@yodatak commented on GitHub (Sep 7, 2023):

I try this and i will say if its fix but we need to modiy the default.conf file in container

nginx.config

    keepalive_timeout  65;

defaut.conf

map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

map $http_x_forwarded_proto $forwarded_protocol {
  default $scheme;

  # Only allow the values 'http' and 'https' for the X-Forwarded-Proto header.
  http http;
  https https;
}

upstream server {
  server immich-server:10001;
  keepalive 2;
}

upstream web {
  server immich-web:10000;
  keepalive 2;
}

server {
  listen 8080;

  access_log off;
  client_max_body_size 50000M;

  # Compression
  gzip on;
  gzip_comp_level 2;
  gzip_min_length 1000;
  gzip_proxied any;
  gzip_vary on;
  gunzip on;

  # text/html is included by default
  gzip_types
    application/javascript
    application/json
    font/ttf
    image/svg+xml
    text/css;

  proxy_buffering off;
  proxy_request_buffering off;
  proxy_buffer_size 16k;
  proxy_busy_buffers_size 24k;
  proxy_buffers 64 4k;
  proxy_force_ranges on;

  proxy_http_version 1.1;
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-Host $http_host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $forwarded_protocol;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;

  location /api {

    rewrite /api/(.*) /$1 break;
    proxy_read_timeout 300s;
    proxy_pass http://server;
  }

  location / {

    proxy_pass http://web;
  }
}

and i reload the container in runtime
nginx -s reload

we must fix it in
https://github.com/immich-app/immich/blob/main/nginx/templates/default.conf.template

@yodatak commented on GitHub (Sep 7, 2023): I try this and i will say if its fix but we need to modiy the default.conf file in container nginx.config ```nginx keepalive_timeout 65; ``` defaut.conf ```nginx map $http_upgrade $connection_upgrade { default upgrade; '' close; } map $http_x_forwarded_proto $forwarded_protocol { default $scheme; # Only allow the values 'http' and 'https' for the X-Forwarded-Proto header. http http; https https; } upstream server { server immich-server:10001; keepalive 2; } upstream web { server immich-web:10000; keepalive 2; } server { listen 8080; access_log off; client_max_body_size 50000M; # Compression gzip on; gzip_comp_level 2; gzip_min_length 1000; gzip_proxied any; gzip_vary on; gunzip on; # text/html is included by default gzip_types application/javascript application/json font/ttf image/svg+xml text/css; proxy_buffering off; proxy_request_buffering off; proxy_buffer_size 16k; proxy_busy_buffers_size 24k; proxy_buffers 64 4k; proxy_force_ranges on; proxy_http_version 1.1; proxy_set_header Host $http_host; proxy_set_header X-Forwarded-Host $http_host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $forwarded_protocol; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; location /api { rewrite /api/(.*) /$1 break; proxy_read_timeout 300s; proxy_pass http://server; } location / { proxy_pass http://web; } } ``` and i reload the container in runtime nginx -s reload we must fix it in https://github.com/immich-app/immich/blob/main/nginx/templates/default.conf.template
OVERLORD commented 2026-02-05 01:14:19 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 8, 2023):

Long timeouts aren't necessarily good, but for this specific issue/endpoint it would probably be worthwhile

@jrasm91 commented on GitHub (Sep 8, 2023): Long timeouts aren't necessarily good, but for this specific issue/endpoint it would probably be worthwhile
OVERLORD commented 2026-02-05 01:14:20 +03:00
Author
Owner
Copy Link

@jrasm91 commented on GitHub (Sep 8, 2023):

We could make the timeout just for the import endpoint.

@jrasm91 commented on GitHub (Sep 8, 2023): We could make the timeout just for the import endpoint.
OVERLORD commented 2026-02-05 01:14:22 +03:00
Author
Owner
Copy Link

@yodatak commented on GitHub (Oct 26, 2023):

Anny news on this i got this issue on different ending point

Error: Connection terminated due to connection timeout

2023-10-26T20:25:55.462204908+02:00 Error: connect ETIMEDOUT
at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
2023-10-26T20:25:55.462209120+02:00 at Object.onceWrapper (node:events:628:28)
at Socket.emit (node:events:514:28)
at Socket._onTimeout (node:net:589:8)
2023-10-26T20:25:55.462215992+02:00 at listOnTimeout (node:internal/timers:573:17)
2023-10-26T20:25:55.462218781+02:00 at process.processTimers (node:internal/timers:514:7) {
2023-10-26T20:25:55.462220819+02:00 errorno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
syscall: 'connect'
}
Error: connect ETIMEDOUT
at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
2023-10-26T20:25:55.462233716+02:00 at Object.onceWrapper (node:events:628:28)
2023-10-26T20:25:55.462237799+02:00 at Socket.emit (node:events:514:28)
at Socket._onTimeout (node:net:589:8)
at listOnTimeout (node:internal/timers:573:17)
2023-10-26T20:25:55.462244319+02:00 at process.processTimers (node:internal/timers:514:7) {
errorno: 'ETIMEDOUT',
2023-10-26T20:25:55.462248581+02:00 code: 'ETIMEDOUT',
2023-10-26T20:25:55.462250657+02:00 syscall: 'connect'
2023-10-26T20:25:55.462252751+02:00 }
2023-10-26T20:25:55.462255039+02:00 Error: connect ETIMEDOUT
2023-10-26T20:25:55.462257197+02:00 at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
2023-10-26T20:25:55.462259347+02:00 at Object.onceWrapper (node:events:628:28)
2023-10-26T20:25:55.462261472+02:00 at Socket.emit (node:events:514:28)
at Socket._onTimeout (node:net:589:8)
at listOnTimeout (node:internal/timers:573:17)
2023-10-26T20:25:55.462270760+02:00 at process.processTimers (node:internal/timers:514:7) {
errorno: 'ETIMEDOUT',
2023-10-26T20:25:55.462275142+02:00 code: 'ETIMEDOUT',
syscall: 'connect'
}
2023-10-26T20:25:55.462281600+02:00 Error: connect ETIMEDOUT
at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
2023-10-26T20:25:55.462285891+02:00 at Object.onceWrapper (node:events:628:28)
at Socket.emit (node:events:514:28)
2023-10-26T20:25:55.462290155+02:00 at Socket._onTimeout (node:net:589:8)
at listOnTimeout (node:internal/timers:573:17)
2023-10-26T20:25:55.462294432+02:00 at process.processTimers (node:internal/timers:514:7) {
2023-10-26T20:25:55.462296511+02:00 errorno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
2023-10-26T20:25:55.462300825+02:00 syscall: 'connect'
2023-10-26T20:25:55.462302901+02:00 }
Error: connect ETIMEDOUT
2023-10-26T20:25:55.462307119+02:00 at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
2023-10-26T20:25:55.462309280+02:00 at Object.onceWrapper (node:events:628:28)
at Socket.emit (node:events:514:28)
2023-10-26T20:25:55.462313786+02:00 at Socket._onTimeout (node:net:589:8)
2023-10-26T20:25:55.462316104+02:00 at listOnTimeout (node:internal/timers:573:17)
2023-10-26T20:25:55.462318139+02:00 at process.processTimers (node:internal/timers:514:7) {
errorno: 'ETIMEDOUT',
2023-10-26T20:25:55.462322457+02:00 code: 'ETIMEDOUT',
syscall: 'connect'
2023-10-26T20:25:55.462326839+02:00 }
[Nest] 8 - 10/26/2023, 8:25:55 PM LOG [MetadataService] Initialized local reverse geocoder with cities500
[Nest] 8 - 10/26/2023, 8:25:55 PM LOG [SearchService] Running bootstrap
[Nest] 8 - 10/26/2023, 8:25:55 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-10-26T20:25:56.028128335+02:00 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2
[Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1
[Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"albums-v2","name":"albums"}]
[Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Collections needing migration: {"assets":true,"albums":false,"faces":true}
[Nest] 8 - 10/26/2023, 8:25:56 PM LOG [NestApplication] Nest application successfully started +303ms
2023-10-26T20:25:56.359155681+02:00 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:10004 [v1.82.1] [PRODUCTION]
2023-10-26T20:27:41.200426155+02:00 [Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout
[Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Error: Connection terminated due to connection timeout
2023-10-26T20:27:41.200470079+02:00 at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73)
at Object.onceWrapper (node:events:628:28)
2023-10-26T20:27:41.200474479+02:00 at Connection.emit (node:events:514:28)
at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-10-26T20:27:41.200478859+02:00 at Socket.emit (node:events:514:28)
at TCP. (node:net:337:12)
2023-10-26T20:27:41.201124331+02:00 [Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Object:
{}
2023-10-26T20:27:41.201137732+02:00

<--- Last few GCs --->
2023-10-26T20:29:50.703361758+02:00
2023-10-26T20:29:50.703366124+02:00 [8:0x592a4310000] 254858 ms: Mark-Compact (reduce) 2046.4 (2083.4) -> 2045.9 (2083.9) MB, 4529.62 / 0.01 ms (+ 293.9 ms in 57 steps since start of marking, biggest step 43.7 ms, walltime since start of marking 4882 ms) (average mu = 0.119, current mu =[8:0x592a4310000] 257716 ms: Mark-Compact (reduce) 2047.0 (2083.9) -> 2045.4 (2083.9) MB, 2854.69 / 0.00 ms (average mu = 0.073, current mu = 0.001) allocation failure; scavenge might not succeed

2023-10-26T20:29:50.703371229+02:00
2023-10-26T20:29:50.703376160+02:00 <--- JS stacktrace --->
2023-10-26T20:29:50.703378497+02:00
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xc99960 node::Abort() [immich_microservices]
2: 0xb6ffcb [immich_microservices]
3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-10-26T20:29:50.721524217+02:00 5: 0x10d06a5 [immich_microservices]
6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices]
11: 0x109be16 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [immich_microservices]
12: 0x108d914 v8::internal::FactoryBasev8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [immich_microservices]
13: 0x1090196 v8::internal::FactoryBasev8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [immich_microservices]
14: 0x10a7324 v8::internal::Factory::NewStringFromUtf8(v8::base::Vector const&, v8::internal::AllocationType) [immich_microservices]
2023-10-26T20:29:50.871139582+02:00 15: 0xed1202 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [immich_microservices]
16: 0xdb7249 [immich_microservices]
17: 0xc6da6f [immich_microservices]
18: 0x7fbfc608d546
Aborted (core dumped)

@yodatak commented on GitHub (Oct 26, 2023): Anny news on this i got this issue on different ending point Error: Connection terminated due to connection timeout 2023-10-26T20:25:55.462204908+02:00 Error: connect ETIMEDOUT at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462209120+02:00 at Object.onceWrapper (node:events:628:28) at Socket.emit (node:events:514:28) at Socket._onTimeout (node:net:589:8) 2023-10-26T20:25:55.462215992+02:00 at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462218781+02:00 at process.processTimers (node:internal/timers:514:7) { 2023-10-26T20:25:55.462220819+02:00 errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' } Error: connect ETIMEDOUT at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462233716+02:00 at Object.onceWrapper (node:events:628:28) 2023-10-26T20:25:55.462237799+02:00 at Socket.emit (node:events:514:28) at Socket._onTimeout (node:net:589:8) at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462244319+02:00 at process.processTimers (node:internal/timers:514:7) { errorno: 'ETIMEDOUT', 2023-10-26T20:25:55.462248581+02:00 code: 'ETIMEDOUT', 2023-10-26T20:25:55.462250657+02:00 syscall: 'connect' 2023-10-26T20:25:55.462252751+02:00 } 2023-10-26T20:25:55.462255039+02:00 Error: connect ETIMEDOUT 2023-10-26T20:25:55.462257197+02:00 at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462259347+02:00 at Object.onceWrapper (node:events:628:28) 2023-10-26T20:25:55.462261472+02:00 at Socket.emit (node:events:514:28) at Socket._onTimeout (node:net:589:8) at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462270760+02:00 at process.processTimers (node:internal/timers:514:7) { errorno: 'ETIMEDOUT', 2023-10-26T20:25:55.462275142+02:00 code: 'ETIMEDOUT', syscall: 'connect' } 2023-10-26T20:25:55.462281600+02:00 Error: connect ETIMEDOUT at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462285891+02:00 at Object.onceWrapper (node:events:628:28) at Socket.emit (node:events:514:28) 2023-10-26T20:25:55.462290155+02:00 at Socket._onTimeout (node:net:589:8) at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462294432+02:00 at process.processTimers (node:internal/timers:514:7) { 2023-10-26T20:25:55.462296511+02:00 errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', 2023-10-26T20:25:55.462300825+02:00 syscall: 'connect' 2023-10-26T20:25:55.462302901+02:00 } Error: connect ETIMEDOUT 2023-10-26T20:25:55.462307119+02:00 at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462309280+02:00 at Object.onceWrapper (node:events:628:28) at Socket.emit (node:events:514:28) 2023-10-26T20:25:55.462313786+02:00 at Socket._onTimeout (node:net:589:8) 2023-10-26T20:25:55.462316104+02:00 at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462318139+02:00 at process.processTimers (node:internal/timers:514:7) { errorno: 'ETIMEDOUT', 2023-10-26T20:25:55.462322457+02:00 code: 'ETIMEDOUT', syscall: 'connect' 2023-10-26T20:25:55.462326839+02:00 } [Nest] 8 - 10/26/2023, 8:25:55 PM LOG [MetadataService] Initialized local reverse geocoder with cities500 [Nest] 8 - 10/26/2023, 8:25:55 PM LOG [SearchService] Running bootstrap [Nest] 8 - 10/26/2023, 8:25:55 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-10-26T20:25:56.028128335+02:00 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"albums-v2","name":"albums"}] [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Collections needing migration: {"assets":true,"albums":false,"faces":true} [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [NestApplication] Nest application successfully started +303ms 2023-10-26T20:25:56.359155681+02:00 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:10004 [v1.82.1] [PRODUCTION] 2023-10-26T20:27:41.200426155+02:00 [Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout [Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Error: Connection terminated due to connection timeout 2023-10-26T20:27:41.200470079+02:00 at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) 2023-10-26T20:27:41.200474479+02:00 at Connection.emit (node:events:514:28) at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12) 2023-10-26T20:27:41.200478859+02:00 at Socket.emit (node:events:514:28) at TCP.<anonymous> (node:net:337:12) 2023-10-26T20:27:41.201124331+02:00 [Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Object: {} 2023-10-26T20:27:41.201137732+02:00 <--- Last few GCs ---> 2023-10-26T20:29:50.703361758+02:00 2023-10-26T20:29:50.703366124+02:00 [8:0x592a4310000] 254858 ms: Mark-Compact (reduce) 2046.4 (2083.4) -> 2045.9 (2083.9) MB, 4529.62 / 0.01 ms (+ 293.9 ms in 57 steps since start of marking, biggest step 43.7 ms, walltime since start of marking 4882 ms) (average mu = 0.119, current mu =[8:0x592a4310000] 257716 ms: Mark-Compact (reduce) 2047.0 (2083.9) -> 2045.4 (2083.9) MB, 2854.69 / 0.00 ms (average mu = 0.073, current mu = 0.001) allocation failure; scavenge might not succeed 2023-10-26T20:29:50.703371229+02:00 2023-10-26T20:29:50.703376160+02:00 <--- JS stacktrace ---> 2023-10-26T20:29:50.703378497+02:00 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices] 2023-10-26T20:29:50.721524217+02:00 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0x109be16 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [immich_microservices] 12: 0x108d914 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [immich_microservices] 13: 0x1090196 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [immich_microservices] 14: 0x10a7324 v8::internal::Factory::NewStringFromUtf8(v8::base::Vector<char const> const&, v8::internal::AllocationType) [immich_microservices] 2023-10-26T20:29:50.871139582+02:00 15: 0xed1202 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [immich_microservices] 16: 0xdb7249 [immich_microservices] 17: 0xc6da6f [immich_microservices] 18: 0x7fbfc608d546 Aborted (core dumped)
OVERLORD commented 2026-02-05 01:14:23 +03:00
Author
Owner
Copy Link

@alextran1502 commented on GitHub (Nov 1, 2023):

not relevance anymore with the library feature

@alextran1502 commented on GitHub (Nov 1, 2023): not relevance anymore with the library feature
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
needs-answer
nice to have
sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#1311
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?