Files
immich/server/src/utils/maintenance.ts

75 lines
2.3 KiB
TypeScript
Raw Normal View History

feat: maintenance mode (#23431) * feat: add a `maintenance.enabled` config flag * feat: implement graceful restart feat: restart when maintenance config is toggled * feat: boot a stripped down maintenance api if enabled * feat: cli command to toggle maintenance mode * chore: fallback IMMICH_SERVER_URL environment variable in process * chore: add additional routes to maintenance controller * fix: don't wait for nest application to close to finish request response * chore: add a failsafe on restart to prevent other exit codes from preventing restart * feat: redirect into/from maintenance page * refactor: use system metadata for maintenance status * refactor: wait on WebSocket connection to refresh * feat: broadcast websocket event on server restart refactor: listen to WS instead of polling * refactor: bubble up maintenance information instead of hijacking in fetch function feat: show modal when server is restarting * chore: increase timeout for ungraceful restart * refactor: deduplicate code between api/maintenance workers * fix: skip config check if database is not initialised * fix: add `maintenanceMode` field to system config test * refactor: move maintenance resolution code to static method in service * chore: clean up linter issues * chore: generate dart openapi * refactor: use try{} block for maintenance mode check * fix: logic error in server redirect * chore: include `maintenanceMode` key in e2e test * chore: add i18n entries for maintenance screens * chore: remove negated condition from hook * fix: should set default value not override in service * fix: minor error in page * feat: initial draft of maintenance module, repo., worker controller, worker service * refactor: move broadcast code into notification service * chore: connect websocket on client if in maintenance * chore: set maintenance module app name * refactor: rename repository to include worker chore: configure websocket adapter * feat: reimplement maintenance mode exit with new module * refactor: add a constant enum for ExitCode * refactor: remove redundant route for maintenance * refactor: only spin up kysely on boot (rather than a Nest app) * refactor(web): move redirect logic into +layout file where modal is setup * feat: add Maintenance permission * refactor: merge common code between api/maintenance * fix: propagate changes from the CLI to servers * feat: maintenance authentication guard * refactor: unify maintenance code into repository feat: add a step to generate maintenance mode token * feat: jwt auth for maintenance * refactor: switch from nest jwt to just jsonwebtokens * feat: log into maintenance mode from CLI command * refactor: use `secret` instead of `token` in jwt terminology chore: log maintenance mode login URL on boot chore: don't make CLI actions reload if already in target state * docs: initial draft for maintenance mode page * refactor: always validate the maintenance auth on the server * feat: add a link to maintenance mode documentation * feat: redirect users back to the last page they were on when exiting maintenance * refactor: provide closeFn in both maintenance repos. * refactor: ensure the user is also redirected by the server * chore: swap jsonwebtoken for jose * refactor: introduce AppRestartEvent w/o secret passing * refactor: use navigation goto * refactor: use `continue` instead of `next` * chore: lint fixes for server * chore: lint fixes for web * test: add mock for maintenance repository * test: add base service dependency to maintenance * chore: remove @types/jsonwebtoken * refactor: close database connection after startup check * refactor: use `request#auth` key * refactor: use service instead of repository chore: read token from cookie if possible chore: rename client event to AppRestartV1 * refactor: more concise redirect logic on web * refactor: move redirect check into utils refactor: update translation strings to be more sensible * refactor: always validate login (i.e. check cookie) * refactor: lint, open-api, remove old dto * refactor: encode at point of usage * refactor: remove business logic from repositories * chore: fix server/web lints * refactor: remove repository mock * chore: fix formatting * test: write service mocks for maintenance mode * test: write cli service tests * fix: catch errors when closing app * fix: always report no maintenance when usual API is available * test: api e2e maintenance spec * chore: add response builder * chore: add helper to set maint. auth cookie * feat: add SSR to maintenance API * test(e2e): write web spec for maintenance * chore: clean up lint issues * chore: format files * feat: perform 302 redirect at server level during maintenance * fix: keep trying to stop immich until it succeeds (CLI issue) * chore: lint/format * refactor: annotate references to other services in worker service * chore: lint * refactor: remove unnecessary await Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com> * refactor: move static methods into util * refactor: assert secret exists in maintenance worker * refactor: remove assertion which isn't necessary anymore * refactor: remove assertion * refactor: remove outer try {} catch block from loadMaintenanceAuth * refactor: undo earlier change to vite.config.ts * chore: update tests due to refactors * revert: vite.config.ts * test: expect string jwt * chore: move blanket exceptions into controllers * test: update tests according with last change * refactor: use respondWithCookie refactor: merge start/end into one route refactor: rename MaintenanceRepository to AppRepository chore: use new ApiTag/Endpoint refactor: apply other requested changes * chore: regenerate openapi * chore: lint/format * chore: remove secureOnly for maint. cookie * refactor: move maintenance worker code into src/maintenance\nfix: various test fixes * refactor: use `action` property for setting maint. mode * refactor: remove Websocket#restartApp in favour of individual methods * chore: incomplete commit * chore: remove stray log * fix: call exitApp from maintenance worker on exit * fix: add app repository mock * fix: ensure maintenance cookies are secure * fix: run playwright tests over secure context (localhost) * test: update other references to 127.0.0.1 * refactor: use serverSideEmitWithAck * chore: correct the logic in tryTerminate * test: juggle cookies ourselves * chore: fix lint error for e2e spec * chore: format e2e test * fix: set cookie secure/non-secure depending on context * chore: format files --------- Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>
2025-11-17 17:15:44 +00:00
import { createAdapter } from '@socket.io/redis-adapter';
import Redis from 'ioredis';
import { SignJWT } from 'jose';
import { randomBytes } from 'node:crypto';
import { Server as SocketIO } from 'socket.io';
import { MaintenanceAuthDto } from 'src/dtos/maintenance.dto';
import { ConfigRepository } from 'src/repositories/config.repository';
import { AppRestartEvent } from 'src/repositories/event.repository';
export function sendOneShotAppRestart(state: AppRestartEvent): void {
const server = new SocketIO();
const { redis } = new ConfigRepository().getEnv();
const pubClient = new Redis(redis);
const subClient = pubClient.duplicate();
server.adapter(createAdapter(pubClient, subClient));
/**
* Keep trying until we manage to stop Immich
*
* Sometimes there appear to be communication
* issues between to the other servers.
*
* This issue only occurs with this method.
*/
async function tryTerminate() {
while (true) {
try {
const responses = await server.serverSideEmitWithAck('AppRestart', state);
if (responses.length > 0) {
return;
}
} catch (error) {
console.error(error);
console.error('Encountered an error while telling Immich to stop.');
}
console.info(
"\nIt doesn't appear that Immich stopped, trying again in a moment.\nIf Immich is already not running, you can ignore this error.",
);
await new Promise((r) => setTimeout(r, 1e3));
}
}
// => corresponds to notification.service.ts#onAppRestart
server.emit('AppRestartV1', state, () => {
void tryTerminate().finally(() => {
pubClient.disconnect();
subClient.disconnect();
});
});
}
export async function createMaintenanceLoginUrl(
baseUrl: string,
auth: MaintenanceAuthDto,
secret: string,
): Promise<string> {
return `${baseUrl}/maintenance?token=${await signMaintenanceJwt(secret, auth)}`;
}
export async function signMaintenanceJwt(secret: string, data: MaintenanceAuthDto): Promise<string> {
const alg = 'HS256';
return await new SignJWT({ ...data })
.setProtectedHeader({ alg })
.setIssuedAt()
.setExpirationTime('4h')
.sign(new TextEncoder().encode(secret));
}
export function generateMaintenanceSecret(): string {
return randomBytes(64).toString('hex');
}