feat: maintenance mode (#23431)

* feat: add a `maintenance.enabled` config flag

* feat: implement graceful restart
feat: restart when maintenance config is toggled

* feat: boot a stripped down maintenance api if enabled

* feat: cli command to toggle maintenance mode

* chore: fallback IMMICH_SERVER_URL environment variable in process

* chore: add additional routes to maintenance controller

* fix: don't wait for nest application to close to finish request response

* chore: add a failsafe on restart to prevent other exit codes from preventing restart

* feat: redirect into/from maintenance page

* refactor: use system metadata for maintenance status

* refactor: wait on WebSocket connection to refresh

* feat: broadcast websocket event on server restart
refactor: listen to WS instead of polling

* refactor: bubble up maintenance information instead of hijacking in fetch function
feat: show modal when server is restarting

* chore: increase timeout for ungraceful restart

* refactor: deduplicate code between api/maintenance workers

* fix: skip config check if database is not initialised

* fix: add `maintenanceMode` field to system config test

* refactor: move maintenance resolution code to static method in service

* chore: clean up linter issues

* chore: generate dart openapi

* refactor: use try{} block for maintenance mode check

* fix: logic error in server redirect

* chore: include `maintenanceMode` key in e2e test

* chore: add i18n entries for maintenance screens

* chore: remove negated condition from hook

* fix: should set default value not override in service

* fix: minor error in page

* feat: initial draft of maintenance module, repo., worker controller, worker service

* refactor: move broadcast code into notification service

* chore: connect websocket on client if in maintenance

* chore: set maintenance module app name

* refactor: rename repository to include worker
chore: configure websocket adapter

* feat: reimplement maintenance mode exit with new module

* refactor: add a constant enum for ExitCode

* refactor: remove redundant route for maintenance

* refactor: only spin up kysely on boot (rather than a Nest app)

* refactor(web): move redirect logic into +layout file where modal is setup

* feat: add Maintenance permission

* refactor: merge common code between api/maintenance

* fix: propagate changes from the CLI to servers

* feat: maintenance authentication guard

* refactor: unify maintenance code into repository
feat: add a step to generate maintenance mode token

* feat: jwt auth for maintenance

* refactor: switch from nest jwt to just jsonwebtokens

* feat: log into maintenance mode from CLI command

* refactor: use `secret` instead of `token` in jwt terminology
chore: log maintenance mode login URL on boot
chore: don't make CLI actions reload if already in target state

* docs: initial draft for maintenance mode page

* refactor: always validate the maintenance auth on the server

* feat: add a link to maintenance mode documentation

* feat: redirect users back to the last page they were on when exiting maintenance

* refactor: provide closeFn in both maintenance repos.

* refactor: ensure the user is also redirected by the server

* chore: swap jsonwebtoken for jose

* refactor: introduce AppRestartEvent w/o secret passing

* refactor: use navigation goto

* refactor: use `continue` instead of `next`

* chore: lint fixes for server

* chore: lint fixes for web

* test: add mock for maintenance repository

* test: add base service dependency to maintenance

* chore: remove @types/jsonwebtoken

* refactor: close database connection after startup check

* refactor: use `request#auth` key

* refactor: use service instead of repository
chore: read token from cookie if possible
chore: rename client event to AppRestartV1

* refactor: more concise redirect logic on web

* refactor: move redirect check into utils
refactor: update translation strings to be more sensible

* refactor: always validate login (i.e. check cookie)

* refactor: lint, open-api, remove old dto

* refactor: encode at point of usage

* refactor: remove business logic from repositories

* chore: fix server/web lints

* refactor: remove repository mock

* chore: fix formatting

* test: write service mocks for maintenance mode

* test: write cli service tests

* fix: catch errors when closing app

* fix: always report no maintenance when usual API is available

* test: api e2e maintenance spec

* chore: add response builder

* chore: add helper to set maint. auth cookie

* feat: add SSR to maintenance API

* test(e2e): write web spec for maintenance

* chore: clean up lint issues

* chore: format files

* feat: perform 302 redirect at server level during maintenance

* fix: keep trying to stop immich until it succeeds (CLI issue)

* chore: lint/format

* refactor: annotate references to other services in worker service

* chore: lint

* refactor: remove unnecessary await

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>

* refactor: move static methods into util

* refactor: assert secret exists in maintenance worker

* refactor: remove assertion which isn't necessary anymore

* refactor: remove assertion

* refactor: remove outer try {} catch block from loadMaintenanceAuth

* refactor: undo earlier change to vite.config.ts

* chore: update tests due to refactors

* revert: vite.config.ts

* test: expect string jwt

* chore: move blanket exceptions into controllers

* test: update tests according with last change

* refactor: use respondWithCookie
refactor: merge start/end into one route
refactor: rename MaintenanceRepository to AppRepository
chore: use new ApiTag/Endpoint
refactor: apply other requested changes

* chore: regenerate openapi

* chore: lint/format

* chore: remove secureOnly for maint. cookie

* refactor: move maintenance worker code into src/maintenance\nfix: various test fixes

* refactor: use `action` property for setting maint. mode

* refactor: remove Websocket#restartApp in favour of individual methods

* chore: incomplete commit

* chore: remove stray log

* fix: call exitApp from maintenance worker on exit

* fix: add app repository mock

* fix: ensure maintenance cookies are secure

* fix: run playwright tests over secure context (localhost)

* test: update other references to 127.0.0.1

* refactor: use serverSideEmitWithAck

* chore: correct the logic in tryTerminate

* test: juggle cookies ourselves

* chore: fix lint error for e2e spec

* chore: format e2e test

* fix: set cookie secure/non-secure depending on context

* chore: format files

---------

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>

web/src/lib/components/admin-settings/MaintenanceSettings.svelte

@@ -0,0 +1,35 @@
+<script lang="ts">
+  import { handleError } from '$lib/utils/handle-error';
+  import { MaintenanceAction, setMaintenanceMode } from '@immich/sdk';
+  import { Button } from '@immich/ui';
+  import { t } from 'svelte-i18n';
+  import { fade } from 'svelte/transition';
+
+  interface Props {
+    disabled?: boolean;
+  }
+
+  let { disabled = false }: Props = $props();
+
+  async function start() {
+    try {
+      await setMaintenanceMode({
+        setMaintenanceModeDto: {
+          action: MaintenanceAction.Start,
+        },
+      });
+    } catch (error) {
+      handleError(error, $t('admin.maintenance_start_error'));
+    }
+  }
+</script>
+
+<div>
+  <div in:fade={{ duration: 500 }}>
+    <div class="ms-4 mt-4 flex items-end gap-4">
+      <Button shape="round" type="submit" {disabled} size="small" onclick={start}
+        >{$t('admin.maintenance_start')}</Button
+      >
+    </div>
+  </div>
+</div>

web/src/lib/constants.ts

@@ -59,6 +59,8 @@ export enum AppRoute {
   FOLDERS = '/folders',
   TAGS = '/tags',
   LOCKED = '/locked',
+
+  MAINTENANCE = '/maintenance',
 }
 
 export enum ProjectionType {

web/src/lib/modals/ServerRestartingModal.svelte

@@ -0,0 +1,16 @@
+<script lang="ts">
+  import { Modal, ModalBody } from '@immich/ui';
+  import { t } from 'svelte-i18n';
+
+  type Props = {
+    onClose: () => void;
+  };
+
+  const { onClose }: Props = $props();
+</script>
+
+<Modal size="small" title={$t('server_restarting_title')} {onClose} icon={false}>
+  <ModalBody>
+    <div class="font-medium">{$t('server_restarting_description')}</div>
+  </ModalBody>
+</Modal>

web/src/lib/stores/maintenance.store.ts

@@ -0,0 +1,4 @@
+import { type MaintenanceAuthDto } from '@immich/sdk';
+import { writable } from 'svelte/store';
+
+export const maintenanceAuth = writable<MaintenanceAuthDto>();

web/src/lib/stores/websocket.ts

@@ -1,3 +1,5 @@
+import { page } from '$app/state';
+import { AppRoute } from '$lib/constants';
 import { authManager } from '$lib/managers/auth-manager.svelte';
 import { notificationManager } from '$lib/stores/notification-manager.svelte';
 import { createEventEmitter } from '$lib/utils/eventemitter';
@@ -13,6 +15,11 @@ export interface ReleaseEvent {
   serverVersion: ServerVersionResponseDto;
   releaseVersion: ServerVersionResponseDto;
 }
+
+interface AppRestartEvent {
+  isMaintenanceMode: boolean;
+}
+
 export interface Events {
   on_upload_success: (asset: AssetResponseDto) => void;
   on_user_delete: (id: string) => void;
@@ -28,6 +35,8 @@ export interface Events {
   on_new_release: (newRelease: ReleaseEvent) => void;
   on_session_delete: (sessionId: string) => void;
   on_notification: (notification: NotificationDto) => void;
+
+  AppRestartV1: (event: AppRestartEvent) => void;
 }
 
 const websocket: Socket<Events> = io({
@@ -42,6 +51,7 @@ export const websocketStore = {
   connected: writable<boolean>(false),
   serverVersion: writable<ServerVersionResponseDto>(),
   release: writable<ReleaseEvent>(),
+  serverRestarting: writable<undefined | AppRestartEvent>(),
 };
 
 export const websocketEvents = createEventEmitter(websocket);
@@ -50,6 +60,7 @@ websocket
   .on('connect', () => websocketStore.connected.set(true))
   .on('disconnect', () => websocketStore.connected.set(false))
   .on('on_server_version', (serverVersion) => websocketStore.serverVersion.set(serverVersion))
+  .on('AppRestartV1', (mode) => websocketStore.serverRestarting.set(mode))
   .on('on_new_release', (releaseVersion) => websocketStore.release.set(releaseVersion))
   .on('on_session_delete', () => authManager.logout())
   .on('on_notification', () => notificationManager.refresh())
@@ -57,11 +68,9 @@ websocket
 
 export const openWebsocketConnection = () => {
   try {
-    if (!get(user)) {
-      return;
+    if (get(user) || page.url.pathname.startsWith(AppRoute.MAINTENANCE)) {
+      websocket.connect();
     }
-
-    websocket.connect();
   } catch (error) {
     console.log('Cannot connect to websocket', error);
   }

web/src/lib/utils/maintenance.ts

@@ -0,0 +1,33 @@
+import { AppRoute } from '$lib/constants';
+import { maintenanceAuth as maintenanceAuth$ } from '$lib/stores/maintenance.store';
+import { maintenanceLogin } from '@immich/sdk';
+
+export function maintenanceCreateUrl(url: URL) {
+  const target = new URL(AppRoute.MAINTENANCE, url.origin);
+  target.searchParams.set('continue', url.pathname + url.search);
+  return target.href;
+}
+
+export function maintenanceReturnUrl(searchParams: URLSearchParams) {
+  return searchParams.get('continue') ?? '/';
+}
+
+export function maintenanceShouldRedirect(maintenanceMode: boolean, currentUrl: URL | Location) {
+  return maintenanceMode !== currentUrl.pathname.startsWith(AppRoute.MAINTENANCE);
+}
+
+export const loadMaintenanceAuth = async () => {
+  const query = new URLSearchParams(location.search);
+
+  try {
+    const auth = await maintenanceLogin({
+      maintenanceLoginDto: {
+        token: query.get('token') ?? undefined,
+      },
+    });
+
+    maintenanceAuth$.set(auth);
+  } catch {
+    // silently fail
+  }
+};

web/src/lib/utils/server.ts

@@ -12,8 +12,11 @@ async function _init(fetch: Fetch) {
   // https://github.com/oazapfts/oazapfts/blob/main/README.md#fetch-options
   defaults.fetch = fetch;
   await initLanguage();
-  await featureFlagsManager.init();
   await serverConfigManager.init();
+
+  if (!serverConfigManager.value.maintenanceMode) {
+    await featureFlagsManager.init();
+  }
 }
 
 export const init = memoize(_init, () => 'singlevalue');

web/src/routes/+layout.svelte

@@ -7,8 +7,10 @@
   import AppleHeader from '$lib/components/shared-components/apple-header.svelte';
   import NavigationLoadingBar from '$lib/components/shared-components/navigation-loading-bar.svelte';
   import UploadPanel from '$lib/components/shared-components/upload-panel.svelte';
+  import { AppRoute } from '$lib/constants';
   import { eventManager } from '$lib/managers/event-manager.svelte';
   import { serverConfigManager } from '$lib/managers/server-config-manager.svelte';
+  import ServerRestartingModal from '$lib/modals/ServerRestartingModal.svelte';
   import VersionAnnouncementModal from '$lib/modals/VersionAnnouncementModal.svelte';
   import { user } from '$lib/stores/user.store';
   import {
@@ -18,6 +20,7 @@
     type ReleaseEvent,
   } from '$lib/stores/websocket';
   import { copyToClipboard, getReleaseType, semverToName } from '$lib/utils';
+  import { maintenanceShouldRedirect } from '$lib/utils/maintenance';
   import { isAssetViewerRoute } from '$lib/utils/navigation';
   import { modalManager, setTranslations } from '@immich/ui';
   import { onMount, type Snippet } from 'svelte';
@@ -70,14 +73,14 @@
     showNavigationLoadingBar = false;
   });
   run(() => {
-    if ($user) {
+    if ($user || page.url.pathname.startsWith(AppRoute.MAINTENANCE)) {
       openWebsocketConnection();
     } else {
       closeWebsocketConnection();
     }
   });
 
-  const { release } = websocketStore;
+  const { release, serverRestarting } = websocketStore;
 
   const handleRelease = async (release?: ReleaseEvent) => {
     if (!release?.isAvailable || !$user.isAdmin) {
@@ -102,6 +105,27 @@
   };
 
   $effect(() => void handleRelease($release));
+
+  serverRestarting.subscribe((isRestarting) => {
+    if (!isRestarting) {
+      return;
+    }
+
+    if (maintenanceShouldRedirect(isRestarting.isMaintenanceMode, location)) {
+      modalManager.show(ServerRestartingModal, {}).catch((error) => console.error('Error [ServerRestartBox]:', error));
+
+      // we will be disconnected momentarily
+      // wait for reconnect then reload
+      let waiting = false;
+      websocketStore.connected.subscribe((connected) => {
+        if (!connected) {
+          waiting = true;
+        } else if (connected && waiting) {
+          location.reload();
+        }
+      });
+    }
+  });
 </script>
 
 <svelte:head>

web/src/routes/+layout.ts

@@ -1,13 +1,22 @@
+import { goto } from '$app/navigation';
+import { serverConfigManager } from '$lib/managers/server-config-manager.svelte';
+import { maintenanceCreateUrl, maintenanceReturnUrl, maintenanceShouldRedirect } from '$lib/utils/maintenance';
 import { init } from '$lib/utils/server';
 import type { LayoutLoad } from './$types';
 
 export const ssr = false;
 export const csr = true;
 
-export const load = (async ({ fetch }) => {
+export const load = (async ({ fetch, url }) => {
   let error;
   try {
     await init(fetch);
+
+    if (maintenanceShouldRedirect(serverConfigManager.value.maintenanceMode, url)) {
+      await goto(
+        serverConfigManager.value.maintenanceMode ? maintenanceCreateUrl(url) : maintenanceReturnUrl(url.searchParams),
+      );
+    }
   } catch (initError) {
     error = initError;
   }

web/src/routes/+page.ts

@@ -12,6 +12,11 @@ export const csr = true;
 export const load = (async ({ fetch }) => {
   try {
     await init(fetch);
+
+    if (serverConfigManager.value.maintenanceMode) {
+      redirect(302, AppRoute.MAINTENANCE);
+    }
+
     const authenticated = await loadUser();
     if (authenticated) {
       redirect(302, AppRoute.PHOTOS);

web/src/routes/admin/system-settings/+page.svelte

@@ -7,6 +7,7 @@
   import LibrarySettings from '$lib/components/admin-settings/LibrarySettings.svelte';
   import LoggingSettings from '$lib/components/admin-settings/LoggingSettings.svelte';
   import MachineLearningSettings from '$lib/components/admin-settings/MachineLearningSettings.svelte';
+  import MaintenanceSettings from '$lib/components/admin-settings/MaintenanceSettings.svelte';
   import MapSettings from '$lib/components/admin-settings/MapSettings.svelte';
   import MetadataSettings from '$lib/components/admin-settings/MetadataSettings.svelte';
   import NewVersionCheckSettings from '$lib/components/admin-settings/NewVersionCheckSettings.svelte';
@@ -40,6 +41,7 @@
     mdiLockOutline,
     mdiMapMarkerOutline,
     mdiPaletteOutline,
+    mdiRestore,
     mdiRobotOutline,
     mdiServerOutline,
     mdiSync,
@@ -113,6 +115,13 @@
       key: 'machine-learning',
       icon: mdiRobotOutline,
     },
+    {
+      component: MaintenanceSettings,
+      title: $t('admin.maintenance_settings'),
+      subtitle: $t('admin.maintenance_settings_description'),
+      key: 'maintenance',
+      icon: mdiRestore,
+    },
     {
       component: MapSettings,
       title: $t('admin.map_gps_settings'),

web/src/routes/maintenance/+page.svelte

@@ -0,0 +1,55 @@
+<script lang="ts">
+  import AuthPageLayout from '$lib/components/layouts/AuthPageLayout.svelte';
+  import FormatMessage from '$lib/elements/FormatMessage.svelte';
+  import { maintenanceAuth } from '$lib/stores/maintenance.store';
+  import { handleError } from '$lib/utils/handle-error';
+  import { MaintenanceAction, setMaintenanceMode } from '@immich/sdk';
+  import { Button, Heading, Link } from '@immich/ui';
+  import { t } from 'svelte-i18n';
+
+  // strip token from URL after load
+  const url = new URL(location.href);
+  if (url.searchParams.get('token')) {
+    url.searchParams.delete('token');
+    history.replaceState({}, document.title, url);
+  }
+
+  async function end() {
+    try {
+      await setMaintenanceMode({
+        setMaintenanceModeDto: {
+          action: MaintenanceAction.End,
+        },
+      });
+    } catch (error) {
+      handleError(error, $t('maintenance_end_error'));
+    }
+  }
+</script>
+
+<AuthPageLayout>
+  <div class="flex flex-col place-items-center text-center gap-4">
+    <Heading size="large" color="primary" tag="h1">{$t('maintenance_title')}</Heading>
+    <p>
+      <FormatMessage key="maintenance_description">
+        {#snippet children({ tag, message })}
+          {#if tag === 'link'}
+            <Link href="https://docs.immich.app/administration/maintenance-mode">
+              {message}
+            </Link>
+          {/if}
+        {/snippet}
+      </FormatMessage>
+    </p>
+    {#if $maintenanceAuth}
+      <p>
+        {$t('maintenance_logged_in_as', {
+          values: {
+            user: $maintenanceAuth.username,
+          },
+        })}
+      </p>
+      <Button onclick={end}>{$t('maintenance_end')}</Button>
+    {/if}
+  </div>
+</AuthPageLayout>

web/src/routes/maintenance/+page.ts

@@ -0,0 +1,6 @@
+import { loadMaintenanceAuth } from '$lib/utils/maintenance';
+import type { PageLoad } from '../admin/$types';
+
+export const load = (async () => {
+  await loadMaintenanceAuth();
+}) satisfies PageLoad;