feat: maintenance mode (#23431)

* feat: add a `maintenance.enabled` config flag

* feat: implement graceful restart
feat: restart when maintenance config is toggled

* feat: boot a stripped down maintenance api if enabled

* feat: cli command to toggle maintenance mode

* chore: fallback IMMICH_SERVER_URL environment variable in process

* chore: add additional routes to maintenance controller

* fix: don't wait for nest application to close to finish request response

* chore: add a failsafe on restart to prevent other exit codes from preventing restart

* feat: redirect into/from maintenance page

* refactor: use system metadata for maintenance status

* refactor: wait on WebSocket connection to refresh

* feat: broadcast websocket event on server restart
refactor: listen to WS instead of polling

* refactor: bubble up maintenance information instead of hijacking in fetch function
feat: show modal when server is restarting

* chore: increase timeout for ungraceful restart

* refactor: deduplicate code between api/maintenance workers

* fix: skip config check if database is not initialised

* fix: add `maintenanceMode` field to system config test

* refactor: move maintenance resolution code to static method in service

* chore: clean up linter issues

* chore: generate dart openapi

* refactor: use try{} block for maintenance mode check

* fix: logic error in server redirect

* chore: include `maintenanceMode` key in e2e test

* chore: add i18n entries for maintenance screens

* chore: remove negated condition from hook

* fix: should set default value not override in service

* fix: minor error in page

* feat: initial draft of maintenance module, repo., worker controller, worker service

* refactor: move broadcast code into notification service

* chore: connect websocket on client if in maintenance

* chore: set maintenance module app name

* refactor: rename repository to include worker
chore: configure websocket adapter

* feat: reimplement maintenance mode exit with new module

* refactor: add a constant enum for ExitCode

* refactor: remove redundant route for maintenance

* refactor: only spin up kysely on boot (rather than a Nest app)

* refactor(web): move redirect logic into +layout file where modal is setup

* feat: add Maintenance permission

* refactor: merge common code between api/maintenance

* fix: propagate changes from the CLI to servers

* feat: maintenance authentication guard

* refactor: unify maintenance code into repository
feat: add a step to generate maintenance mode token

* feat: jwt auth for maintenance

* refactor: switch from nest jwt to just jsonwebtokens

* feat: log into maintenance mode from CLI command

* refactor: use `secret` instead of `token` in jwt terminology
chore: log maintenance mode login URL on boot
chore: don't make CLI actions reload if already in target state

* docs: initial draft for maintenance mode page

* refactor: always validate the maintenance auth on the server

* feat: add a link to maintenance mode documentation

* feat: redirect users back to the last page they were on when exiting maintenance

* refactor: provide closeFn in both maintenance repos.

* refactor: ensure the user is also redirected by the server

* chore: swap jsonwebtoken for jose

* refactor: introduce AppRestartEvent w/o secret passing

* refactor: use navigation goto

* refactor: use `continue` instead of `next`

* chore: lint fixes for server

* chore: lint fixes for web

* test: add mock for maintenance repository

* test: add base service dependency to maintenance

* chore: remove @types/jsonwebtoken

* refactor: close database connection after startup check

* refactor: use `request#auth` key

* refactor: use service instead of repository
chore: read token from cookie if possible
chore: rename client event to AppRestartV1

* refactor: more concise redirect logic on web

* refactor: move redirect check into utils
refactor: update translation strings to be more sensible

* refactor: always validate login (i.e. check cookie)

* refactor: lint, open-api, remove old dto

* refactor: encode at point of usage

* refactor: remove business logic from repositories

* chore: fix server/web lints

* refactor: remove repository mock

* chore: fix formatting

* test: write service mocks for maintenance mode

* test: write cli service tests

* fix: catch errors when closing app

* fix: always report no maintenance when usual API is available

* test: api e2e maintenance spec

* chore: add response builder

* chore: add helper to set maint. auth cookie

* feat: add SSR to maintenance API

* test(e2e): write web spec for maintenance

* chore: clean up lint issues

* chore: format files

* feat: perform 302 redirect at server level during maintenance

* fix: keep trying to stop immich until it succeeds (CLI issue)

* chore: lint/format

* refactor: annotate references to other services in worker service

* chore: lint

* refactor: remove unnecessary await

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>

* refactor: move static methods into util

* refactor: assert secret exists in maintenance worker

* refactor: remove assertion which isn't necessary anymore

* refactor: remove assertion

* refactor: remove outer try {} catch block from loadMaintenanceAuth

* refactor: undo earlier change to vite.config.ts

* chore: update tests due to refactors

* revert: vite.config.ts

* test: expect string jwt

* chore: move blanket exceptions into controllers

* test: update tests according with last change

* refactor: use respondWithCookie
refactor: merge start/end into one route
refactor: rename MaintenanceRepository to AppRepository
chore: use new ApiTag/Endpoint
refactor: apply other requested changes

* chore: regenerate openapi

* chore: lint/format

* chore: remove secureOnly for maint. cookie

* refactor: move maintenance worker code into src/maintenance\nfix: various test fixes

* refactor: use `action` property for setting maint. mode

* refactor: remove Websocket#restartApp in favour of individual methods

* chore: incomplete commit

* chore: remove stray log

* fix: call exitApp from maintenance worker on exit

* fix: add app repository mock

* fix: ensure maintenance cookies are secure

* fix: run playwright tests over secure context (localhost)

* test: update other references to 127.0.0.1

* refactor: use serverSideEmitWithAck

* chore: correct the logic in tryTerminate

* test: juggle cookies ourselves

* chore: fix lint error for e2e spec

* chore: format e2e test

* fix: set cookie secure/non-secure depending on context

* chore: format files

---------

Co-authored-by: Daniel Dietzler <36593685+danieldietzler@users.noreply.github.com>

mobile/openapi/lib/model/maintenance_action.dart

@@ -0,0 +1,85 @@
+//
+// AUTO-GENERATED FILE, DO NOT MODIFY!
+//
+// @dart=2.18
+
+// ignore_for_file: unused_element, unused_import
+// ignore_for_file: always_put_required_named_parameters_first
+// ignore_for_file: constant_identifier_names
+// ignore_for_file: lines_longer_than_80_chars
+
+part of openapi.api;
+
+
+class MaintenanceAction {
+  /// Instantiate a new enum with the provided [value].
+  const MaintenanceAction._(this.value);
+
+  /// The underlying value of this enum member.
+  final String value;
+
+  @override
+  String toString() => value;
+
+  String toJson() => value;
+
+  static const start = MaintenanceAction._(r'start');
+  static const end = MaintenanceAction._(r'end');
+
+  /// List of all possible values in this [enum][MaintenanceAction].
+  static const values = <MaintenanceAction>[
+    start,
+    end,
+  ];
+
+  static MaintenanceAction? fromJson(dynamic value) => MaintenanceActionTypeTransformer().decode(value);
+
+  static List<MaintenanceAction> listFromJson(dynamic json, {bool growable = false,}) {
+    final result = <MaintenanceAction>[];
+    if (json is List && json.isNotEmpty) {
+      for (final row in json) {
+        final value = MaintenanceAction.fromJson(row);
+        if (value != null) {
+          result.add(value);
+        }
+      }
+    }
+    return result.toList(growable: growable);
+  }
+}
+
+/// Transformation class that can [encode] an instance of [MaintenanceAction] to String,
+/// and [decode] dynamic data back to [MaintenanceAction].
+class MaintenanceActionTypeTransformer {
+  factory MaintenanceActionTypeTransformer() => _instance ??= const MaintenanceActionTypeTransformer._();
+
+  const MaintenanceActionTypeTransformer._();
+
+  String encode(MaintenanceAction data) => data.value;
+
+  /// Decodes a [dynamic value][data] to a MaintenanceAction.
+  ///
+  /// If [allowNull] is true and the [dynamic value][data] cannot be decoded successfully,
+  /// then null is returned. However, if [allowNull] is false and the [dynamic value][data]
+  /// cannot be decoded successfully, then an [UnimplementedError] is thrown.
+  ///
+  /// The [allowNull] is very handy when an API changes and a new enum value is added or removed,
+  /// and users are still using an old app with the old code.
+  MaintenanceAction? decode(dynamic data, {bool allowNull = true}) {
+    if (data != null) {
+      switch (data) {
+        case r'start': return MaintenanceAction.start;
+        case r'end': return MaintenanceAction.end;
+        default:
+          if (!allowNull) {
+            throw ArgumentError('Unknown enum value to decode: $data');
+          }
+      }
+    }
+    return null;
+  }
+
+  /// Singleton [MaintenanceActionTypeTransformer] instance.
+  static MaintenanceActionTypeTransformer? _instance;
+}
+

mobile/openapi/lib/model/maintenance_auth_dto.dart

@@ -0,0 +1,99 @@
+//
+// AUTO-GENERATED FILE, DO NOT MODIFY!
+//
+// @dart=2.18
+
+// ignore_for_file: unused_element, unused_import
+// ignore_for_file: always_put_required_named_parameters_first
+// ignore_for_file: constant_identifier_names
+// ignore_for_file: lines_longer_than_80_chars
+
+part of openapi.api;
+
+class MaintenanceAuthDto {
+  /// Returns a new [MaintenanceAuthDto] instance.
+  MaintenanceAuthDto({
+    required this.username,
+  });
+
+  String username;
+
+  @override
+  bool operator ==(Object other) => identical(this, other) || other is MaintenanceAuthDto &&
+    other.username == username;
+
+  @override
+  int get hashCode =>
+    // ignore: unnecessary_parenthesis
+    (username.hashCode);
+
+  @override
+  String toString() => 'MaintenanceAuthDto[username=$username]';
+
+  Map<String, dynamic> toJson() {
+    final json = <String, dynamic>{};
+      json[r'username'] = this.username;
+    return json;
+  }
+
+  /// Returns a new [MaintenanceAuthDto] instance and imports its values from
+  /// [value] if it's a [Map], null otherwise.
+  // ignore: prefer_constructors_over_static_methods
+  static MaintenanceAuthDto? fromJson(dynamic value) {
+    upgradeDto(value, "MaintenanceAuthDto");
+    if (value is Map) {
+      final json = value.cast<String, dynamic>();
+
+      return MaintenanceAuthDto(
+        username: mapValueOfType<String>(json, r'username')!,
+      );
+    }
+    return null;
+  }
+
+  static List<MaintenanceAuthDto> listFromJson(dynamic json, {bool growable = false,}) {
+    final result = <MaintenanceAuthDto>[];
+    if (json is List && json.isNotEmpty) {
+      for (final row in json) {
+        final value = MaintenanceAuthDto.fromJson(row);
+        if (value != null) {
+          result.add(value);
+        }
+      }
+    }
+    return result.toList(growable: growable);
+  }
+
+  static Map<String, MaintenanceAuthDto> mapFromJson(dynamic json) {
+    final map = <String, MaintenanceAuthDto>{};
+    if (json is Map && json.isNotEmpty) {
+      json = json.cast<String, dynamic>(); // ignore: parameter_assignments
+      for (final entry in json.entries) {
+        final value = MaintenanceAuthDto.fromJson(entry.value);
+        if (value != null) {
+          map[entry.key] = value;
+        }
+      }
+    }
+    return map;
+  }
+
+  // maps a json object with a list of MaintenanceAuthDto-objects as value to a dart map
+  static Map<String, List<MaintenanceAuthDto>> mapListFromJson(dynamic json, {bool growable = false,}) {
+    final map = <String, List<MaintenanceAuthDto>>{};
+    if (json is Map && json.isNotEmpty) {
+      // ignore: parameter_assignments
+      json = json.cast<String, dynamic>();
+      for (final entry in json.entries) {
+        map[entry.key] = MaintenanceAuthDto.listFromJson(entry.value, growable: growable,);
+      }
+    }
+    return map;
+  }
+
+  /// The list of required keys that must be present in a JSON.
+  static const requiredKeys = <String>{
+    'username',
+  };
+}
+

mobile/openapi/lib/model/maintenance_login_dto.dart

@@ -0,0 +1,108 @@
+//
+// AUTO-GENERATED FILE, DO NOT MODIFY!
+//
+// @dart=2.18
+
+// ignore_for_file: unused_element, unused_import
+// ignore_for_file: always_put_required_named_parameters_first
+// ignore_for_file: constant_identifier_names
+// ignore_for_file: lines_longer_than_80_chars
+
+part of openapi.api;
+
+class MaintenanceLoginDto {
+  /// Returns a new [MaintenanceLoginDto] instance.
+  MaintenanceLoginDto({
+    this.token,
+  });
+
+  ///
+  /// Please note: This property should have been non-nullable! Since the specification file
+  /// does not include a default value (using the "default:" property), however, the generated
+  /// source code must fall back to having a nullable type.
+  /// Consider adding a "default:" property in the specification file to hide this note.
+  ///
+  String? token;
+
+  @override
+  bool operator ==(Object other) => identical(this, other) || other is MaintenanceLoginDto &&
+    other.token == token;
+
+  @override
+  int get hashCode =>
+    // ignore: unnecessary_parenthesis
+    (token == null ? 0 : token!.hashCode);
+
+  @override
+  String toString() => 'MaintenanceLoginDto[token=$token]';
+
+  Map<String, dynamic> toJson() {
+    final json = <String, dynamic>{};
+    if (this.token != null) {
+      json[r'token'] = this.token;
+    } else {
+    //  json[r'token'] = null;
+    }
+    return json;
+  }
+
+  /// Returns a new [MaintenanceLoginDto] instance and imports its values from
+  /// [value] if it's a [Map], null otherwise.
+  // ignore: prefer_constructors_over_static_methods
+  static MaintenanceLoginDto? fromJson(dynamic value) {
+    upgradeDto(value, "MaintenanceLoginDto");
+    if (value is Map) {
+      final json = value.cast<String, dynamic>();
+
+      return MaintenanceLoginDto(
+        token: mapValueOfType<String>(json, r'token'),
+      );
+    }
+    return null;
+  }
+
+  static List<MaintenanceLoginDto> listFromJson(dynamic json, {bool growable = false,}) {
+    final result = <MaintenanceLoginDto>[];
+    if (json is List && json.isNotEmpty) {
+      for (final row in json) {
+        final value = MaintenanceLoginDto.fromJson(row);
+        if (value != null) {
+          result.add(value);
+        }
+      }
+    }
+    return result.toList(growable: growable);
+  }
+
+  static Map<String, MaintenanceLoginDto> mapFromJson(dynamic json) {
+    final map = <String, MaintenanceLoginDto>{};
+    if (json is Map && json.isNotEmpty) {
+      json = json.cast<String, dynamic>(); // ignore: parameter_assignments
+      for (final entry in json.entries) {
+        final value = MaintenanceLoginDto.fromJson(entry.value);
+        if (value != null) {
+          map[entry.key] = value;
+        }
+      }
+    }
+    return map;
+  }
+
+  // maps a json object with a list of MaintenanceLoginDto-objects as value to a dart map
+  static Map<String, List<MaintenanceLoginDto>> mapListFromJson(dynamic json, {bool growable = false,}) {
+    final map = <String, List<MaintenanceLoginDto>>{};
+    if (json is Map && json.isNotEmpty) {
+      // ignore: parameter_assignments
+      json = json.cast<String, dynamic>();
+      for (final entry in json.entries) {
+        map[entry.key] = MaintenanceLoginDto.listFromJson(entry.value, growable: growable,);
+      }
+    }
+    return map;
+  }
+
+  /// The list of required keys that must be present in a JSON.
+  static const requiredKeys = <String>{
+  };
+}
+

mobile/openapi/lib/model/permission.dart

@@ -73,6 +73,7 @@ class Permission {
   static const libraryPeriodStatistics = Permission._(r'library.statistics');
   static const timelinePeriodRead = Permission._(r'timeline.read');
   static const timelinePeriodDownload = Permission._(r'timeline.download');
+  static const maintenance = Permission._(r'maintenance');
   static const memoryPeriodCreate = Permission._(r'memory.create');
   static const memoryPeriodRead = Permission._(r'memory.read');
   static const memoryPeriodUpdate = Permission._(r'memory.update');
@@ -214,6 +215,7 @@ class Permission {
     libraryPeriodStatistics,
     timelinePeriodRead,
     timelinePeriodDownload,
+    maintenance,
     memoryPeriodCreate,
     memoryPeriodRead,
     memoryPeriodUpdate,
@@ -390,6 +392,7 @@ class PermissionTypeTransformer {
         case r'library.statistics': return Permission.libraryPeriodStatistics;
         case r'timeline.read': return Permission.timelinePeriodRead;
         case r'timeline.download': return Permission.timelinePeriodDownload;
+        case r'maintenance': return Permission.maintenance;
         case r'memory.create': return Permission.memoryPeriodCreate;
         case r'memory.read': return Permission.memoryPeriodRead;
         case r'memory.update': return Permission.memoryPeriodUpdate;

mobile/openapi/lib/model/server_config_dto.dart

@@ -17,6 +17,7 @@ class ServerConfigDto {
     required this.isInitialized,
     required this.isOnboarded,
     required this.loginPageMessage,
+    required this.maintenanceMode,
     required this.mapDarkStyleUrl,
     required this.mapLightStyleUrl,
     required this.oauthButtonText,
@@ -33,6 +34,8 @@ class ServerConfigDto {
 
   String loginPageMessage;
 
+  bool maintenanceMode;
+
   String mapDarkStyleUrl;
 
   String mapLightStyleUrl;
@@ -51,6 +54,7 @@ class ServerConfigDto {
     other.isInitialized == isInitialized &&
     other.isOnboarded == isOnboarded &&
     other.loginPageMessage == loginPageMessage &&
+    other.maintenanceMode == maintenanceMode &&
     other.mapDarkStyleUrl == mapDarkStyleUrl &&
     other.mapLightStyleUrl == mapLightStyleUrl &&
     other.oauthButtonText == oauthButtonText &&
@@ -65,6 +69,7 @@ class ServerConfigDto {
     (isInitialized.hashCode) +
     (isOnboarded.hashCode) +
     (loginPageMessage.hashCode) +
+    (maintenanceMode.hashCode) +
     (mapDarkStyleUrl.hashCode) +
     (mapLightStyleUrl.hashCode) +
     (oauthButtonText.hashCode) +
@@ -73,7 +78,7 @@ class ServerConfigDto {
     (userDeleteDelay.hashCode);
 
   @override
-  String toString() => 'ServerConfigDto[externalDomain=$externalDomain, isInitialized=$isInitialized, isOnboarded=$isOnboarded, loginPageMessage=$loginPageMessage, mapDarkStyleUrl=$mapDarkStyleUrl, mapLightStyleUrl=$mapLightStyleUrl, oauthButtonText=$oauthButtonText, publicUsers=$publicUsers, trashDays=$trashDays, userDeleteDelay=$userDeleteDelay]';
+  String toString() => 'ServerConfigDto[externalDomain=$externalDomain, isInitialized=$isInitialized, isOnboarded=$isOnboarded, loginPageMessage=$loginPageMessage, maintenanceMode=$maintenanceMode, mapDarkStyleUrl=$mapDarkStyleUrl, mapLightStyleUrl=$mapLightStyleUrl, oauthButtonText=$oauthButtonText, publicUsers=$publicUsers, trashDays=$trashDays, userDeleteDelay=$userDeleteDelay]';
 
   Map<String, dynamic> toJson() {
     final json = <String, dynamic>{};
@@ -81,6 +86,7 @@ class ServerConfigDto {
       json[r'isInitialized'] = this.isInitialized;
       json[r'isOnboarded'] = this.isOnboarded;
       json[r'loginPageMessage'] = this.loginPageMessage;
+      json[r'maintenanceMode'] = this.maintenanceMode;
       json[r'mapDarkStyleUrl'] = this.mapDarkStyleUrl;
       json[r'mapLightStyleUrl'] = this.mapLightStyleUrl;
       json[r'oauthButtonText'] = this.oauthButtonText;
@@ -103,6 +109,7 @@ class ServerConfigDto {
         isInitialized: mapValueOfType<bool>(json, r'isInitialized')!,
         isOnboarded: mapValueOfType<bool>(json, r'isOnboarded')!,
         loginPageMessage: mapValueOfType<String>(json, r'loginPageMessage')!,
+        maintenanceMode: mapValueOfType<bool>(json, r'maintenanceMode')!,
         mapDarkStyleUrl: mapValueOfType<String>(json, r'mapDarkStyleUrl')!,
         mapLightStyleUrl: mapValueOfType<String>(json, r'mapLightStyleUrl')!,
         oauthButtonText: mapValueOfType<String>(json, r'oauthButtonText')!,
@@ -160,6 +167,7 @@ class ServerConfigDto {
     'isInitialized',
     'isOnboarded',
     'loginPageMessage',
+    'maintenanceMode',
     'mapDarkStyleUrl',
     'mapLightStyleUrl',
     'oauthButtonText',

mobile/openapi/lib/model/set_maintenance_mode_dto.dart

@@ -0,0 +1,99 @@
+//
+// AUTO-GENERATED FILE, DO NOT MODIFY!
+//
+// @dart=2.18
+
+// ignore_for_file: unused_element, unused_import
+// ignore_for_file: always_put_required_named_parameters_first
+// ignore_for_file: constant_identifier_names
+// ignore_for_file: lines_longer_than_80_chars
+
+part of openapi.api;
+
+class SetMaintenanceModeDto {
+  /// Returns a new [SetMaintenanceModeDto] instance.
+  SetMaintenanceModeDto({
+    required this.action,
+  });
+
+  MaintenanceAction action;
+
+  @override
+  bool operator ==(Object other) => identical(this, other) || other is SetMaintenanceModeDto &&
+    other.action == action;
+
+  @override
+  int get hashCode =>
+    // ignore: unnecessary_parenthesis
+    (action.hashCode);
+
+  @override
+  String toString() => 'SetMaintenanceModeDto[action=$action]';
+
+  Map<String, dynamic> toJson() {
+    final json = <String, dynamic>{};
+      json[r'action'] = this.action;
+    return json;
+  }
+
+  /// Returns a new [SetMaintenanceModeDto] instance and imports its values from
+  /// [value] if it's a [Map], null otherwise.
+  // ignore: prefer_constructors_over_static_methods
+  static SetMaintenanceModeDto? fromJson(dynamic value) {
+    upgradeDto(value, "SetMaintenanceModeDto");
+    if (value is Map) {
+      final json = value.cast<String, dynamic>();
+
+      return SetMaintenanceModeDto(
+        action: MaintenanceAction.fromJson(json[r'action'])!,
+      );
+    }
+    return null;
+  }
+
+  static List<SetMaintenanceModeDto> listFromJson(dynamic json, {bool growable = false,}) {
+    final result = <SetMaintenanceModeDto>[];
+    if (json is List && json.isNotEmpty) {
+      for (final row in json) {
+        final value = SetMaintenanceModeDto.fromJson(row);
+        if (value != null) {
+          result.add(value);
+        }
+      }
+    }
+    return result.toList(growable: growable);
+  }
+
+  static Map<String, SetMaintenanceModeDto> mapFromJson(dynamic json) {
+    final map = <String, SetMaintenanceModeDto>{};
+    if (json is Map && json.isNotEmpty) {
+      json = json.cast<String, dynamic>(); // ignore: parameter_assignments
+      for (final entry in json.entries) {
+        final value = SetMaintenanceModeDto.fromJson(entry.value);
+        if (value != null) {
+          map[entry.key] = value;
+        }
+      }
+    }
+    return map;
+  }
+
+  // maps a json object with a list of SetMaintenanceModeDto-objects as value to a dart map
+  static Map<String, List<SetMaintenanceModeDto>> mapListFromJson(dynamic json, {bool growable = false,}) {
+    final map = <String, List<SetMaintenanceModeDto>>{};
+    if (json is Map && json.isNotEmpty) {
+      // ignore: parameter_assignments
+      json = json.cast<String, dynamic>();
+      for (final entry in json.entries) {
+        map[entry.key] = SetMaintenanceModeDto.listFromJson(entry.value, growable: growable,);
+      }
+    }
+    return map;
+  }
+
+  /// The list of required keys that must be present in a JSON.
+  static const requiredKeys = <String>{
+    'action',
+  };
+}
+