feat(server): near-duplicate detection (#8228)

* duplicate detection job, entity, config

* queueing

* job panel, update api

* use embedding in db instead of fetching

* disable concurrency

* only queue visible assets

* handle multiple duplicateIds

* update concurrent queue check

* add provider

* add web placeholder, server endpoint, migration, various fixes

* update sql

* select embedding by default

* rename variable

* simplify

* remove separate entity, handle re-running with different threshold, set default back to 0.02

* fix tests

* add tests

* add index to entity

* formatting

* update asset mock

* fix `upsertJobStatus` signature

* update sql

* formatting

* default to 0.03

* optimize clustering

* use asset's `duplicateId` if present

* update sql

* update tests

* expose admin setting

* refactor

* formatting

* skip if ml is disabled

* debug trash e2e

* remove from web

* remove from sidebar

* test if ml is disabled

* update sql

* separate duplicate detection from clip in config, disable by default for now

* fix doc

* lower minimum `maxDistance`

* update api

* Add and Use Duplicate Detection Feature Flag (#9364)

* Add Duplicate Detection Flag

* Use Duplicate Detection Flag

* Attempt Fixes for Failing Checks

* lower minimum `maxDistance`

* fix tests

---------

Co-authored-by: mertalev <101130780+mertalev@users.noreply.github.com>

* chore: fixes and additions after rebase

* chore: update api (remove new Role enum)

* fix: left join smart search so getAll works without machine learning

* test: trash e2e go back to checking length of assets is zero

* chore: regen api after rebase

* test: fix tests after rebase

* redundant join

---------

Co-authored-by: Nicholas Flamy <30300649+NicholasFlamy@users.noreply.github.com>
Co-authored-by: Zack Pollard <zackpollard@ymail.com>
Co-authored-by: Zack Pollard <zack@futo.org>

mobile/openapi/.openapi-generator/FILES

@@ -68,6 +68,7 @@ doc/DownloadApi.md
 doc/DownloadArchiveInfo.md
 doc/DownloadInfoDto.md
 doc/DownloadResponseDto.md
+doc/DuplicateDetectionConfig.md
 doc/EntityType.md
 doc/ExifResponseDto.md
 doc/FaceApi.md
@@ -308,6 +309,7 @@ lib/model/delete_user_dto.dart
 lib/model/download_archive_info.dart
 lib/model/download_info_dto.dart
 lib/model/download_response_dto.dart
+lib/model/duplicate_detection_config.dart
 lib/model/entity_type.dart
 lib/model/exif_response_dto.dart
 lib/model/face_dto.dart
@@ -501,6 +503,7 @@ test/download_api_test.dart
 test/download_archive_info_test.dart
 test/download_info_dto_test.dart
 test/download_response_dto_test.dart
+test/duplicate_detection_config_test.dart
 test/entity_type_test.dart
 test/exif_response_dto_test.dart
 test/face_api_test.dart

mobile/openapi/README.md

@@ -98,6 +98,7 @@ Class | Method | HTTP request | Description
 *AssetApi* | [**deleteAssets**](doc//AssetApi.md#deleteassets) | **DELETE** /asset | 
 *AssetApi* | [**getAllAssets**](doc//AssetApi.md#getallassets) | **GET** /asset | 
 *AssetApi* | [**getAllUserAssetsByDeviceId**](doc//AssetApi.md#getalluserassetsbydeviceid) | **GET** /asset/device/{deviceId} | 
+*AssetApi* | [**getAssetDuplicates**](doc//AssetApi.md#getassetduplicates) | **GET** /asset/duplicates | 
 *AssetApi* | [**getAssetInfo**](doc//AssetApi.md#getassetinfo) | **GET** /asset/{id} | 
 *AssetApi* | [**getAssetStatistics**](doc//AssetApi.md#getassetstatistics) | **GET** /asset/statistics | 
 *AssetApi* | [**getAssetThumbnail**](doc//AssetApi.md#getassetthumbnail) | **GET** /asset/thumbnail/{id} | 
@@ -282,6 +283,7 @@ Class | Method | HTTP request | Description
  - [DownloadArchiveInfo](doc//DownloadArchiveInfo.md)
  - [DownloadInfoDto](doc//DownloadInfoDto.md)
  - [DownloadResponseDto](doc//DownloadResponseDto.md)
+ - [DuplicateDetectionConfig](doc//DuplicateDetectionConfig.md)
  - [EntityType](doc//EntityType.md)
  - [ExifResponseDto](doc//ExifResponseDto.md)
  - [FaceDto](doc//FaceDto.md)

mobile/openapi/doc/AllJobStatusResponseDto.md

@@ -9,6 +9,7 @@ import 'package:openapi/api.dart';
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
 **backgroundTask** | [**JobStatusDto**](JobStatusDto.md) |  | 
+**duplicateDetection** | [**JobStatusDto**](JobStatusDto.md) |  | 
 **faceDetection** | [**JobStatusDto**](JobStatusDto.md) |  | 
 **facialRecognition** | [**JobStatusDto**](JobStatusDto.md) |  | 
 **library_** | [**JobStatusDto**](JobStatusDto.md) |  | 

mobile/openapi/doc/AssetApi.md

@@ -14,6 +14,7 @@ Method | HTTP request | Description
 [**deleteAssets**](AssetApi.md#deleteassets) | **DELETE** /asset | 
 [**getAllAssets**](AssetApi.md#getallassets) | **GET** /asset | 
 [**getAllUserAssetsByDeviceId**](AssetApi.md#getalluserassetsbydeviceid) | **GET** /asset/device/{deviceId} | 
+[**getAssetDuplicates**](AssetApi.md#getassetduplicates) | **GET** /asset/duplicates | 
 [**getAssetInfo**](AssetApi.md#getassetinfo) | **GET** /asset/{id} | 
 [**getAssetStatistics**](AssetApi.md#getassetstatistics) | **GET** /asset/statistics | 
 [**getAssetThumbnail**](AssetApi.md#getassetthumbnail) | **GET** /asset/thumbnail/{id} | 
@@ -324,6 +325,43 @@ Name | Type | Description  | Notes
 
 [[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
 
+# **getAssetDuplicates**
+> List<AssetResponseDto> getAssetDuplicates()
+
+
+
+### Example
+```dart
+import 'package:openapi/api.dart';
+
+final api_instance = AssetApi();
+
+try {
+    final result = api_instance.getAssetDuplicates();
+    print(result);
+} catch (e) {
+    print('Exception when calling AssetApi->getAssetDuplicates: $e\n');
+}
+```
+
+### Parameters
+This endpoint does not need any parameter.
+
+### Return type
+
+[**List<AssetResponseDto>**](AssetResponseDto.md)
+
+### Authorization
+
+No authorization required
+
+### HTTP request headers
+
+ - **Content-Type**: Not defined
+ - **Accept**: application/json
+
+[[Back to top]](#) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to Model list]](../README.md#documentation-for-models) [[Back to README]](../README.md)
+
 # **getAssetInfo**
 > AssetResponseDto getAssetInfo(id, key)
 

mobile/openapi/doc/DuplicateDetectionConfig.md

@@ -0,0 +1,16 @@
+# openapi.model.DuplicateDetectionConfig
+
+## Load the model package
+```dart
+import 'package:openapi/api.dart';
+```
+
+## Properties
+Name | Type | Description | Notes
+------------ | ------------- | ------------- | -------------
+**enabled** | **bool** |  | 
+**maxDistance** | **double** |  | 
+
+[[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md)
+
+

mobile/openapi/doc/ServerFeaturesDto.md

@@ -9,6 +9,7 @@ import 'package:openapi/api.dart';
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
 **configFile** | **bool** |  | 
+**duplicateDetection** | **bool** |  | 
 **email** | **bool** |  | 
 **facialRecognition** | **bool** |  | 
 **map** | **bool** |  | 

mobile/openapi/doc/SystemConfigMachineLearningDto.md

@@ -9,6 +9,7 @@ import 'package:openapi/api.dart';
 Name | Type | Description | Notes
 ------------ | ------------- | ------------- | -------------
 **clip** | [**CLIPConfig**](CLIPConfig.md) |  | 
+**duplicateDetection** | [**DuplicateDetectionConfig**](DuplicateDetectionConfig.md) |  | 
 **enabled** | **bool** |  | 
 **facialRecognition** | [**RecognitionConfig**](RecognitionConfig.md) |  | 
 **url** | **String** |  | 

mobile/openapi/lib/api.dart

@@ -114,6 +114,7 @@ part 'model/delete_user_dto.dart';
 part 'model/download_archive_info.dart';
 part 'model/download_info_dto.dart';
 part 'model/download_response_dto.dart';
+part 'model/duplicate_detection_config.dart';
 part 'model/entity_type.dart';
 part 'model/exif_response_dto.dart';
 part 'model/face_dto.dart';

mobile/openapi/lib/api/asset_api.dart

@@ -326,6 +326,50 @@ class AssetApi {
     return null;
   }
 
+  /// Performs an HTTP 'GET /asset/duplicates' operation and returns the [Response].
+  Future<Response> getAssetDuplicatesWithHttpInfo() async {
+    // ignore: prefer_const_declarations
+    final path = r'/asset/duplicates';
+
+    // ignore: prefer_final_locals
+    Object? postBody;
+
+    final queryParams = <QueryParam>[];
+    final headerParams = <String, String>{};
+    final formParams = <String, String>{};
+
+    const contentTypes = <String>[];
+
+
+    return apiClient.invokeAPI(
+      path,
+      'GET',
+      queryParams,
+      postBody,
+      headerParams,
+      formParams,
+      contentTypes.isEmpty ? null : contentTypes.first,
+    );
+  }
+
+  Future<List<AssetResponseDto>?> getAssetDuplicates() async {
+    final response = await getAssetDuplicatesWithHttpInfo();
+    if (response.statusCode >= HttpStatus.badRequest) {
+      throw ApiException(response.statusCode, await _decodeBodyBytes(response));
+    }
+    // When a remote server returns no body with a status of 204, we shall not decode it.
+    // At the time of writing this, `dart:convert` will throw an "Unexpected end of input"
+    // FormatException when trying to decode an empty string.
+    if (response.body.isNotEmpty && response.statusCode != HttpStatus.noContent) {
+      final responseBody = await _decodeBodyBytes(response);
+      return (await apiClient.deserializeAsync(responseBody, 'List<AssetResponseDto>') as List)
+        .cast<AssetResponseDto>()
+        .toList(growable: false);
+
+    }
+    return null;
+  }
+
   /// Performs an HTTP 'GET /asset/{id}' operation and returns the [Response].
   /// Parameters:
   ///

mobile/openapi/lib/api_client.dart

@@ -298,6 +298,8 @@ class ApiClient {
           return DownloadInfoDto.fromJson(value);
         case 'DownloadResponseDto':
           return DownloadResponseDto.fromJson(value);
+        case 'DuplicateDetectionConfig':
+          return DuplicateDetectionConfig.fromJson(value);
         case 'EntityType':
           return EntityTypeTypeTransformer().decode(value);
         case 'ExifResponseDto':

mobile/openapi/lib/model/all_job_status_response_dto.dart

@@ -14,6 +14,7 @@ class AllJobStatusResponseDto {
   /// Returns a new [AllJobStatusResponseDto] instance.
   AllJobStatusResponseDto({
     required this.backgroundTask,
+    required this.duplicateDetection,
     required this.faceDetection,
     required this.facialRecognition,
     required this.library_,
@@ -30,6 +31,8 @@ class AllJobStatusResponseDto {
 
   JobStatusDto backgroundTask;
 
+  JobStatusDto duplicateDetection;
+
   JobStatusDto faceDetection;
 
   JobStatusDto facialRecognition;
@@ -57,6 +60,7 @@ class AllJobStatusResponseDto {
   @override
   bool operator ==(Object other) => identical(this, other) || other is AllJobStatusResponseDto &&
     other.backgroundTask == backgroundTask &&
+    other.duplicateDetection == duplicateDetection &&
     other.faceDetection == faceDetection &&
     other.facialRecognition == facialRecognition &&
     other.library_ == library_ &&
@@ -74,6 +78,7 @@ class AllJobStatusResponseDto {
   int get hashCode =>
     // ignore: unnecessary_parenthesis
     (backgroundTask.hashCode) +
+    (duplicateDetection.hashCode) +
     (faceDetection.hashCode) +
     (facialRecognition.hashCode) +
     (library_.hashCode) +
@@ -88,11 +93,12 @@ class AllJobStatusResponseDto {
     (videoConversion.hashCode);
 
   @override
-  String toString() => 'AllJobStatusResponseDto[backgroundTask=$backgroundTask, faceDetection=$faceDetection, facialRecognition=$facialRecognition, library_=$library_, metadataExtraction=$metadataExtraction, migration=$migration, notifications=$notifications, search=$search, sidecar=$sidecar, smartSearch=$smartSearch, storageTemplateMigration=$storageTemplateMigration, thumbnailGeneration=$thumbnailGeneration, videoConversion=$videoConversion]';
+  String toString() => 'AllJobStatusResponseDto[backgroundTask=$backgroundTask, duplicateDetection=$duplicateDetection, faceDetection=$faceDetection, facialRecognition=$facialRecognition, library_=$library_, metadataExtraction=$metadataExtraction, migration=$migration, notifications=$notifications, search=$search, sidecar=$sidecar, smartSearch=$smartSearch, storageTemplateMigration=$storageTemplateMigration, thumbnailGeneration=$thumbnailGeneration, videoConversion=$videoConversion]';
 
   Map<String, dynamic> toJson() {
     final json = <String, dynamic>{};
       json[r'backgroundTask'] = this.backgroundTask;
+      json[r'duplicateDetection'] = this.duplicateDetection;
       json[r'faceDetection'] = this.faceDetection;
       json[r'facialRecognition'] = this.facialRecognition;
       json[r'library'] = this.library_;
@@ -117,6 +123,7 @@ class AllJobStatusResponseDto {
 
       return AllJobStatusResponseDto(
         backgroundTask: JobStatusDto.fromJson(json[r'backgroundTask'])!,
+        duplicateDetection: JobStatusDto.fromJson(json[r'duplicateDetection'])!,
         faceDetection: JobStatusDto.fromJson(json[r'faceDetection'])!,
         facialRecognition: JobStatusDto.fromJson(json[r'facialRecognition'])!,
         library_: JobStatusDto.fromJson(json[r'library'])!,
@@ -177,6 +184,7 @@ class AllJobStatusResponseDto {
   /// The list of required keys that must be present in a JSON.
   static const requiredKeys = <String>{
     'backgroundTask',
+    'duplicateDetection',
     'faceDetection',
     'facialRecognition',
     'library',

mobile/openapi/lib/model/duplicate_detection_config.dart

@@ -0,0 +1,108 @@
+//
+// AUTO-GENERATED FILE, DO NOT MODIFY!
+//
+// @dart=2.18
+
+// ignore_for_file: unused_element, unused_import
+// ignore_for_file: always_put_required_named_parameters_first
+// ignore_for_file: constant_identifier_names
+// ignore_for_file: lines_longer_than_80_chars
+
+part of openapi.api;
+
+class DuplicateDetectionConfig {
+  /// Returns a new [DuplicateDetectionConfig] instance.
+  DuplicateDetectionConfig({
+    required this.enabled,
+    required this.maxDistance,
+  });
+
+  bool enabled;
+
+  /// Minimum value: 0.001
+  /// Maximum value: 0.1
+  double maxDistance;
+
+  @override
+  bool operator ==(Object other) => identical(this, other) || other is DuplicateDetectionConfig &&
+    other.enabled == enabled &&
+    other.maxDistance == maxDistance;
+
+  @override
+  int get hashCode =>
+    // ignore: unnecessary_parenthesis
+    (enabled.hashCode) +
+    (maxDistance.hashCode);
+
+  @override
+  String toString() => 'DuplicateDetectionConfig[enabled=$enabled, maxDistance=$maxDistance]';
+
+  Map<String, dynamic> toJson() {
+    final json = <String, dynamic>{};
+      json[r'enabled'] = this.enabled;
+      json[r'maxDistance'] = this.maxDistance;
+    return json;
+  }
+
+  /// Returns a new [DuplicateDetectionConfig] instance and imports its values from
+  /// [value] if it's a [Map], null otherwise.
+  // ignore: prefer_constructors_over_static_methods
+  static DuplicateDetectionConfig? fromJson(dynamic value) {
+    if (value is Map) {
+      final json = value.cast<String, dynamic>();
+
+      return DuplicateDetectionConfig(
+        enabled: mapValueOfType<bool>(json, r'enabled')!,
+        maxDistance: mapValueOfType<double>(json, r'maxDistance')!,
+      );
+    }
+    return null;
+  }
+
+  static List<DuplicateDetectionConfig> listFromJson(dynamic json, {bool growable = false,}) {
+    final result = <DuplicateDetectionConfig>[];
+    if (json is List && json.isNotEmpty) {
+      for (final row in json) {
+        final value = DuplicateDetectionConfig.fromJson(row);
+        if (value != null) {
+          result.add(value);
+        }
+      }
+    }
+    return result.toList(growable: growable);
+  }
+
+  static Map<String, DuplicateDetectionConfig> mapFromJson(dynamic json) {
+    final map = <String, DuplicateDetectionConfig>{};
+    if (json is Map && json.isNotEmpty) {
+      json = json.cast<String, dynamic>(); // ignore: parameter_assignments
+      for (final entry in json.entries) {
+        final value = DuplicateDetectionConfig.fromJson(entry.value);
+        if (value != null) {
+          map[entry.key] = value;
+        }
+      }
+    }
+    return map;
+  }
+
+  // maps a json object with a list of DuplicateDetectionConfig-objects as value to a dart map
+  static Map<String, List<DuplicateDetectionConfig>> mapListFromJson(dynamic json, {bool growable = false,}) {
+    final map = <String, List<DuplicateDetectionConfig>>{};
+    if (json is Map && json.isNotEmpty) {
+      // ignore: parameter_assignments
+      json = json.cast<String, dynamic>();
+      for (final entry in json.entries) {
+        map[entry.key] = DuplicateDetectionConfig.listFromJson(entry.value, growable: growable,);
+      }
+    }
+    return map;
+  }
+
+  /// The list of required keys that must be present in a JSON.
+  static const requiredKeys = <String>{
+    'enabled',
+    'maxDistance',
+  };
+}
+

mobile/openapi/lib/model/job_name.dart

@@ -29,6 +29,7 @@ class JobName {
   static const faceDetection = JobName._(r'faceDetection');
   static const facialRecognition = JobName._(r'facialRecognition');
   static const smartSearch = JobName._(r'smartSearch');
+  static const duplicateDetection = JobName._(r'duplicateDetection');
   static const backgroundTask = JobName._(r'backgroundTask');
   static const storageTemplateMigration = JobName._(r'storageTemplateMigration');
   static const migration = JobName._(r'migration');
@@ -45,6 +46,7 @@ class JobName {
     faceDetection,
     facialRecognition,
     smartSearch,
+    duplicateDetection,
     backgroundTask,
     storageTemplateMigration,
     migration,
@@ -96,6 +98,7 @@ class JobNameTypeTransformer {
         case r'faceDetection': return JobName.faceDetection;
         case r'facialRecognition': return JobName.facialRecognition;
         case r'smartSearch': return JobName.smartSearch;
+        case r'duplicateDetection': return JobName.duplicateDetection;
         case r'backgroundTask': return JobName.backgroundTask;
         case r'storageTemplateMigration': return JobName.storageTemplateMigration;
         case r'migration': return JobName.migration;

mobile/openapi/lib/model/server_features_dto.dart

@@ -14,6 +14,7 @@ class ServerFeaturesDto {
   /// Returns a new [ServerFeaturesDto] instance.
   ServerFeaturesDto({
     required this.configFile,
+    required this.duplicateDetection,
     required this.email,
     required this.facialRecognition,
     required this.map,
@@ -29,6 +30,8 @@ class ServerFeaturesDto {
 
   bool configFile;
 
+  bool duplicateDetection;
+
   bool email;
 
   bool facialRecognition;
@@ -54,6 +57,7 @@ class ServerFeaturesDto {
   @override
   bool operator ==(Object other) => identical(this, other) || other is ServerFeaturesDto &&
     other.configFile == configFile &&
+    other.duplicateDetection == duplicateDetection &&
     other.email == email &&
     other.facialRecognition == facialRecognition &&
     other.map == map &&
@@ -70,6 +74,7 @@ class ServerFeaturesDto {
   int get hashCode =>
     // ignore: unnecessary_parenthesis
     (configFile.hashCode) +
+    (duplicateDetection.hashCode) +
     (email.hashCode) +
     (facialRecognition.hashCode) +
     (map.hashCode) +
@@ -83,11 +88,12 @@ class ServerFeaturesDto {
     (trash.hashCode);
 
   @override
-  String toString() => 'ServerFeaturesDto[configFile=$configFile, email=$email, facialRecognition=$facialRecognition, map=$map, oauth=$oauth, oauthAutoLaunch=$oauthAutoLaunch, passwordLogin=$passwordLogin, reverseGeocoding=$reverseGeocoding, search=$search, sidecar=$sidecar, smartSearch=$smartSearch, trash=$trash]';
+  String toString() => 'ServerFeaturesDto[configFile=$configFile, duplicateDetection=$duplicateDetection, email=$email, facialRecognition=$facialRecognition, map=$map, oauth=$oauth, oauthAutoLaunch=$oauthAutoLaunch, passwordLogin=$passwordLogin, reverseGeocoding=$reverseGeocoding, search=$search, sidecar=$sidecar, smartSearch=$smartSearch, trash=$trash]';
 
   Map<String, dynamic> toJson() {
     final json = <String, dynamic>{};
       json[r'configFile'] = this.configFile;
+      json[r'duplicateDetection'] = this.duplicateDetection;
       json[r'email'] = this.email;
       json[r'facialRecognition'] = this.facialRecognition;
       json[r'map'] = this.map;
@@ -111,6 +117,7 @@ class ServerFeaturesDto {
 
       return ServerFeaturesDto(
         configFile: mapValueOfType<bool>(json, r'configFile')!,
+        duplicateDetection: mapValueOfType<bool>(json, r'duplicateDetection')!,
         email: mapValueOfType<bool>(json, r'email')!,
         facialRecognition: mapValueOfType<bool>(json, r'facialRecognition')!,
         map: mapValueOfType<bool>(json, r'map')!,
@@ -170,6 +177,7 @@ class ServerFeaturesDto {
   /// The list of required keys that must be present in a JSON.
   static const requiredKeys = <String>{
     'configFile',
+    'duplicateDetection',
     'email',
     'facialRecognition',
     'map',

mobile/openapi/lib/model/system_config_machine_learning_dto.dart

@@ -14,6 +14,7 @@ class SystemConfigMachineLearningDto {
   /// Returns a new [SystemConfigMachineLearningDto] instance.
   SystemConfigMachineLearningDto({
     required this.clip,
+    required this.duplicateDetection,
     required this.enabled,
     required this.facialRecognition,
     required this.url,
@@ -21,6 +22,8 @@ class SystemConfigMachineLearningDto {
 
   CLIPConfig clip;
 
+  DuplicateDetectionConfig duplicateDetection;
+
   bool enabled;
 
   RecognitionConfig facialRecognition;
@@ -30,6 +33,7 @@ class SystemConfigMachineLearningDto {
   @override
   bool operator ==(Object other) => identical(this, other) || other is SystemConfigMachineLearningDto &&
     other.clip == clip &&
+    other.duplicateDetection == duplicateDetection &&
     other.enabled == enabled &&
     other.facialRecognition == facialRecognition &&
     other.url == url;
@@ -38,16 +42,18 @@ class SystemConfigMachineLearningDto {
   int get hashCode =>
     // ignore: unnecessary_parenthesis
     (clip.hashCode) +
+    (duplicateDetection.hashCode) +
     (enabled.hashCode) +
     (facialRecognition.hashCode) +
     (url.hashCode);
 
   @override
-  String toString() => 'SystemConfigMachineLearningDto[clip=$clip, enabled=$enabled, facialRecognition=$facialRecognition, url=$url]';
+  String toString() => 'SystemConfigMachineLearningDto[clip=$clip, duplicateDetection=$duplicateDetection, enabled=$enabled, facialRecognition=$facialRecognition, url=$url]';
 
   Map<String, dynamic> toJson() {
     final json = <String, dynamic>{};
       json[r'clip'] = this.clip;
+      json[r'duplicateDetection'] = this.duplicateDetection;
       json[r'enabled'] = this.enabled;
       json[r'facialRecognition'] = this.facialRecognition;
       json[r'url'] = this.url;
@@ -63,6 +69,7 @@ class SystemConfigMachineLearningDto {
 
       return SystemConfigMachineLearningDto(
         clip: CLIPConfig.fromJson(json[r'clip'])!,
+        duplicateDetection: DuplicateDetectionConfig.fromJson(json[r'duplicateDetection'])!,
         enabled: mapValueOfType<bool>(json, r'enabled')!,
         facialRecognition: RecognitionConfig.fromJson(json[r'facialRecognition'])!,
         url: mapValueOfType<String>(json, r'url')!,
@@ -114,6 +121,7 @@ class SystemConfigMachineLearningDto {
   /// The list of required keys that must be present in a JSON.
   static const requiredKeys = <String>{
     'clip',
+    'duplicateDetection',
     'enabled',
     'facialRecognition',
     'url',

mobile/openapi/test/all_job_status_response_dto_test.dart

@@ -21,6 +21,11 @@ void main() {
       // TODO
     });
 
+    // JobStatusDto duplicateDetection
+    test('to test the property `duplicateDetection`', () async {
+      // TODO
+    });
+
     // JobStatusDto faceDetection
     test('to test the property `faceDetection`', () async {
       // TODO

mobile/openapi/test/asset_api_test.dart

@@ -50,6 +50,11 @@ void main() {
       // TODO
     });
 
+    //Future<List<AssetResponseDto>> getAssetDuplicates() async
+    test('test getAssetDuplicates', () async {
+      // TODO
+    });
+
     //Future<AssetResponseDto> getAssetInfo(String id, { String key }) async
     test('test getAssetInfo', () async {
       // TODO

mobile/openapi/test/duplicate_detection_config_test.dart

@@ -0,0 +1,32 @@
+//
+// AUTO-GENERATED FILE, DO NOT MODIFY!
+//
+// @dart=2.18
+
+// ignore_for_file: unused_element, unused_import
+// ignore_for_file: always_put_required_named_parameters_first
+// ignore_for_file: constant_identifier_names
+// ignore_for_file: lines_longer_than_80_chars
+
+import 'package:openapi/api.dart';
+import 'package:test/test.dart';
+
+// tests for DuplicateDetectionConfig
+void main() {
+  // final instance = DuplicateDetectionConfig();
+
+  group('test DuplicateDetectionConfig', () {
+    // bool enabled
+    test('to test the property `enabled`', () async {
+      // TODO
+    });
+
+    // double maxDistance
+    test('to test the property `maxDistance`', () async {
+      // TODO
+    });
+
+
+  });
+
+}

mobile/openapi/test/server_features_dto_test.dart

@@ -21,6 +21,11 @@ void main() {
       // TODO
     });
 
+    // bool duplicateDetection
+    test('to test the property `duplicateDetection`', () async {
+      // TODO
+    });
+
     // bool email
     test('to test the property `email`', () async {
       // TODO

mobile/openapi/test/system_config_machine_learning_dto_test.dart

@@ -21,6 +21,11 @@ void main() {
       // TODO
     });
 
+    // DuplicateDetectionConfig duplicateDetection
+    test('to test the property `duplicateDetection`', () async {
+      // TODO
+    });
+
     // bool enabled
     test('to test the property `enabled`', () async {
       // TODO