[PR #1108] [MERGED] Updates to import/export #1056

New Issue

OVERLORD · 2026-02-04T21:13:43+03:00

OVERLORD commented

2026-02-04 21:13:43 +03:00

📋 Pull Request Information

Original PR: https://github.com/pocket-id/pocket-id/pull/1108
Author: @ItalyPaleAle
Created: 11/20/2025
Status: ✅ Merged
Merged: 11/25/2025
Merged by: @stonith404

Base: feat/data-export-import ← Head: import-export-updates

📝 Commits (9)

98d922f WIP
c81314e Lots of work to allow import from the new format
366682d Some more work & fixes
8239d55 Fixed and supports nullable types
ff6a0c4 Always migrate to target version
6b5f08c Apply suggestions from code review
8afe614 Revert change
c84cac2 💄
8c04e62 Add test to ensure migrations are defined for both databases

📊 Changes

10 files changed (+450 additions, -277 deletions)

View changed files

📝 backend/go.mod (+10 -10)
📝 backend/go.sum (+28 -24)
📝 backend/internal/cmds/import.go (+39 -20)
📝 backend/internal/model/types/date_time.go (+9 -0)
📝 backend/internal/service/export_service.go (+73 -69)
📝 backend/internal/service/import_service.go (+101 -154)
➕ backend/internal/utils/db_util.go (+116 -0)
➕ backend/resources/files_test.go (+72 -0)
➕ backend/resources/migrations/postgres/20251117141000_export_normalization.down.sql (+1 -0)
➕ backend/resources/migrations/postgres/20251117141000_export_normalization.up.sql (+1 -0)

📄 Description

PR against #998

Reads the column types from the database to correctly scan and insert data into the database. Does not use heuristics such as checking if data "looks like a UTF-8 string".
Handles NULL-able columns with "double pointers"
Does not require disabling Foreign Key constraints. This is done by hard coding a list of 3 tables that need to be migrated first, in order, as they are the target of FK relationships. The problem with disabling FKs is that on Postgres it requires super-user permissions.
Fixes an error indicating that locks cannot be released after an import, since the kv table was dropped and all locks were destroyed
Fixes importing into an empty database when there's no KV table

Tested that this also allows switching between Postgres and SQLite

Note: I have not updated the E2E tests

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/pocket-id/pocket-id/pull/1108 **Author:** [@ItalyPaleAle](https://github.com/ItalyPaleAle) **Created:** 11/20/2025 **Status:** ✅ Merged **Merged:** 11/25/2025 **Merged by:** [@stonith404](https://github.com/stonith404) **Base:** `feat/data-export-import` ← **Head:** `import-export-updates` --- ### 📝 Commits (9) - [`98d922f`](https://github.com/pocket-id/pocket-id/commit/98d922f23e6fe1bfb83246f991cc548f17765faa) WIP - [`c81314e`](https://github.com/pocket-id/pocket-id/commit/c81314ecc02b0a2c2e3540102bb897246f9336bc) Lots of work to allow import from the new format - [`366682d`](https://github.com/pocket-id/pocket-id/commit/366682d28f4203e95c7a4a6a43ff2ac7869d0673) Some more work & fixes - [`8239d55`](https://github.com/pocket-id/pocket-id/commit/8239d552acee0a0f28c0843351b487689f561d67) Fixed and supports nullable types - [`ff6a0c4`](https://github.com/pocket-id/pocket-id/commit/ff6a0c49afd9149af1a011edf00a0c2d22213ad2) Always migrate to target version - [`6b5f08c`](https://github.com/pocket-id/pocket-id/commit/6b5f08ca4a7f3c821a04486c05033cf90cf355ad) Apply suggestions from code review - [`8afe614`](https://github.com/pocket-id/pocket-id/commit/8afe61410406f466a1c37bd1ba7a01057c0b9183) Revert change - [`c84cac2`](https://github.com/pocket-id/pocket-id/commit/c84cac2560753f34f79eecc186487d8dba71a1a1) 💄 - [`8c04e62`](https://github.com/pocket-id/pocket-id/commit/8c04e629e416de7933af2876b51f1ce14ed10279) Add test to ensure migrations are defined for both databases ### 📊 Changes **10 files changed** (+450 additions, -277 deletions) <details> <summary>View changed files</summary> 📝 `backend/go.mod` (+10 -10) 📝 `backend/go.sum` (+28 -24) 📝 `backend/internal/cmds/import.go` (+39 -20) 📝 `backend/internal/model/types/date_time.go` (+9 -0) 📝 `backend/internal/service/export_service.go` (+73 -69) 📝 `backend/internal/service/import_service.go` (+101 -154) ➕ `backend/internal/utils/db_util.go` (+116 -0) ➕ `backend/resources/files_test.go` (+72 -0) ➕ `backend/resources/migrations/postgres/20251117141000_export_normalization.down.sql` (+1 -0) ➕ `backend/resources/migrations/postgres/20251117141000_export_normalization.up.sql` (+1 -0) </details> ### 📄 Description PR against #998 1. Reads the column types from the database to correctly scan and insert data into the database. Does not use heuristics such as checking if data "looks like a UTF-8 string". 2. Handles NULL-able columns with "double pointers" 3. Does not require disabling Foreign Key constraints. This is done by hard coding a list of 3 tables that need to be migrated first, in order, as they are the target of FK relationships. The problem with disabling FKs is that on Postgres it requires super-user permissions. 3. Fixes an error indicating that locks cannot be released after an import, since the kv table was dropped and all locks were destroyed 4. Fixes importing into an empty database when there's no KV table Tested that this also allows switching between Postgres and SQLite Note: I have not updated the E2E tests --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

OVERLORD added the pull-request label 2026-02-04 21:13:43 +03:00

OVERLORD closed this issue

2026-02-04 21:13:47 +03:00

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/pocket-id#1056