immich-app/immich
1
0
Fork 0
mirror of https://github.com/immich-app/immich.git synced 2026-07-16 05:34:30 +03:00
Code Issues 783 Packages Projects Releases 22 Wiki Activity

Immich v2.4.1 → v2.5.0 Manual Upgrade Failures #8343

New Issue
Closed
opened 2026-02-05 13:40:39 +03:00 by OVERLORD · 0 comments
OVERLORD commented 2026-02-05 13:40:39 +03:00
Owner
Copy Link

Originally created by @microsoftShannon on GitHub (Jan 29, 2026).

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

  • Yes

The bug

Environment

Proxmox Setup:

  • Proxmox VE 8.x
  • LXC Container (Unprivileged)
  • Container ID:
  • OS: Debian 13 (Trixie)
  • CPU: 4 cores
  • RAM: 4GB (later increased to 8GB)
  • Storage: 20GB on local SSD

Immich Installation:

  • Installed via: Community Scripts ct/immich.sh
  • Original Version: v2.4.1
  • Installation Type: Native (not Docker)
  • Services: systemd (immich-web, immich-ml)
  • Database: PostgreSQL 16
  • Storage: NFS mount (bind-mounted from Proxmox host)

Network:

Issue Description

Attempted manual upgrade from v2.4.1 to v2.5.0 following standard procedure, but encountered multiple cascading failures related to module resolution, Python dependencies, and file structure issues.

Attempted Upgrade Process

Step 1: Pre-Upgrade Backup ✅

# Created PostgreSQL backup
su - postgres -c "pg_dump immich -F c -f /var/lib/postgresql/backups/immich_pre_upgrade_$(date +%Y%m%d).backup"
# Result: 238MB backup created successfully

Step 2: Download & Extract v2.5.0 Source

cd /opt/immich
systemctl stop immich-web immich-ml
rm -rf source
wget https://github.com/immich-app/immich/archive/refs/tags/v2.5.0.tar.gz
tar -xzf v2.5.0.tar.gz
mv immich-2.5.0 source
rm v2.5.0.tar.gz

Step 3: Rebuild Server ❌

cd /opt/immich/source/server
pnpm install  # Completed successfully
pnpm build    # Completed successfully

# Copied built files to app directory
rm -rf /opt/immich/app/*
cp -r dist/* /opt/immich/app/
cp package.json /opt/immich/app/
cp -r node_modules /opt/immich/app/

Problem 1: Wrong directory structure - Files were flattened instead of preserving dist/ folder hierarchy.

Errors Encountered

Error 1: Module Not Found (kysely)

Symptom:

Error: Cannot find module 'kysely'
Require stack:
- /opt/immich/app/dist/main.js

Root Cause:

  • Copied dist/* instead of dist/ directory
  • Node modules symlinks pointing to wrong paths
  • pnpm workspace structure not preserved

Attempted Fixes:

  1. ✅ Moved files into proper dist/ structure
  2. ✅ Reinstalled node_modules: pnpm install --prod
  3. ❌ Symlinks still broken: /opt/immich/app/node_modules/kysely -> .pnpm/kysely@0.28.2/node_modules/kysely
  4. ❌ Target path didn't exist properly

Error 2: Python Dependencies Missing

Symptom:

# ML service failing
ModuleNotFoundError: No module named 'gunicorn'

Root Cause:

  • Machine learning Python dependencies not installed
  • No pip available (Python 3.13 minimal install)

Attempted Fixes:

  1. ❌ pip install --break-system-packages -e . - pip not found
  2. ❌ pip3 install --break-system-packages -e . - pip3 not found
  3. ✅ uv pip install --system -e . - Worked, but...

Error 3: Python Executable Not Found

Symptom:

FileNotFoundError: [Errno 2] No such file or directory: 'python'

Root Cause:

  • System has python3 but not python
  • ML startup script expects python command

Fix Applied:

ln -s /usr/bin/python3 /usr/bin/python

Error 4: Module Resolution Persists

Even after multiple attempts to fix node_modules:

  • Reinstalling in /opt/immich/app
  • Copying from /opt/immich/source/server
  • Fixing ownership (chown -R immich:immich)

Consistent Error:

Error: Cannot find module 'kysely'

The pnpm symlink structure worked when running as root but failed when running as the immich user via systemd.

Database Restoration Issues

When attempting to restore from backup after failed upgrade:

Error:

ERROR: insert or update on table "album" violates foreign key constraint
ERROR: duplicate key value violates unique constraint

Cause: Fresh v2.4.1 installation had pre-created schema that conflicted with backup.

Solution:

su - postgres -c "dropdb immich"
su - postgres -c "createdb immich -O immich"
gunzip -c backup.sql.gz | su - postgres -c "psql immich"

Resolution

Abandoned manual upgrade and performed clean rebuild:

  1. Destroyed broken container: pct destroy xxx
  2. Rebuilt using community script: Fresh v2.4.1 installation
  3. Restored database backup: All 54,542 photos recovered
  4. Reconnected NFS storage: Bind-mounted Waihona
  5. Fixed database password mismatch:
   su - postgres -c "psql -c \"ALTER USER immich WITH PASSWORD 'PASSWORD_FROM_ENV';\""

Total time: ~3 hours of troubleshooting, 30 minutes for clean rebuild.

Recommendations

For Community Script Maintainers

  1. Add proper update mechanism to ct/immich.sh:
   function update_script() {
     # Check version
     # Stop services
     # Backup database automatically
     # Download new version
     # Rebuild with proper structure
     # Migrate if needed
     # Restart services
   }

  1. Document upgrade path clearly:

    • Which versions support in-place upgrades
    • When clean reinstall is recommended
    • How to preserve data during upgrades

  2. Add version-specific migration scripts for major version changes

  3. Ensure Python dependencies are installed with correct package manager (uv)

  4. Create python symlink during initial installation

  5. Test pnpm workspace structure in unprivileged containers

For Users

DO NOT attempt manual upgrades. Wait for official community script support:

# When officially supported, use:
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)"
# And select "Update" option (when available)

Before any upgrade attempt:

  1. ✅ Create Proxmox container backup
  2. ✅ Verify database backups exist (daily at 2 AM)
  3. ✅ Document current configuration
  4. ✅ Test backup restoration

Technical Details

File Structure Issues

Expected:

/opt/immich/app/
├── dist/
│   ├── main.js
│   ├── *.js
│   └── [subdirectories]
├── node_modules/
├── package.json
└── machine-learning/
    └── ml_start.sh

What happened during manual upgrade:

/opt/immich/app/
├── main.js        # ❌ Flattened!
├── *.js           # ❌ All in root
├── node_modules/
│   └── kysely -> ../../node_modules/.pnpm/...  # ❌ Broken symlink
└── package.json

Working Configuration (v2.4.1)

Database Backup Schedule:

"backup": {
  "database": {
    "cronExpression": "0 02 * * *",
    "enabled": true,
    "keepLastAmount": 14
  }
}

Storage:

  • Original photos: /mnt/xxx/library/ (never modified)
  • Thumbnails: /mnt/xxx/thumbs/
  • Backups: /mnt/xxx/backups/

Related Issues

  • pnpm workspace symlinks in unprivileged LXC containers
  • Python package management in Debian 13 (minimal install)
  • Module resolution when running as non-root user

System Information

# Container
root@immich:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
VERSION="13 (trixie)"

# Node.js
root@immich:~# node --version
v24.13.0

# pnpm
root@immich:~# pnpm --version
10.28.0

# Python
root@immich:~# python3 --version
Python 3.13.5

# PostgreSQL
root@immich:~# su - postgres -c "psql --version"
psql (PostgreSQL) 16.11

Conclusion

Manual upgrades of Immich in the community-scripts environment are not recommended due to:

  • Complex pnpm workspace structure
  • Python dependency management challenges
  • Service file structure expectations
  • Database migration requirements

Recommendation: Wait for official community script update support for v2.5.0 before upgrading.

Labels: bug, enhancement, documentation, immich, upgrade
Milestone: Future release with update support

The OS that Immich Server is running on

Debian 13 (Trixie)

Version of Immich Server

2.5.0

Version of Immich Mobile App

2.4.1 build .239

Platform with the issue

  • Server
  • Web
  • Mobile

Device make and model

Iphone 16 Pro Max

Your docker-compose.yml content

N/A - Not using Docker

Installation Method: Native systemd deployment via Proxmox Community Scripts
Repository: https://github.com/community-scripts/ProxmoxVE
Script: ct/immich.sh
Deployment: LXC container with systemd services

See systemd service configuration above for equivalent deployment details.

Your .env content

bash# Database Configuration
DB_HOSTNAME=127.0.0.1
DB_USERNAME=immich
DB_PASSWORD=********  # REDACTED
DB_DATABASE_NAME=immich
DB_VECTOR_EXTENSION=vectorchord

# Storage Configuration
IMMICH_MEDIA_LOCATION=/mnt/waihona

Reproduction steps

Reproduction Steps for Immich v2.4.1 → v2.5.0 Upgrade Failure
markdown## Prerequisites

Environment

  • Proxmox VE 8.x
  • LXC Container (Unprivileged)
  • OS: Debian GNU/Linux 13 (Trixie)
  • Architecture: x86_64

Initial Installation

# Install Immich v2.4.1 using Proxmox Community Scripts
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)"

Container Configuration:

  • Type: Unprivileged
  • CPU: 4 cores
  • RAM: 4096 MB
  • Disk: 20 GB
  • Network: DHCP (example: 192.168.x.x)

Verify Working Installation

# Access web interface
http://:2283

# Verify services running
systemctl status immich-web immich-ml
# Both should show: active (running)

# Verify version
cat /opt/immich/app/package.json | grep version
# Should show: "version": "2.4.1"

Reproduction Steps

Step 1: Create Database Backup

# Enter container (from Proxmox host)
pct enter 

# Create backup directory
mkdir -p /var/lib/postgresql/backups
chown postgres:postgres /var/lib/postgresql/backups

# Create PostgreSQL backup
su - postgres -c "pg_dump immich -F c -f /var/lib/postgresql/backups/immich_pre_upgrade_$(date +%Y%m%d).backup"

# Verify backup created
ls -lh /var/lib/postgresql/backups/
# Expected: Backup file ~200-300MB

Step 2: Stop Immich Services

systemctl stop immich-web
systemctl stop immich-ml

# Verify stopped
systemctl status immich-web immich-ml
# Both should show: inactive (dead)

Step 3: Download v2.5.0 Source

cd /opt/immich

# Remove old source
rm -rf source

# Download v2.5.0
wget https://github.com/immich-app/immich/archive/refs/tags/v2.5.0.tar.gz
tar -xzf v2.5.0.tar.gz
mv immich-2.5.0 source
rm v2.5.0.tar.gz

# Verify download
ls -la /opt/immich/source/

Step 4: Build Server Application

cd /opt/immich/source/server

# Install dependencies
pnpm install

Expected Output:

Packages: +2553
Progress: resolved 2706, reused 2135, downloaded 410, added 2553, done
Done in 38.9s using pnpm v10.28.0

# Build application
pnpm build

Expected Output:

> immich@2.5.0 build /opt/immich/source/server
> nest build

Step 5: Copy Built Files to App Directory

⚠️ This step demonstrates the incorrect method that causes the bug:

cd /opt/immich

# Clear app directory
rm -rf app/*

# Copy files (WRONG - flattens directory structure)
cp -r source/server/dist/* app/
cp source/server/package.json app/
cp -r source/server/node_modules app/

Result: Files are incorrectly flattened:

/opt/immich/app/
├── main.js              # ❌ Should be in dist/
├── app.module.js        # ❌ Should be in dist/
├── controllers/         # ❌ Should be in dist/
├── node_modules/        # ✓ Correct location
└── package.json         # ✓ Correct location

Expected structure:

/opt/immich/app/
├── dist/                # ✓ Subdirectory should exist
│   ├── main.js
│   ├── app.module.js
│   └── controllers/
├── node_modules/
└── package.json

Step 6: Attempt to Start Web Service

systemctl start immich-web

# Check status
systemctl status immich-web

Expected Error:

× immich-web.service - immich Web Service
     Active: failed (Result: exit-code)

Check logs:

tail -30 /var/log/immich/web.log

Expected Error in Logs:

Error: Cannot find module 'kysely'
Require stack:
- /opt/immich/app/dist/main.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1421:15)
    ...
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/opt/immich/app/dist/main.js' ]

Attempted Fixes (All Failed)

Fix Attempt #1: Correct Directory Structure

cd /opt/immich/app

# Create dist directory
mkdir -p dist

# Move files into dist/
mv *.js *.map *.d.ts *.tsbuildinfo dist/
mv bin commands controllers cores dtos emails maintenance middleware \
   repositories schema services sql-tools types utils workers dist/

# Verify structure
ls -la /opt/immich/app/dist/main.js
# Should exist now

Restart service:

systemctl restart immich-web
tail -20 /var/log/immich/web.log

Result: Same MODULE_NOT_FOUND error persists

Fix Attempt #2: Reinstall node_modules

cd /opt/immich/app

# Remove existing node_modules
rm -rf node_modules

# Reinstall dependencies
pnpm install --prod

Expected Output:

Packages: +636
Done in 8.1s using pnpm v10.28.0

Check symlink structure:

ls -la /opt/immich/app/node_modules/kysely

Output:

lrwxrwxrwx 1 root root 39 -> .pnpm/kysely@0.28.2/node_modules/kysely

Verify target exists:

ls -la /opt/immich/app/node_modules/.pnpm/kysely@0.28.2/node_modules/kysely

Output:

drwxr-xr-x 4 root root 4096 dist/
-rw-r--r-- 3 1082 package.json
...

Restart service:

systemctl restart immich-web

Result: Still fails with MODULE_NOT_FOUND

Fix Attempt #3: Test Module Resolution

# Test as root user
cd /opt/immich/app
node -e "console.log(require('kysely'))"

Output:

{
  Kysely: [Getter],
  sql: [Getter],
  ...
}

✅ Works as root!

But service running as immich user still fails:

systemctl start immich-web
tail /var/log/immich/web.log

Result: MODULE_NOT_FOUND when running as immich user

Fix Attempt #4: Fix Ownership

# Set proper ownership
chown -R immich:immich /opt/immich/app

# Restart service
systemctl restart immich-web

Result: Still fails with same error

Machine Learning Service Issues

Issue #1: Python Dependencies Missing

systemctl start immich-ml
systemctl status immich-ml

Expected Error:

× immich-ml.service - immich Machine-Learning
     Active: failed (Result: exit-code)

Check logs:

tail /var/log/immich/ml.log

Expected Error:

ModuleNotFoundError: No module named 'gunicorn'

Attempt to install:

cd /opt/immich/source/machine-learning

pip install --break-system-packages -e .

Result:

bash: pip: command not found

pip3 install --break-system-packages -e .

Result:

bash: pip3: command not found

python3 -m pip install --break-system-packages -e .

Result:

/bin/python3: No module named pip

Issue #2: Install with uv (Partial Success)

# Check if uv is available
which uv

Output:

/usr/local/bin/uv

Install Python dependencies:

cd /opt/immich/source/machine-learning
uv pip install --system -e .

Expected Output:

Successfully installed [packages]

Try to start ML service:

systemctl start immich-ml
tail /var/log/immich/ml.log

Expected Error:

FileNotFoundError: [Errno 2] No such file or directory: 'python'

Issue #3: Python Executable Not Found

Fix:

# Create python symlink
ln -s /usr/bin/python3 /usr/bin/python

# Verify
which python
python --version

Output:

/usr/bin/python
Python 3.13.5

Start ML service:

systemctl start immich-ml
systemctl status immich-ml

Result: ✅ ML service now starts successfully

● immich-ml.service - immich Machine-Learning
     Active: active (running)

Final State

After all attempted fixes:

  • ✅ ML Service: Running (after Python fixes)
  • ❌ Web Service: Failed (MODULE_NOT_FOUND persists)
  • ❌ Database: Not migrated
  • ❌ Photos: Inaccessible via web UI

System Information

# Container OS
cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
VERSION="13 (trixie)"
VERSION_ID="13"

# Node.js version
node --version

v24.13.0

# pnpm version
pnpm --version

10.28.0

# Python version
python3 --version

Python 3.13.5

# PostgreSQL version
su - postgres -c "psql --version"

psql (PostgreSQL) 16.11

Working Restoration Method

After failed upgrade attempts, complete restoration was achieved:

Step 1: Destroy Broken Container

# From Proxmox host
pct stop 
pct destroy

Step 2: Reinstall Fresh v2.4.1

# Use community script
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)"

# Use same container ID and configuration as before

Step 3: Reconnect NFS Storage

# On Proxmox host
mount -t nfs :/volume1/storage /mnt/storage

# Bind mount to container
pct stop 
pct set  -mp0 /mnt/storage,mp=/mnt/storage
pct start

Step 4: Restore Database

# Enter container
pct enter 

# Drop and recreate database
su - postgres -c "dropdb immich"
su - postgres -c "createdb immich -O immich"

# Restore from backup
gunzip -c /mnt/storage/backups/immich_pre_upgrade_20260127.backup.sql.gz | \
  su - postgres -c "psql immich"

# Verify restoration
su - postgres -c "psql -d immich -c 'SELECT COUNT(*) FROM asset;'"
# Should show your photo count

Step 5: Fix Database Password

# Get password from .env
grep DB_PASSWORD /opt/immich/.env

# Set password in PostgreSQL
su - postgres -c "psql -c \"ALTER USER immich WITH PASSWORD '';\""

Step 6: Configure Storage Location

# Add storage location to .env
echo "IMMICH_MEDIA_LOCATION=/mnt/storage" >> /opt/immich/.env

Step 7: Start Services

systemctl start immich-ml
systemctl start immich-web

# Verify both running
systemctl status immich-web immich-ml

Result: ✅ Full restoration on v2.4.1 with all photos accessible

Root Causes Identified

  1. pnpm Workspace Symlinks

    • Symlinks work for root user but fail for non-root systemd services
    • Unprivileged LXC container permission issues
    • Module resolution path incorrect when files copied vs installed

  2. Python Package Management

    • Debian 13 minimal install lacks pip/pip3
    • Only uv package manager available
    • Python symlink not created by default

  3. No Migration Path

    • Community script designed for fresh installs only
    • No documented upgrade procedure
    • No automated database migration

  4. File Structure Assumptions

    • Systemd service expects specific directory layout
    • Simple file copy breaks pnpm workspace structure
    • Build artifacts require precise placement

Expected Behavior

The community script should provide:

  1. Update mechanism:
   bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)"
   # Should detect existing installation and offer update option

  1. Automatic database backup before upgrade

  2. Proper file structure preservation during upgrade

  3. Database migration handled automatically

  4. Service restart with health checks

  5. Rollback capability if upgrade fails

Workaround for Now

Do NOT attempt manual upgrade. Wait for official community script support for v2.5.0.

To stay safe:

Additional Notes

  • Total troubleshooting time: ~3 hours
  • Clean rebuild and restoration time: ~30 minutes
  • No photo data lost (stored on NFS, independent of container)
  • Database successfully restored from automatic backup

Relevant log output

Additional information

No response

Originally created by @microsoftShannon on GitHub (Jan 29, 2026). ### I have searched the existing issues, both open and closed, to make sure this is not a duplicate report. - [x] Yes ### The bug ## Environment **Proxmox Setup:** - Proxmox VE 8.x - LXC Container (Unprivileged) - Container ID: - OS: Debian 13 (Trixie) - CPU: 4 cores - RAM: 4GB (later increased to 8GB) - Storage: 20GB on local SSD **Immich Installation:** - Installed via: [Community Scripts](https://github.com/community-scripts/ProxmoxVE) `ct/immich.sh` - Original Version: v2.4.1 - Installation Type: Native (not Docker) - Services: systemd (immich-web, immich-ml) - Database: PostgreSQL 16 - Storage: NFS mount (bind-mounted from Proxmox host) **Network:** - IP: x.x.x.x (DHCP reservation) - Access: http://x.x.x.x:2283 ## Issue Description Attempted manual upgrade from v2.4.1 to v2.5.0 following standard procedure, but encountered multiple cascading failures related to module resolution, Python dependencies, and file structure issues. ## Attempted Upgrade Process ### Step 1: Pre-Upgrade Backup ✅ ```bash # Created PostgreSQL backup su - postgres -c "pg_dump immich -F c -f /var/lib/postgresql/backups/immich_pre_upgrade_$(date +%Y%m%d).backup" # Result: 238MB backup created successfully ``` ### Step 2: Download & Extract v2.5.0 Source ```bash cd /opt/immich systemctl stop immich-web immich-ml rm -rf source wget https://github.com/immich-app/immich/archive/refs/tags/v2.5.0.tar.gz tar -xzf v2.5.0.tar.gz mv immich-2.5.0 source rm v2.5.0.tar.gz ``` ### Step 3: Rebuild Server ❌ ```bash cd /opt/immich/source/server pnpm install # Completed successfully pnpm build # Completed successfully # Copied built files to app directory rm -rf /opt/immich/app/* cp -r dist/* /opt/immich/app/ cp package.json /opt/immich/app/ cp -r node_modules /opt/immich/app/ ``` **Problem 1: Wrong directory structure** - Files were flattened instead of preserving `dist/` folder hierarchy. ## Errors Encountered ### Error 1: Module Not Found (kysely) **Symptom:** ``` Error: Cannot find module 'kysely' Require stack: - /opt/immich/app/dist/main.js ``` **Root Cause:** - Copied `dist/*` instead of `dist/` directory - Node modules symlinks pointing to wrong paths - pnpm workspace structure not preserved **Attempted Fixes:** 1. ✅ Moved files into proper `dist/` structure 2. ✅ Reinstalled node_modules: `pnpm install --prod` 3. ❌ Symlinks still broken: `/opt/immich/app/node_modules/kysely -> .pnpm/kysely@0.28.2/node_modules/kysely` 4. ❌ Target path didn't exist properly ### Error 2: Python Dependencies Missing **Symptom:** ```bash # ML service failing ModuleNotFoundError: No module named 'gunicorn' ``` **Root Cause:** - Machine learning Python dependencies not installed - No `pip` available (Python 3.13 minimal install) **Attempted Fixes:** 1. ❌ `pip install --break-system-packages -e .` - pip not found 2. ❌ `pip3 install --break-system-packages -e .` - pip3 not found 3. ✅ `uv pip install --system -e .` - Worked, but... ### Error 3: Python Executable Not Found **Symptom:** ``` FileNotFoundError: [Errno 2] No such file or directory: 'python' ``` **Root Cause:** - System has `python3` but not `python` - ML startup script expects `python` command **Fix Applied:** ```bash ln -s /usr/bin/python3 /usr/bin/python ``` ### Error 4: Module Resolution Persists Even after multiple attempts to fix node_modules: - Reinstalling in `/opt/immich/app` - Copying from `/opt/immich/source/server` - Fixing ownership (`chown -R immich:immich`) **Consistent Error:** ``` Error: Cannot find module 'kysely' ``` The pnpm symlink structure worked when running as root but failed when running as the `immich` user via systemd. ## Database Restoration Issues When attempting to restore from backup after failed upgrade: **Error:** ``` ERROR: insert or update on table "album" violates foreign key constraint ERROR: duplicate key value violates unique constraint ``` **Cause:** Fresh v2.4.1 installation had pre-created schema that conflicted with backup. **Solution:** ```bash su - postgres -c "dropdb immich" su - postgres -c "createdb immich -O immich" gunzip -c backup.sql.gz | su - postgres -c "psql immich" ``` ## Resolution **Abandoned manual upgrade** and performed clean rebuild: 1. **Destroyed broken container:** `pct destroy xxx` 2. **Rebuilt using community script:** Fresh v2.4.1 installation 3. **Restored database backup:** All 54,542 photos recovered 4. **Reconnected NFS storage:** Bind-mounted Waihona 5. **Fixed database password mismatch:** ```bash su - postgres -c "psql -c \"ALTER USER immich WITH PASSWORD 'PASSWORD_FROM_ENV';\"" ``` **Total time:** ~3 hours of troubleshooting, 30 minutes for clean rebuild. ## Recommendations ### For Community Script Maintainers 1. **Add proper update mechanism to `ct/immich.sh`:** ```bash function update_script() { # Check version # Stop services # Backup database automatically # Download new version # Rebuild with proper structure # Migrate if needed # Restart services } ``` 2. **Document upgrade path clearly:** - Which versions support in-place upgrades - When clean reinstall is recommended - How to preserve data during upgrades 3. **Add version-specific migration scripts** for major version changes 4. **Ensure Python dependencies are installed** with correct package manager (uv) 5. **Create `python` symlink** during initial installation 6. **Test pnpm workspace structure** in unprivileged containers ### For Users **DO NOT attempt manual upgrades.** Wait for official community script support: ```bash # When officially supported, use: bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)" # And select "Update" option (when available) ``` **Before any upgrade attempt:** 1. ✅ Create Proxmox container backup 2. ✅ Verify database backups exist (daily at 2 AM) 3. ✅ Document current configuration 4. ✅ Test backup restoration ## Technical Details ### File Structure Issues **Expected:** ``` /opt/immich/app/ ├── dist/ │ ├── main.js │ ├── *.js │ └── [subdirectories] ├── node_modules/ ├── package.json └── machine-learning/ └── ml_start.sh ``` **What happened during manual upgrade:** ``` /opt/immich/app/ ├── main.js # ❌ Flattened! ├── *.js # ❌ All in root ├── node_modules/ │ └── kysely -> ../../node_modules/.pnpm/... # ❌ Broken symlink └── package.json ``` ### Working Configuration (v2.4.1) **Database Backup Schedule:** ```json "backup": { "database": { "cronExpression": "0 02 * * *", "enabled": true, "keepLastAmount": 14 } } ``` **Storage:** - Original photos: `/mnt/xxx/library/` (never modified) - Thumbnails: `/mnt/xxx/thumbs/` - Backups: `/mnt/xxx/backups/` ## Related Issues - pnpm workspace symlinks in unprivileged LXC containers - Python package management in Debian 13 (minimal install) - Module resolution when running as non-root user ## System Information ```bash # Container root@immich:~# cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 13 (trixie)" VERSION="13 (trixie)" # Node.js root@immich:~# node --version v24.13.0 # pnpm root@immich:~# pnpm --version 10.28.0 # Python root@immich:~# python3 --version Python 3.13.5 # PostgreSQL root@immich:~# su - postgres -c "psql --version" psql (PostgreSQL) 16.11 ``` ## Conclusion Manual upgrades of Immich in the community-scripts environment are **not recommended** due to: - Complex pnpm workspace structure - Python dependency management challenges - Service file structure expectations - Database migration requirements **Recommendation:** Wait for official community script update support for v2.5.0 before upgrading. --- **Labels:** `bug`, `enhancement`, `documentation`, `immich`, `upgrade` **Milestone:** Future release with update support ### The OS that Immich Server is running on Debian 13 (Trixie) ### Version of Immich Server 2.5.0 ### Version of Immich Mobile App 2.4.1 build .239 ### Platform with the issue - [x] Server - [ ] Web - [ ] Mobile ### Device make and model Iphone 16 Pro Max ### Your docker-compose.yml content ```YAML N/A - Not using Docker Installation Method: Native systemd deployment via Proxmox Community Scripts Repository: https://github.com/community-scripts/ProxmoxVE Script: ct/immich.sh Deployment: LXC container with systemd services See systemd service configuration above for equivalent deployment details. ``` ### Your .env content ```Shell bash# Database Configuration DB_HOSTNAME=127.0.0.1 DB_USERNAME=immich DB_PASSWORD=******** # REDACTED DB_DATABASE_NAME=immich DB_VECTOR_EXTENSION=vectorchord # Storage Configuration IMMICH_MEDIA_LOCATION=/mnt/waihona ``` ### Reproduction steps Reproduction Steps for Immich v2.4.1 → v2.5.0 Upgrade Failure markdown## Prerequisites ### Environment - Proxmox VE 8.x - LXC Container (Unprivileged) - OS: Debian GNU/Linux 13 (Trixie) - Architecture: x86_64 ### Initial Installation ```bash # Install Immich v2.4.1 using Proxmox Community Scripts bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)" ``` **Container Configuration:** - Type: Unprivileged - CPU: 4 cores - RAM: 4096 MB - Disk: 20 GB - Network: DHCP (example: 192.168.x.x) ### Verify Working Installation ```bash # Access web interface http://:2283 # Verify services running systemctl status immich-web immich-ml # Both should show: active (running) # Verify version cat /opt/immich/app/package.json | grep version # Should show: "version": "2.4.1" ``` --- ## Reproduction Steps ### Step 1: Create Database Backup ```bash # Enter container (from Proxmox host) pct enter # Create backup directory mkdir -p /var/lib/postgresql/backups chown postgres:postgres /var/lib/postgresql/backups # Create PostgreSQL backup su - postgres -c "pg_dump immich -F c -f /var/lib/postgresql/backups/immich_pre_upgrade_$(date +%Y%m%d).backup" # Verify backup created ls -lh /var/lib/postgresql/backups/ # Expected: Backup file ~200-300MB ``` ### Step 2: Stop Immich Services ```bash systemctl stop immich-web systemctl stop immich-ml # Verify stopped systemctl status immich-web immich-ml # Both should show: inactive (dead) ``` ### Step 3: Download v2.5.0 Source ```bash cd /opt/immich # Remove old source rm -rf source # Download v2.5.0 wget https://github.com/immich-app/immich/archive/refs/tags/v2.5.0.tar.gz tar -xzf v2.5.0.tar.gz mv immich-2.5.0 source rm v2.5.0.tar.gz # Verify download ls -la /opt/immich/source/ ``` ### Step 4: Build Server Application ```bash cd /opt/immich/source/server # Install dependencies pnpm install ``` **Expected Output:** ``` Packages: +2553 Progress: resolved 2706, reused 2135, downloaded 410, added 2553, done Done in 38.9s using pnpm v10.28.0 ``` ```bash # Build application pnpm build ``` **Expected Output:** ``` > immich@2.5.0 build /opt/immich/source/server > nest build ``` ### Step 5: Copy Built Files to App Directory **⚠️ This step demonstrates the incorrect method that causes the bug:** ```bash cd /opt/immich # Clear app directory rm -rf app/* # Copy files (WRONG - flattens directory structure) cp -r source/server/dist/* app/ cp source/server/package.json app/ cp -r source/server/node_modules app/ ``` **Result:** Files are incorrectly flattened: ``` /opt/immich/app/ ├── main.js # ❌ Should be in dist/ ├── app.module.js # ❌ Should be in dist/ ├── controllers/ # ❌ Should be in dist/ ├── node_modules/ # ✓ Correct location └── package.json # ✓ Correct location ``` **Expected structure:** ``` /opt/immich/app/ ├── dist/ # ✓ Subdirectory should exist │ ├── main.js │ ├── app.module.js │ └── controllers/ ├── node_modules/ └── package.json ``` ### Step 6: Attempt to Start Web Service ```bash systemctl start immich-web # Check status systemctl status immich-web ``` **Expected Error:** ``` × immich-web.service - immich Web Service Active: failed (Result: exit-code) ``` **Check logs:** ```bash tail -30 /var/log/immich/web.log ``` **Expected Error in Logs:** ``` Error: Cannot find module 'kysely' Require stack: - /opt/immich/app/dist/main.js at Module._resolveFilename (node:internal/modules/cjs/loader:1421:15) ... code: 'MODULE_NOT_FOUND', requireStack: [ '/opt/immich/app/dist/main.js' ] ``` --- ## Attempted Fixes (All Failed) ### Fix Attempt #1: Correct Directory Structure ```bash cd /opt/immich/app # Create dist directory mkdir -p dist # Move files into dist/ mv *.js *.map *.d.ts *.tsbuildinfo dist/ mv bin commands controllers cores dtos emails maintenance middleware \ repositories schema services sql-tools types utils workers dist/ # Verify structure ls -la /opt/immich/app/dist/main.js # Should exist now ``` **Restart service:** ```bash systemctl restart immich-web tail -20 /var/log/immich/web.log ``` **Result:** Same MODULE_NOT_FOUND error persists --- ### Fix Attempt #2: Reinstall node_modules ```bash cd /opt/immich/app # Remove existing node_modules rm -rf node_modules # Reinstall dependencies pnpm install --prod ``` **Expected Output:** ``` Packages: +636 Done in 8.1s using pnpm v10.28.0 ``` **Check symlink structure:** ```bash ls -la /opt/immich/app/node_modules/kysely ``` **Output:** ``` lrwxrwxrwx 1 root root 39 -> .pnpm/kysely@0.28.2/node_modules/kysely ``` **Verify target exists:** ```bash ls -la /opt/immich/app/node_modules/.pnpm/kysely@0.28.2/node_modules/kysely ``` **Output:** ``` drwxr-xr-x 4 root root 4096 dist/ -rw-r--r-- 3 1082 package.json ... ``` **Restart service:** ```bash systemctl restart immich-web ``` **Result:** Still fails with MODULE_NOT_FOUND --- ### Fix Attempt #3: Test Module Resolution ```bash # Test as root user cd /opt/immich/app node -e "console.log(require('kysely'))" ``` **Output:** ``` { Kysely: [Getter], sql: [Getter], ... } ``` ✅ **Works as root!** **But service running as immich user still fails:** ```bash systemctl start immich-web tail /var/log/immich/web.log ``` **Result:** MODULE_NOT_FOUND when running as immich user --- ### Fix Attempt #4: Fix Ownership ```bash # Set proper ownership chown -R immich:immich /opt/immich/app # Restart service systemctl restart immich-web ``` **Result:** Still fails with same error --- ## Machine Learning Service Issues ### Issue #1: Python Dependencies Missing ```bash systemctl start immich-ml systemctl status immich-ml ``` **Expected Error:** ``` × immich-ml.service - immich Machine-Learning Active: failed (Result: exit-code) ``` **Check logs:** ```bash tail /var/log/immich/ml.log ``` **Expected Error:** ``` ModuleNotFoundError: No module named 'gunicorn' ``` **Attempt to install:** ```bash cd /opt/immich/source/machine-learning pip install --break-system-packages -e . ``` **Result:** ``` bash: pip: command not found ``` ```bash pip3 install --break-system-packages -e . ``` **Result:** ``` bash: pip3: command not found ``` ```bash python3 -m pip install --break-system-packages -e . ``` **Result:** ``` /bin/python3: No module named pip ``` --- ### Issue #2: Install with uv (Partial Success) ```bash # Check if uv is available which uv ``` **Output:** ``` /usr/local/bin/uv ``` **Install Python dependencies:** ```bash cd /opt/immich/source/machine-learning uv pip install --system -e . ``` **Expected Output:** ``` Successfully installed [packages] ``` **Try to start ML service:** ```bash systemctl start immich-ml tail /var/log/immich/ml.log ``` **Expected Error:** ``` FileNotFoundError: [Errno 2] No such file or directory: 'python' ``` --- ### Issue #3: Python Executable Not Found **Fix:** ```bash # Create python symlink ln -s /usr/bin/python3 /usr/bin/python # Verify which python python --version ``` **Output:** ``` /usr/bin/python Python 3.13.5 ``` **Start ML service:** ```bash systemctl start immich-ml systemctl status immich-ml ``` **Result:** ✅ ML service now starts successfully ``` ● immich-ml.service - immich Machine-Learning Active: active (running) ``` --- ## Final State After all attempted fixes: - ✅ **ML Service:** Running (after Python fixes) - ❌ **Web Service:** Failed (MODULE_NOT_FOUND persists) - ❌ **Database:** Not migrated - ❌ **Photos:** Inaccessible via web UI --- ## System Information ```bash # Container OS cat /etc/os-release ``` ``` PRETTY_NAME="Debian GNU/Linux 13 (trixie)" VERSION="13 (trixie)" VERSION_ID="13" ``` ```bash # Node.js version node --version ``` ``` v24.13.0 ``` ```bash # pnpm version pnpm --version ``` ``` 10.28.0 ``` ```bash # Python version python3 --version ``` ``` Python 3.13.5 ``` ```bash # PostgreSQL version su - postgres -c "psql --version" ``` ``` psql (PostgreSQL) 16.11 ``` --- ## Working Restoration Method After failed upgrade attempts, complete restoration was achieved: ### Step 1: Destroy Broken Container ```bash # From Proxmox host pct stop pct destroy ``` ### Step 2: Reinstall Fresh v2.4.1 ```bash # Use community script bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)" # Use same container ID and configuration as before ``` ### Step 3: Reconnect NFS Storage ```bash # On Proxmox host mount -t nfs :/volume1/storage /mnt/storage # Bind mount to container pct stop pct set -mp0 /mnt/storage,mp=/mnt/storage pct start ``` ### Step 4: Restore Database ```bash # Enter container pct enter # Drop and recreate database su - postgres -c "dropdb immich" su - postgres -c "createdb immich -O immich" # Restore from backup gunzip -c /mnt/storage/backups/immich_pre_upgrade_20260127.backup.sql.gz | \ su - postgres -c "psql immich" # Verify restoration su - postgres -c "psql -d immich -c 'SELECT COUNT(*) FROM asset;'" # Should show your photo count ``` ### Step 5: Fix Database Password ```bash # Get password from .env grep DB_PASSWORD /opt/immich/.env # Set password in PostgreSQL su - postgres -c "psql -c \"ALTER USER immich WITH PASSWORD '';\"" ``` ### Step 6: Configure Storage Location ```bash # Add storage location to .env echo "IMMICH_MEDIA_LOCATION=/mnt/storage" >> /opt/immich/.env ``` ### Step 7: Start Services ```bash systemctl start immich-ml systemctl start immich-web # Verify both running systemctl status immich-web immich-ml ``` **Result:** ✅ Full restoration on v2.4.1 with all photos accessible --- ## Root Causes Identified 1. **pnpm Workspace Symlinks** - Symlinks work for root user but fail for non-root systemd services - Unprivileged LXC container permission issues - Module resolution path incorrect when files copied vs installed 2. **Python Package Management** - Debian 13 minimal install lacks pip/pip3 - Only `uv` package manager available - Python symlink not created by default 3. **No Migration Path** - Community script designed for fresh installs only - No documented upgrade procedure - No automated database migration 4. **File Structure Assumptions** - Systemd service expects specific directory layout - Simple file copy breaks pnpm workspace structure - Build artifacts require precise placement --- ## Expected Behavior The community script should provide: 1. **Update mechanism:** ```bash bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVE/main/ct/immich.sh)" # Should detect existing installation and offer update option ``` 2. **Automatic database backup** before upgrade 3. **Proper file structure preservation** during upgrade 4. **Database migration** handled automatically 5. **Service restart** with health checks 6. **Rollback capability** if upgrade fails --- ## Workaround for Now **Do NOT attempt manual upgrade.** Wait for official community script support for v2.5.0. **To stay safe:** - Remain on v2.4.1 - Ensure daily database backups are running - Create Proxmox container backups regularly - Monitor https://github.com/community-scripts/ProxmoxVE for updates --- ## Additional Notes - Total troubleshooting time: ~3 hours - Clean rebuild and restoration time: ~30 minutes - No photo data lost (stored on NFS, independent of container) - Database successfully restored from automatic backup ### Relevant log output ```shell ``` ### Additional information _No response_
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
accessibility
changelog:enhancement
changelog:security
changelog:skip
changelog:translation
cli
date-time
dependencies
documentation
external-library
format
good first issue
mobile-beta
mobile-beta
mobile-beta
needs-answer
nice to have
pull-request
Mirrored from GitHub Pull Request
 sharing
tech-debt
📱mobile
🖥️web
🗄️server
🧠machine-learning
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
OVERLORD
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: immich-app/immich#8343
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?