[Bug] High Ram Usage When Transcoding with NVIDIA NVENC #4767

Closed
opened 2026-02-07 01:08:29 +03:00 by OVERLORD · 5 comments
Owner

Originally created by @LoopedSdrawkcaB on GitHub (Apr 5, 2023).

When NVIDIA NVENC is selected to transcode, memory usage slowly creeps up and uses the 16GB allocated to the VM. I have reverted back to no hardware transcoding and the memory issue disappears. When closing the stream it seems the memory usage hangs and then drops down to about 5GB of usage until I reboot the machine.

Software

  • Proxmox 7.3-3
  • Ubuntu 22.04.2
  • Docker / Docker Compose
  • Jellyfin 10.8.9
  • Fresh Install
  • Nvidia Driver Version 525.89.02

Hardware

  • GPU is P4000
  • CPU Xeon(R) W-2133 CPU
  • 32GB Ram
Originally created by @LoopedSdrawkcaB on GitHub (Apr 5, 2023). When NVIDIA NVENC is selected to transcode, memory usage slowly creeps up and uses the 16GB allocated to the VM. I have reverted back to no hardware transcoding and the memory issue disappears. When closing the stream it seems the memory usage hangs and then drops down to about 5GB of usage until I reboot the machine. Software - Proxmox 7.3-3 - Ubuntu 22.04.2 - Docker / Docker Compose - Jellyfin 10.8.9 - Fresh Install - Nvidia Driver Version 525.89.02 Hardware - GPU is P4000 - CPU Xeon(R) W-2133 CPU - 32GB Ram
OVERLORD added the media playback label 2026-02-07 01:08:29 +03:00
Author
Owner

@nyanmisaka commented on GitHub (Apr 6, 2023):

@LoopedSdrawkcaB Can you share the ffmpeg transcoding log created by the stream you mentioned?

@nyanmisaka commented on GitHub (Apr 6, 2023): @LoopedSdrawkcaB Can you share the ffmpeg transcoding log created by the stream you mentioned?
Author
Owner

@LoopedSdrawkcaB commented on GitHub (Apr 6, 2023):

Transcode Log.txt

Here it is

@LoopedSdrawkcaB commented on GitHub (Apr 6, 2023): [Transcode Log.txt](https://github.com/jellyfin/jellyfin/files/11163178/Transcode.Log.txt) Here it is
Author
Owner

@nyanmisaka commented on GitHub (Apr 6, 2023):

No errors in the log. I don't see any reason NVENC or CUDA will eat up such memory.

Can you check the RAM usage via top or nvidia-smi and find out which process is the cause?

@nyanmisaka commented on GitHub (Apr 6, 2023): No errors in the log. I don't see any reason NVENC or CUDA will eat up such memory. Can you check the RAM usage via `top` or `nvidia-smi` and find out which process is the cause?
Author
Owner

@Otako77 commented on GitHub (Apr 6, 2023):

My system is significantly weaker than yours and I have no problems.
My system:
I3-6100
8GB RAM
GTX-1650 4GB (5.25)
Ubuntu Linux, all native, no dockers

I can encode 5x UHD HDR -> Full-HD 10 Mbit or 7 other streams until the system is at its limit.

Image 1: System is idle. 1.93 RAM is used
Bildschirmfoto 2023-04-06 um 14 42 45

Figure 2: System encodes 4 streams from UHD HDR->FULL-HD 10 Mbit (3 GB Ram is used)
Bildschirmfoto 2023-04-06 um 14 47 21

Figure 3: System is idle again, 2GB RAM is in use

Bildschirmfoto 2023-04-06 um 14 51 10

The transcoding uses about 250 MB RAM per UHD-FULL-HD stream

@Otako77 commented on GitHub (Apr 6, 2023): My system is significantly weaker than yours and I have no problems. My system: I3-6100 8GB RAM GTX-1650 4GB (5.25) Ubuntu Linux, all native, no dockers I can encode 5x UHD HDR -> Full-HD 10 Mbit or 7 other streams until the system is at its limit. Image 1: System is idle. 1.93 RAM is used <img width="1422" alt="Bildschirmfoto 2023-04-06 um 14 42 45" src="https://user-images.githubusercontent.com/88830944/230386179-e96d9716-6339-441c-bb90-c8fea845f0a9.png"> Figure 2: System encodes 4 streams from UHD HDR->FULL-HD 10 Mbit (3 GB Ram is used) <img width="1422" alt="Bildschirmfoto 2023-04-06 um 14 47 21" src="https://user-images.githubusercontent.com/88830944/230386233-ed7ac70f-1423-4c26-bff2-dfb9bbe6da25.png"> Figure 3: System is idle again, 2GB RAM is in use <img width="1202" alt="Bildschirmfoto 2023-04-06 um 14 51 10" src="https://user-images.githubusercontent.com/88830944/230386283-3d951b52-8a83-492e-9e97-578d4fb3be86.png"> The transcoding uses about 250 MB RAM per UHD-FULL-HD stream
Author
Owner

@nyanmisaka commented on GitHub (Apr 15, 2023):

Close as cannot reproduce on both Linux and Windows.

@nyanmisaka commented on GitHub (Apr 15, 2023): Close as cannot reproduce on both Linux and Windows.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/jellyfin#4767