mirror of
https://github.com/jellyfin/jellyfin.git
synced 2026-05-04 18:09:12 +03:00
[Issue]: Very slow to load initially, and also dashboard components #4062
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ragebflame on GitHub (Jul 26, 2022).
Please describe your bug
When initially accessing Jellyfin via the Browser or Android app, it is very slow to load any content. It is also very slow for some of the Dashboard components to be populated. The Jellyfin logs below are from navigating to the Dashboard page. The server Info is very slow to load.
Server: Jellyfin-laptop
Version: 10.8.1
Operating System: Linux
Architecture: X64
From the Debug level logs I can see that the request is taking 4 seconds to complete:
[20:49:47] [INF] [21] Microsoft.AspNetCore.Hosting.Diagnostics: Request finished HTTP/1.1 GET http://media-laptop.lan:9004/jellyfin-laptop/System/Info - - - 200 - application/json;+charset=utf-8 4043.0172msBrowsing media and playback is all working without issue. This has only started occurring since upgrading to 10.8.0.
Steps to reproduce:
Jellyfin Version
Other
if other:
10.8.1
Environment
Jellyfin logs
FFmpeg logs
No response
Please attach any browser or client logs here
Requesting http://media-laptop.lan:9004/jellyfin-laptop/Sessions?ActiveWithinSeconds=960
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Requesting http://media-laptop.lan:9004/jellyfin-laptop/ScheduledTasks
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Sending web socket message: SessionsStart
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Sending web socket message: ScheduledTasksInfoStart
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Requesting http://media-laptop.lan:9004/jellyfin-laptop/System/Info
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Requesting http://media-laptop.lan:9004/jellyfin-laptop/System/ActivityLog/Entries?startIndex=0&limit=7&minDate=2022-07-25T20%3A28%3A45.311Z&hasUserId=true
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Sending web socket message: ActivityLogEntryStart
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Requesting http://media-laptop.lan:9004/jellyfin-laptop/System/ActivityLog/Entries?startIndex=0&limit=4&minDate=2022-07-19T20%3A28%3A45.312Z&hasUserId=false
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Sending web socket message: ActivityLogEntryStart
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Requesting http://media-laptop.lan:9004/jellyfin-laptop/LiveTv/Recordings?UserId=b68358d0aa4e4943a4e294fc4ec8adcc&IsInProgress=true&Fields=CanDelete%2CPrimaryImageAspectRatio&EnableTotalRecordCount=false&EnableImageTypes=Primary%2CThumb%2CBackdrop
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Requesting http://media-laptop.lan:9004/jellyfin-laptop/ScheduledTasks?IsEnabled=true
main.jellyfin.bundle.js?709b29ddaf40620d691b:2 Sending web socket message: ScheduledTasksInfoStart
Please attach any screenshots here
No response
Code of Conduct
@ServeurpersoCom commented on GitHub (Jul 29, 2022):
Same problem here. it run smooth for about few hours and fastly slow down especialy on the admin pannel.
Tested native install on Debian 11 without VM or Docker. i9 12900K 64GB DDR5 !
Admin dashboard kill about 25% CPU and very slow to populate. Restart the service solve the problem for few hours.
@ragebflame commented on GitHub (Sep 26, 2022):
Issue still occurring in 10.8.5
@FleyX commented on GitHub (Sep 27, 2022):
It could be avoided by useing IP and PORT ,not domain.
@Cueball666uk commented on GitHub (Sep 29, 2022):
Having the same issue. Slow very to load, but then fine whilst using. But slow again the next time.
@r0ckinit commented on GitHub (Oct 2, 2022):
Same issue here as well
@bloominstrong commented on GitHub (Oct 3, 2022):
I have been experiencing this issue as well, to the point that the Android App will timeout trying to connect when selecting the server by entering the URL.
I have been doing some testing and it is definitely to do with the the disk i/o and name resolution when accessing http://<JELLYFIN_DOMAIN>/System/Info/Public and is only affecting versions from 10.8.0 (rolling back to 10.7.7 resolves the issue). Other pages do not appear affected (http://<JELLYFIN_DOMAIN>/health returns straight away) but I have not found a good example that would need to read from disk.
As FleyX mentioned the work around is to access the server via http://<IP_ADDRESS>/ rather than http://<YOUR_DOMAIN>/ http vs https and port mapping does not matter.
An easy way to test is to
curl -v http://<JELLYFIN_DOMAIN>/System/Info/Publicand look forX-Response-Time-ms:For myself when using my domain the response is ~5 seconds and when using IP address the response is sub 5ms (10.7.7 is a similar response time for both domain name and IP address).
My disk IO is constrained due to ZFS with a high number of snapshots so if I clone the ZFS volume the issue is resolved as that resolves the disk IO, if I spin up a brand new Jellyfin instance (new default config) on the same ZFS volume the issue remains so the issue is not related to the Jellyfin config but the underlying disk IO.
So that explains the symptoms and why spinning up a new Jellyfin instance does not show this issue. As to the cause looking at the 10.8.0 commit log there appears to be a few changes about domain resolution. I have not worked in dotnet before but will work through these and test reverting them to see if I can identify the cause over the next couple of weeks as I have time.
I will also see if I can find an easy way to replicate the issue so others can assist.
@ServeurpersoCom commented on GitHub (Oct 4, 2022):
For me the problem (since 10.8.0 and to 10.8.5 last stable) appear when there is about 10 users and when I go to admin panel. User information don't appear and server page and streams freeze for everyone for about 10 seconds and way more if I keep on the admin panel page. This is an important bug make Jellyfin unusable and make me rollback to 10.7.7
@bloominstrong commented on GitHub (Oct 7, 2022):
Looking at your issue it does sound slightly different, as this issue seems to center around the <JELLYFIN_SERVER>/System/Info path taking a long time to return when using the domain as opposed to just the IP address.
I know you already rolled yours back but does using IP vs domain name give you a difference?
@Cueball666uk commented on GitHub (Oct 7, 2022):
Yes, it's much faster without the domain. But I would appreciate using the domain instead of my IP address. I had no such issue on 10.7.... but I also didn't notice this issue until i started using the domain, my downgrade back too 10.8.4 has not saved me from the issue either.
@ServeurpersoCom commented on GitHub (Oct 7, 2022):
No difference for me. Admin dashboard (only the "who / user list") consume about 33% of an i9 12900K for few minutes. If I keep on this dashboard this freeze entire Jellyfin server, if I keep only few seconds, it consume also 33% for about 10 seconds and no freeze.
@ServeurpersoCom commented on GitHub (Oct 7, 2022):
I use Apache SSL Websocket reverse proxy, jellyfin server get the same request in all case (domain or public IP)
@cvium commented on GitHub (Oct 8, 2022):
It's not something I can reproduce. Maybe some debug logging can shed some more light on this. Alternatively, you can also send me your database files
@ragebflame commented on GitHub (Oct 8, 2022):
This is correct. If I access my Jellyfin instance via IP & PORT rather than via the reverse proxy (linuxserver/swag) I get no unusual slowness. Sounds like the issue may not be with Jellyfin, but rather my reverse proxy setup.
@Cueball666uk commented on GitHub (Oct 8, 2022):
I'm using Jellyfin with a certificate directly added to it (for my domain) and a port forwarded on my router. No reverse proxy at all... But I'm still experiencing the slowdown via domain/port vs IP/port
@ragebflame commented on GitHub (Oct 8, 2022):
Okay, so possibly still a Jellyfin issue. I've not had time to investigate things further on my end.
@mrautlan commented on GitHub (Oct 10, 2022):
I have exactly the same issue. Access via IP and port is fast, via domain and port is very slow. I'm also not using a reverse proxy. A similar issue occurs when I use the Jellyfin app (iOS) to connect to the server via domain/port - "unable to connect to the server". IP/port works fine.
@Cueball666uk commented on GitHub (Oct 11, 2022):
Yes, it's much faster without the domain. But I would appreciate using the
domain instead of my IP address. I had no such issue on 10.7....
On Fri, 7 Oct 2022, 12:16 pm bloominstrong, @.***>
wrote:
@cvium commented on GitHub (Oct 11, 2022):
May I suggest actually doing what I'm asking? https://github.com/jellyfin/jellyfin/issues/8179#issuecomment-1272243846
@bloominstrong commented on GitHub (Oct 12, 2022):
I have not been able to reproduce the issue by limiting IO on a fresh instance but cloning the volume that my instance runs on removes the issue so it does not seem tied to something specific in the database.
Cranking my logging up to
I get the following covering the JELLYFIN-DOMIAN/System/Info/Public request
Specifically this line seems to be the one to blame.
[11:10:51] [DBG] [22] Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker: Executed controller factory for controller Jellyfin.Api.Controllers.SystemController (Jellyfin.Api)Following that leads to Emby.Server.Implementations/Emby.Server.Implementations)/ApplicationHost.cs and the git diff to v10.7.7 the two commits that affect that function are Dynamically populate LocalAddress based on HTTP request and Improve platform checks with the former seeming the likely culprit.
@cvium commented on GitHub (Oct 12, 2022):
I don't really see how either of those PRs could affect it in such a way
@SuperCatss commented on GitHub (Oct 19, 2022):
Same problem, when I access via LAN it is fast and when I access via internet (via reverse proxy as well as port forwarding) it is very slow.
@cvium commented on GitHub (Oct 19, 2022):
Works for me
@SuperCatss commented on GitHub (Oct 20, 2022):
When I go to the domain name, the request "/System/Info/Public" usually takes 40 seconds to respond. When I access via LAN, it only takes 1 second.
@cvium commented on GitHub (Oct 20, 2022):
There's shockingly few logs being posted. I need a complete debug log. Enable debug logging, restart Jellyfin and upload it.
@matvey00z commented on GitHub (Oct 20, 2022):
I have similar issue here when run jellyfin in docker with access to internal docker network only, running behind nginx reverse proxy on a public domain. The corresponding logline is:
If I allow access to the internet, basically adding
to my compose.yml, the issue disappears and it loads much faster.
Also, I do observe that switching from the domain name to ip address solves eliminates this long load too.
So, it seems that it tries to do some external requests and then times out.
And imo it looks similar to #6918
@matvey00z commented on GitHub (Oct 20, 2022):
One more observation.
This request takes 10 seconds:
This responds immediately:
Meaning it's a hostname in the request what matters, not the actual path the request goes through.
@cvium commented on GitHub (Oct 20, 2022):
I cannot reproduce it, but if I were to guess, I'd say it's this line
4c1286fd24/MediaBrowser.Common/Net/IPHost.cs (L431), which is triggered by this call tosource.Addresse299adc819/Jellyfin.Networking/Manager/NetworkManager.cs (L362), which is called by3ff78b687d/Emby.Server.Implementations/ApplicationHost.cs (L1077). It was introduced by this PR https://github.com/jellyfin/jellyfin/pull/6486.A PR is probably coming soon.
@ServeurpersoCom commented on GitHub (Oct 30, 2022):
Look fixed for me on 10.8.6, problem hard to reproduce but I'm about 90% sure with a few days of heavy use.
@matvey00z commented on GitHub (Nov 7, 2022):
Confirm. Missed the 10.8.6 update but 10.8.7 works fine for me.
@ServeurpersoCom commented on GitHub (Nov 3, 2023):
I learned a lot tonight. On last stable, with 42 testing users the server saturate with 100% I/O on jbd2/nvme0n1p2- This process is the EXT4 kernel thread for FS Journal.
echo 1 > /sys/kernel/debug/tracing/events/ext4/ext4_sync_file_enter/enable
cat /sys/kernel/debug/tracing/trace_pipe
.NET ThreadPool-2460896 [009] .... 50322.096955: ext4_sync_file_enter: dev 259,2 ino 60555275 parent 60590080 datasync 0
.NET ThreadPool-2460519 [003] .... 50322.247079: ext4_sync_file_enter: dev 259,2 ino 60555275 parent 60590080 datasync 0
With a very high frequency SPAM from .NET ThreadPool, from (lsof) library.db SQLite writing (WAL disabled),
I moved this database only (db file and journal file) to a tmpfs Ramdisk, and now I get 0% I/O problem and a very low server load with the same amount of users, and I pushed up to 200 fake random testing users without any performance problem (only network is the limit because I don't use any video transcoding).
BUT even with this huge performance bump, the Daskboard problem is not solved : Even a very short access to it freeze the entire server for all users (freeze at 0kbps) and stream upload resume few seconds later.
The dashboard issue is not related to the SQLite performance contrary to what I believed. There is a serious bug that I can't dig deeper into:(