mirror of
https://github.com/jellyfin/jellyfin.git
synced 2026-05-04 18:09:12 +03:00
[Issue]: Insanely slow startup out of nowhere #5085
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @OdinVex on GitHub (Sep 15, 2023).
Please describe your bug
Last week I could restart Jellyfin and it'd be up and running within 10s, this is with < 1.5K movies < 150 shows and < 10K music. Now it can take 15 minutes to get to the point of binding ports. I forgot to backup the
databasefolder or I'd rollback and test that. I noticed something peculiar. When first starting Jellyfin regardless of how it was shut down, it fillslibrary.db-journalfrom 0B to the exact same size as thelibrary.db, then the server starts and binds to ports and such. I tried a brand new database, perfectly fast startup, imported all content painstakingly, rebooted server, immediately back to 15 minutes startup time. Only one thing changed between last week and this, I added a few media items (movies).NVMe storage, plenty of space, simple dedicated quad-core 16GB NAS.
Jellyfin Version
Other
if other:
10.8.10
Environment
Jellyfin logs
FFmpeg logs
No response
Please attach any browser or client logs here
No response
Please attach any screenshots here
No response
Code of Conduct
@OdinVex commented on GitHub (Sep 15, 2023):
It doesn't matter how I shut down the service, it seems to rebuild the
library.dbevery start now. I've been running 10.8.10 for a little while, now. An sqlite check, recover, etc, nothing found, no issues found in any way. For comparison, startup used average 10s. Definitely under 30s.@srcrist commented on GitHub (Oct 8, 2023):
Closing this issue as it does not describe an actual Jellyfin bug. The creation of the rollback journal (library.db-journal) is a normal and expected part of sqlite functionality. To the extent that it is taking a long time to do so, the most likely culprit would be the speed of the storage upon which it sits. Your template is mostly incomplete and you left out a lot of detail which might provide context clues for where the bottleneck might exist, but there isn't anything in this issue that suggests a bug in Jellyfin's code. Please visit our support communities on the forum or our Matrix/Discord server and people can help you try to troubleshoot the issue.
@OdinVex commented on GitHub (Oct 8, 2023):
2x1TB 4GB/s NVMe (RAID 1, just for DB speed, I rely on external backups of course). Not a slow storage issue and it went from < 10s to 15m in just a day? Nah, something is wrong. The template is quite literally just that. It quickly gets to library loading but then takes forever (spitting NOTHING out) and then 15m later continues with no indication anything ever happened except time gone by. It never did this before and then suddenly did it (can only think updating Jellyfin caused it). Log literally had nothing in it that was different from a small library under 50 items (self-host test) versus itself. The library rollback journal simply gets re-created every launch. It's culled in size and then re-created.
@srcrist commented on GitHub (Oct 8, 2023):
I agree with you that something is wrong, but that thing just doesn't have anything to do with the Jellyfin code. Rollback journals are temporary files, and they are created and destroyed quite regularly. https://www.sqlite.org/tempfiles.html
@OdinVex commented on GitHub (Oct 8, 2023):
I'm familiar with rollback logs being used, though not fully as to each and every case when. I think my biggest issue is why it's being triggered (such a slow rebuild despite 2x1 RAID1 NVMe 4GB/s PCIE4.0) and why Jellyfin isn't communicating the reason it happens, or what is happening during the .db process. I understand the general situations in which it happens, but Jellyfin isn't reporting the state or reason. I tested the db file externally, it's intact and has integrity.
Edit: It's a data sanitation error for
.recoverhanging.typedbaseitems.overviewcontained unsanitized data on various entries for multiple entries. I've set theoverviewto an empty string for those offending artists,.recovercontinues now with no hang (250MB db). Offending entries were for "Nobuo Uematsu" (Musician, data was from Wikipedia) and "In Mac Anu" (forgot to check source). They contain non-ascii characters that hung.recover. Other entries contain Japanese characters and they're fine. Something about Nobuo's name in the first paragraph from Wikipedia that causes it to bork output. Has no effect on Jellyfin startup time, sadly.@OdinVex commented on GitHub (Jan 13, 2024):
Followup: Started from scratch, brand new installation, new files, no plugins. Issue present after adding all libraries. I might have some spare time to add only one library at a time to see if it is a specific one. May test removing each library one by one and restarting to see if it's an issue that can disappear (instead of some permanent thing after adding a library). Sqlite db testing shows no errors or issues whatsoever. Currently it takes between 10min-20min on average to start. I've also switched to the Docker, no change in start time.
@cryodream commented on GitHub (Jan 26, 2024):
I have been having this same problem for ever, it seems. 15-25 minutes startup time every time, for no reason and nothing in the log about why it takes so long, or what it happening...
Hardware:
Software:
The server is pretty much never above 30% - single digits almost all the time.
I know that the library is kinda big, but c'mon... Almost
half an hoursometimes for the program to start?It drives me completely crazy. I basically need to plan the restart way in advance. And every time I restart Jellyfin, I catch myself sitting there and pondering going back to Plex. I don't, of course, but I feel like I'm getting there...
Edit.
Just looked through my notes, and the first time I wrote down about this problem was:
@OdinVex commented on GitHub (Jan 26, 2024):
That single SSD is probably inefficient for a few users.. Core speed is also a bit low (per), but I'm certain there are multiple issues plaguing Jellyfin. An issue that causes a "recovery?" rebuild every start, threading/IO issues, cache-access issues (typically relieved by using tmpfs but even NVMe in RAID getting 7GB/s read and well over 3M IOPS has issues with just one user if there is load). As for the rebuild time, check our the
.dbfile when you start the server and at a point you might see another file being created slowly getting to the.dbsize...if so, you have the same issue I do.@cryodream commented on GitHub (Jan 26, 2024):
Sorry, but I very much doubt that the answer is in my hardware, TBH.
2 yearsago.I can try 2x Kioxia U.3 8TB NVMe drives in raid0 to test the slow drive theory, but I have a feeling it will not change a thing.
This feels like a bug or db/orm technical debt, etc. that will get fixed someday, hopefully.
@OdinVex commented on GitHub (Jan 26, 2024):
I didn't mean that the issue is your hardware, but rather a single 1TB SSD is capable of maybe 50 simultaneous streams with practically nothing else going on. IO being the biggest issue for media servers...I wouldn't have put the media on that drive because you mentioned that's where the containers and volumes are. I'm actually more towards what I said about you possibly having the same issue I'm having. If I watch the filesystem where the db is located and start the server it'll eventually not progress anymore but it will create a second db file that builds until it reaches the original db's size, then the process progresses and starts. Eg the long startup. Mine takes 12-18 minutes to start depending upon load.
@cryodream commented on GitHub (Jan 26, 2024):
@OdinVex thanks for the info.
I will try to check the filesystem happenings when on next Jellyfin restarts. If I won't forget, tbh...
I will most definitely try the new NVMe SSDs and I'm building a new Ryzen system next week, so maybe I'll do some quick test on that too. Fresh and clean unRAID, nothing else but fresh Jellyfin.
I'm pessimistic, though... because posts like these (from 3 years ago), do not give me hope that there is a hardware and not coding problem:
His 350MB files in 4.1 seconds vs my 824M file in half an hour... heh
@OdinVex commented on GitHub (Jan 26, 2024):
Again, I did not say there is a hardware problem. Pay attention. I was merely pointing out that it would not be ideal. I did not say it would be slow.
Mine is about his size but takes 12-18 minutes. I do remember this issue started suddenly and I can't remember if it was a specific Jellyfin version or if it was because I added so many entries for Music. (I had originally not used Jellyfin for music.) Either way, I think it's a bug related to Jellyfin or the Sqlite interaction or Sqlite. Given Sqlite's stability...I tend to think Jellyfin.
@cryodream commented on GitHub (Jan 26, 2024):
Oh, sorry, I wasn't clear enough. I do NOT think that you insist it's my hardware problem.
I've had this problem for so long, and I'm so tired with it... I'm tired of my family getting angry with me, that they can't watch their shows, etc.
I was just rambling and wishful thinking that maybe, just maybe, somehow it could be a problem on my end - then I COULD fix it. While at the same time, being reasonable and understanding that imho, it is most definitely not solvable by me on my end... sadly.
Edit. P.S.
Oh, and yes, iirc it just started happening one day. But it's been so long, that I cannot remember why. Not sure I noticed straight away too. All I can remember is thinking, that my Jellyfin could NOT be taking 20 minutes to start. I was sure I borked it and I did re-create my container and the whole server setup a couple of times, till I understood, that it was happening no matter what I did, and nothing helped, ie: it just became a fact of life from that point...
@OdinVex commented on GitHub (Jan 26, 2024):
Ah I see. That would be a relief, it was something local, but I'm thinking about debugging the issue myself by throwing in various poor-man's debug-prints to figure out the issue myself, though I doubt I'll be successful, I don't like debugging .NET with Sqlite.
@cryodream commented on GitHub (Jan 26, 2024):
I'm not a programmer, just a hobbyist, also I stopped touching C# about 6 years ago. I forgot everything I ever new about it, heh.
So I'm in no position to "get in there" and find something.
I really really hate .net
@OdinVex commented on GitHub (Jan 26, 2024):
New database, single start, imported, shutdown, database "has errors". Not the filesystem either, tmpfs, kept entirely in RAM. Recovered to a new sql, imported to a new db, no errors, startup...rebuilds it again. Something's wrong with either the content or Jellyfin.
Edit: Jellyfin corrupts the database by simply scanning items in. When it attempts to open the database on startup it 'repairs' and continues, adds the items back in and the issue happens all over again.
{null}(and it's supposed to be a long for use as a time!).{null}(and it's supposed to be either true or false!).It isn't just those entries. Multiple spots include duplicate guids or null values for
NOT NULLdata. Everything else in the entries is fine...save for missing info. A total of 2.8M entries were moved into a "lost and found" table during repair. The original data set only keeps 100K entries total.@rpatterson commented on GitHub (Jun 19, 2024):
I think I'm experiencing the same issue, having read through the description and comments: same ~15-20 minutes startup time, same behavior with the
library.db*files. I haven't repeated the fresh start or most of the other debugging described. I'm grateful to @OdinVex for all that. I did try restoring the DB from the most recentlibrary.db.bak*file that was from before this issue started, but the issue persists.Given that multiple users have confirmed this issue and that @OdinVex has confirmed it happens to fresh DB's, I think it's safe to say there is at least one bug in Jellyfin involved in this. Even if removing some content with bad or weird metadata from the library works around this issue, I'd say it's a bug in Jellyfin that bad data is allowed to cause something like this. Can @srcrist or someone else please re-open this issue?
@OdinVex commented on GitHub (Jun 19, 2024):
Given the nature...finding multiple GUIDs the same...it suggests thread issues, but maybe it's a herring. Still, it's definitely an issue.
@rpatterson commented on GitHub (Jun 20, 2024):
I'm adding this in case it provides a clue, but to be clear I doubt a plugin is the cause because @OdinVex already reproduced the issue on a clean install.
When I was taking a first stab at debugging this issue I noticed in the logs that the bulk of that long startup time was between loading the Trakt plugin and loading the DLNA plugin. This was consistent, the vast majority of the time always passes between those two log messages. I uninstalled and restarted to see if the DLNA plugin was the issue. After that, the bulk of the startup time was between Trakt plugin and the KodiSyncQueue plugin.
@OdinVex commented on GitHub (Jun 20, 2024):
And to add, it doesn't always happen right away, it's almost random as if it's a thread synchronization issue.
@rpatterson commented on GitHub (Jul 7, 2024):
TL;DR: The root causes seem to be malformed
**/*.nfofiles and Jellyfin failing to sanitize it's inputs.Debugging this has consumed a bunch of time. What follows are my notes and findings as I tried various debugging, testing and workarounds. I know they're long, but I hope some of them might be helpful to others some day.
When I started observing this issue, I noted 3 backup DBs:
I encountered another issue, frequent
The data is NULL at ordinal 1errors in the logs. Thanks to @Shadowghost for the workaround to rollback the DB and re-apply theFixAudioDatamigration. Later the jellyfin process started exhausting memory during the scheduled library scan, withRESgrowing to consume almost all physical RAM andVIRTgrowing to 260+ GB, effectively hanging the host with loads of 150+. I noted from the DB backups that the last time it was a reasonable size corresponds with that migration:I originally started noticing the long startup times because
HEALTHCHECKfailures were causing recurring restarts of the Jellyfin container. Maybe the same restarts also interrupted the migration and maybe that's a root cause of my DB issues? That seems likely because I exported my play states to Trakt, restored the./data/library.db.bak2backup, reverted that./config/migrations.xmlchange, and started the container with theHEALTHCHECK"disabled":It started and was responsive right away and applied the reverted migration, so far so good, nothing obvious in the logs. Then I scanned the whole library in ~2 hours with ~7 GB
RESmemory but not the ~22 GBRESthat crashes the container. It did, however, grow./data/library.dbback to the same 4.3 GB size as the broken DB. Running the "Optimize Database" scheduled task completed quickly and had no impact on size. I ran the Trakt import scheduled task and it completed without issue and without significant impact on theRESor the DB size. Then I added the "TV Shows" library to the Kodi add-on and theRESrocketed back up to ~22 GB, the host became unresponsive and the container crashed or was killed by something. I reproduced that through multiple attempts to add the library in Kodi.Next I tried removing the
./data/library.dbfile altogether and re-scanning the whole library. The./data/library.dbDB again grew to several GB, or many times the old DB size for roughly the same collection of content. Eventually, the library scan grew memory back to ~22 GB and the container died again. So I've reproduced the ~22 GBRESand crash both by scanning the library in Jellyfin and adding the library in Kodi.Next I tried moving the whole
/configdirectory aside to start from scratch and add the libraries one at a time. I confirmed that the jumps in the sizes of both theRESmemory and the./data/library.dbDB happened when adding my "Shows" library. Just adding the "Shows" library, even without any additional plugins, was enough to crash the container. This probably rules out the Kodi plugin as a cause and it's certainly not the only cause. I did some ad-hoc logging with:I noticed that the DB size would jump in size, 1+ GB at a time and doubling the size the first time, rather than growing continuously. I have no idea if this means anything, maybe it's just some sort of commit batching. Unfortunately, the
Verboselogging from the container that corresponds to those jumps doesn't provide any clues: multiple series, multiple components. The growth inRESsize was also not continuous, though not as dramatic as the DB size and maybe just jumping up because the data has grown. At this point, I suspected that the growth of the DB was the cause of the other symptoms. If the DB grows unreasonably huge, maybe that also increases theRESusage as the app tries to work with that huge data and then the host runs out of physical RAM and kills the container. And maybe what Jellyfin loads from the DB on startup is now so huge that it slows startup.Then I tried disabling all metadata and image downloader and fetcher plugins. IOW use only local
**/*.nfofiles for metadata. Adding the "Shows" library resulted in similarRES/DB sizes. This inspired a hunch, maybe the DB growth is caused by huge, malformed**/*.nfofiles that nonetheless contain XML. If Jellyfin isn't defensive about the content in valid XML, then absurdly huge data could very well cause all observed problems, especially if that data is duplicated in the DB, such as for storage and for indexed searches. So I measured the sizes of the**/*.nfofiles and indeed there are some huge files:Inspecting the worst offenders, I found that they contained extremely long
<title>elements with tons of repeated strings separated by/, including things that are definitely not parts of the title, such as actors' names. Data as important as<title>is almost certainly duplicated in the DB for indexed searching. I'm irrationally obsessed with preserving "original" data as much as possible, so I wrote a script to selectively fix the long<title>elements, but you could just as easily move these files aside and let Jellyfin or another tool re-download the metadata and write a cleaned file. That dropped the**/*.nfosizes by close to an order of magnitude:Now the worst offenders contained multiple top-level
<episodedetails>elements and none of them are for multi-episode files. The biggest had 178 apparently duplicate elements. Again, I wrote a script to remove all but the first elements, but it would be much simpler to move them aside and re-download as with the long<title>elements. That dropped the size of the largest**/*.nfofile to a reasonable size and looks like reasonable content:Note that something was overwriting the script's changes and restoring the duplicate
<episodedetails>elements, but the fixed remained in place when I stopped other media servers running against the same library, Plex and Emby in my case. I'm still debugging this. I wonder if that could be a root cause of some of the malformed**/*.nfofiles? Probably not, because the extremely long<title>elements accounted for most of the bloat both in the files and probably in the DB as well and those weren't restored. I would assume whatever XML library Jellyfin is using would ignore anything after the first close of the first top-level element. Please reply if you know anything about that!Next, again from a fresh
/config/, I added the "Shows" library. Now both DB andRESsizes grow gradually, end up quite reasonable, and I haven't observed a crash since cleaning up the**/*.nfofiles. The progress of the library scan also subjectively seems much more continuous without the long pauses/hangs I've observed for long before these issues (linked to just one of many issues related to scan hangs), so I wonder if bad**/*.nfometadata is a cause of that too.I note that the
VIRTmemory size is still 260+ GB. I don't know much about Linux virtual memory and it's not causing any problems I'm aware of so maybe there's a valid reason for that. It does seem huge and Jellyfin isn't the only process that deals with the same library of files and none of the others have anywhere close to thatVIRTsize. I also run another Jellyfin instance on a much smaller library and it also has 260+ GBVIRT. So maybe there's another issue here?I repeated the restore of the
./data/library.db.bak2backup as before, re-scanned all my libraries in Jellyfin, imported play states from Trakt, and repaired all libraries in Kodi. Startup is quick, <30 seconds, and both DB andRESsizes remain reasonable, and I have all my original data. IOW, I seem to be back up and running!Something still probably changed in Jellyfin, because these malformed
**/*.nfofiles have been in the library for years, confirmed with$ ls -l --time=creation. Even if they were damaged recently, I think this should still be considered a bug in Jellyfin, @srcrist, because it's vulnerable to malformed data from external sources and being defensive about that and sanitizing inputs is Jellyfin's responsibility even if it's also a bug in whatever generated the malformed XML. A few bad**/*.nfofiles from public torrents, for example, should not be able to bring down the whole media server.At a minimum, the component that ingests
**/*.nfoXML should truncate strings longer than a global string length limit. I think next most important would be to truncate lists and mappings longer than another global limit. I think it would also be good to audit which**/*.nfoXML "fields" end up getting indexed in the DB and truncating those values to a more defensive, shorter limit because long data that is indexed is likely to affect DB performance and truncating absurdly long data in the indexes isn't likely to be experienced as "lost data" by users. From the bad**/*.nfoXML I found in my library, it should be pretty easy to set these limits at points that will easily allow, for example, for unusually long but real<title>elements without allowing absurdly long, unreal elements that might DOS the DB.Just in case I'm coming off as pushy or entitled, I totally understand this is a volunteer community project and I'm not arguing that anyone should do this work. I'm arguing that we categorize this as a bug. I totally understand closing it as
not planneduntil someone wants to tackle this. If I didn't have to start by learning a new stack, .Net, I might take a stab at it myself.Finally, it may be that @OdinVex is experiencing a different issue. What is the current size of your DB? Did your DB jump in size around the time of the
FixAudioDatamigration? What is the total size of**/*.nfofiles in your library? If these turn out to be different issues, I'm happy to open a new one for what I've found.@OdinVex commented on GitHub (Jul 7, 2024):
@rpatterson This issue can happen even on brand new installs. The DB has never "jumped in size". I think you misread some stuff about the DB issue in this one. The DB is always rebuilt each startup before it resumes execution, not to mention actual SQL records in a dump are literal duplicate GUIDs and NOT-NULL columns having NULL values. Yours definitely is a different issue. 104.2MB of nfo files total in all libraries, just to answer the question. Edit: This issue also has no error logs of any kind. Sqlite literally 'rebuilds' the DB silently, never spitting errors or anything, Jellyfin doesn't even know it is happening.
@rpatterson commented on GitHub (Jul 7, 2024):
Do you mean an empty library or just a fresh
/config/with the same library files? Because I tested the latter as described in my previous comment.If you're referring to what you observed in the description:
Then yup, that's what I observed as well and with a 4.3 GB
./data/library.db, what ever was doing that seemed to take most of the ~20 minute startup times I observed. These are the observations I described in earlier comments that corresponded to yours and indicated we may have the same issue.Ok, thanks for comparing notes. I'll open a separate issue.
@OdinVex commented on GitHub (Jul 7, 2024):
I meant exactly what I said, a BRAND NEW INSTALL even with new unlibraried files and simply rebooting periodically with no touching. It does not always happen though, it's random. Once the issue is triggered then it will permanently cause the slow startup. Even a rebuilt .db file will still exhibit the issue, that's what makes me wonder if maaaybe the nfo files are also a problem? Maybe we have the same issue but different symptoms or causes. Bulk import (even of clean files) seems to trigger it sometimes too. Also, everyone using Docker Jellyfin should greatly increase the
stop_grace_period: 60mor Docker will kill it in 10s (the default). This should be highlighted front and center on Jellyfin's documentation. Premature shutdowns are dangerous for databases. Anything using Sqlite, MariaDB, MySQL, PostreSQL, etc, whether they're a client or a server.Edit: Again, it may be the same issue but different symptoms/causes since you mentioned the rebuilding bit. Still, duplicate GUIDs in unique columns is 100% a database issue (Sqlite should never have allowed it..), specifically threaded access. I'm not a fan of Sqlite because of the pitfalls around threaded access (rather, people might assume they can use single-threaded and then write their own multithreaded (broken) means of accessing it and this can break it). https://www.sqlite.org/threadsafe.html Duplicate GUIDs when the database specifically has those columns set up to prevent it is a failure in Sqlite (whether due to a bug or just misunderstanding of how to use it or the software's implementation in accessing it). I'm imagining it might be due to threading because it's often wrong and definitely can cause this kind of thing so it's where I'm pointing my finger in case any dev knows the backend and maybe this could spark some thought.
@rpatterson commented on GitHub (Jul 7, 2024):
Which is AMBIGUOUS, hence my asking for clarification. ;-) Does the "install" contain one or more libraries that contain media files?
What does "unlibraried" mean?
Yeah, that's why I thought it best to share my findings here first.
@OdinVex commented on GitHub (Jul 7, 2024):
I don't see it as ambiguous at all, I specifically mentioned earlier in the thread creating a new test library with new files on a new install even without plugins. I don't know how to make it any clearer, so I'll repeat myself. New install, new library, new files, overly obvious: there is nothing 'Jellyfin' about it, no nfo files, nothing. New files never having been imported into a library or in any way modified, no nfo files, no thumbnails, nothing, just literally a random NEW movie I've never had. No re-using folders, no shared folders, no overlays, no network filesystems, 100% local, 100% new, I don't think I can be any clearer, I don't think it's possible to be. Edit: Issue doesn't happen with a library containing one item, not sure if it'd be possible (maybe it could insert itself twice? some entries in the library .dump are duplicates by GUIDs). I slowly worked up to just adding one movie and one TV show, sometimes it'd trigger, sometimes it wouldn't, with literally 'nothing' happening except the server simply running unused. I think it's a thread-safety issue.
@rpatterson commented on GitHub (Jul 7, 2024):
There it is, that's what was ambiguous, that's not:
So your testbed does contain at least one library containing media files but no metadata files. Thanks for clarifying and best of luck!
@OdinVex commented on GitHub (Jul 7, 2024):
Not ambiguous. You just don't understand. Nothing pre-exists when I create a new installation. Nothing. NOTHING. NOTHING. NOTHING. No Docker volumes with any pre-existing anything, no Docker binds with pre-existing anything, nothing. Each deployment has literally nothing in it. After Jellyfin starts I'll upload some simple MKVs or MP4s or FLACs, nothing else. No metadata, nothing. No pre-existing libraries, no pre-existing files/folders, no pre-existing anything. Nothing, nothing, nothing. After Jellyfin scans any new MKV/MP4/FLAC I'd restart it. Sometimes the issue happens, sometimes it doesn't. With fewer items the issue is much faster. With more items the issue is much slower. Edit: So when I said nothing, I literally meant nothing. What I said is what I meant and it is clear. There's no partiality to "nothing".
I do know roughly when the Issue first ever appeared. It was somewhere around 10.4 or 10.6 that it began, but I can't recall when.
Edit: What I find so odd is that the library will rebuild to the exact same size...with the duplicate GUIDs and NOT NULLs being NULL. This suggests something 'stupid' about the rebuild process, that it isn't taking into account GUIDs can't be duplicate (they're UNIQUE) and NOT NULLs being set NULL can't be either, so I'm baffled as to why that happens too.
@slimshizn commented on GitHub (Aug 28, 2024):
I'm seeing these same issues on the latest release (10.9.10). Sometimes it will start right away, others it will take up to 30 minutes to start. No logs of any kind to go by.
@clseibold commented on GitHub (Feb 13, 2025):
Yep, same issues on the latest version of Jellyfin stable. The response from the Jellyfin developers is about the worse response one could get from an open source project - basically the "I don't care, it's not our problem" response. @srcrist If Jellyfin is taking a long time to load, like mine just took about 10 minutes to load, then that is a jellyfin bug. It might be Jellyfin not interacting with my hardware correctly. It might be some plugins acting up. It might be Jellyfin not working with some filesystems or other software on my linux machine. In every single case Jellyfin is not correctly interacting with something else, and in the case of the plugins, that means you have a terrible plugin API. So it is a Jellyfin bug, unless you're trying to tell everyone here that keeps having this issue that they are delusional. Is that the official position of the devs?
I also see Jellyfin's web homepage take forever to load (over a minute, sometimes multiple minutes), and then magically get significantly faster when Jellyfin is restarted. These are serious issues and I expect the devs stop trying to place the blame on irrelevant crap and fix your damn code, because I'm almost about ready to switch back to emby...
@thornbill commented on GitHub (Feb 13, 2025):
Use whatever works for you 🤷♂️
@clseibold commented on GitHub (Feb 13, 2025):
@thornbill Explain to me why Jellyfin takes anything more than 1 minute to start up? 1 minute should be the maximum start up time. Anything more than that, and the devs are doing it wrong. Plain and simple.
For once will you people just take some damn accountability and criticism? Because there is no way in hell you can try to tell me that it's acceptable that Jellyfin takes 10 minutes. It shouldn't even take 5 minutes. It should take 1 minute, just like most servers. Firebird starts up in a minute. Apache starts up in a minute. SQLite takes a minute or less. All of my own server software takes less than a minute. Gophernicus takes less than a minute. Nginx takes less than a minute.
So please, tell me your next excuse.
I'm not even saying y'all have to fix this. Just don't tell me Jellyfin isn't having the issues that I'm seeing it is having. Got it?
@maru801 commented on GitHub (Feb 13, 2025):
Please open an appropriate issue and post the relevant logs there. We can't attempt to help if we don't know what's going on with your system.
@rpatterson commented on GitHub (Feb 13, 2025):
I didn't hear anyone say they don't care, nor that a bug isn't the project's problem. I agree that many community-driven open source projects have an issue in how they respond to new tickets. Too often they find any reason to dismiss an issue and close it as
invalid. There's just no need for that when they can just say "This might be a real issue but the core developers have a long list of high priority issues ATM. If anyone else cares to tackle this, feel free to re-open and submit a PR. Until then, we're closing this issue aswon't fixin order to manage our issues." Triaging issues is much more difficult and time consuming that most realize. It's easy to do it wrong especially when they'd much rather be writing code. That said, I don't understand how anyone could imagine that bullying, threatening, and coming from a place of entitlement will help make any of that better, much less inspire anyone qualified to dig deeper into this issue. I know you've made following this issue worse for me.This is a community-driven open source project, and from my understanding that means that much, if not most or all, of the development and maintenance is done by volunteers. Report issues and capture findings, great! Tell the project volunteers to change the way they triage issues or otherwise what to do with their volunteer time, nope! You can keep capturing findings for closed issues and maintainers can re-open when the status changes and work starts. Most importantly, the constructive response to an issue you're fed up with is to do something about it. Learn what you need to learn, fix the issue and submit a PR; you can usually get help with the learning part. Or contribute to the project some other way (documentation, maybe even... triaging issues!) and use the connections you make in doing so to advocate for your personal favorite issue.
Huge thanks from me to the Jellyfin community! It's so awesome to have an open source media server that's competitive with the proprietary projects and I'm so grateful for that.
Regardless, please keep any further comments on this issue to users providing additional detail, which can help diagnose and identify root causes, or anyone tackling this issue and actually starting work on it. Thanks!
P.S. I am not involved with this project's community. I'm just a user with experience in the maintenance and development of open source community-driven projects. If any of what I said is wrong for this project, feel free to delete this comment or LMK and I will.
@clseibold commented on GitHub (Feb 13, 2025):
@rpatterson @maru801 Thanks for the responses. Don't get me wrong... I love Jellyfin and think it's great. But like... it has serious issues: either structural, or bug-wise, and telling people they are not issues is frankly disrespectful to the users of a self-established project. If this were some random repository with 5 stars, 2 collaborators, and little updates, then I wouldn't be saying this, but it's not. By establishing yourselves as a serious project, you have taken on accountability to your users. If you don't want that, then you should not have started this project.
This issue is marked as closed and "Not Planned", and there was a comment above literally saying "it does not describe an actual Jellyfin bug". You also shouldn't be closing issues that are still potential issues. That just don't make sense, and it bothers me that so many open source projects do this.
Like I said, by starting such a big project, the core members involved have taken on the accountability towards their large user base, especially when you have sponsors. If y'all don't want to fix issues, fine by me, but you cannot expect people to not criticize you. THAT is entitlement! You also cannot expect every single user to be a programmer who will do everything for you. THAT is disrespectful, especially when the purpose of programming projects is to make software for its users. But first and foremost, telling the users that the issues they are seeing are not actually issues is about the lowest one could go.
Regardless, Jellyfin upon start up should open the sqlite database, and then start up the web server and immediately open the sockets for the web interface and DLNA and other such things. And that's it. Everything else should be concurrent! Rebuild the database for all I care, just don't block the freaking web ui while you do it! If this is how it already works, then there's clearly something else wrong.
@clseibold commented on GitHub (Feb 13, 2025):
FYI @rpatterson , this "open source project" also has over 1000 people donating to it, is being financed by the Open Source Collective, and has two very big sponsors (DigitalOcean and JetBrains), so yes, I do think it's appropriate that users should be able to tell the core developers that they need to actually do something about the serious issues that uses are seeing. Sorry. Don't take donations if you don't want to be held accountable.
@srcrist commented on GitHub (Feb 13, 2025):
I have had about enough of this. This issue is closed. If you have an issue to submit with evidence of some sort of actual reproducible bug, please submit it in a new issue. Otherwise please end this discussion. This issue did not demonstrate an actual bug, and it has been closed. Move on.
Nobody gets paid to write this software for you. We do not have customer service. There is no manager for you to complain to. Fix it, submit an issue, or log off.
@maru801 commented on GitHub (Feb 13, 2025):
The actual main team behind Jellyfin is not that big as what you're describing it as.
You can see the current Jellyfin team leaders here.
A good chunk of work done to the project has come from volunteers that come and go.
There's no contract signed when sponsoring someone. It's perfectly fine to stop donating your money if you feel/see that the project or client is no longer serving your needs.
That being said, any of the core team members are not obligated to do anything for the community due to getting donations of any sort. There is no contract here and is not a form or source of employment for them. Jellyfin, for all involved, is a fun side project that people put time in out of desire, not need.
You need to clarify if these donations are all repeating periodically, or a one time thing.
Furthermore, the donations to the open collective fund is strictly for the hosting costs of the Jellyfin website and other cases related to some development. There was even a call for people to stop donating to it because it was getting too big for its purpose.
@gnattu commented on GitHub (Feb 13, 2025):
Web UI can’t function if the database isn’t up and running. Ideally, the database shouldn’t be rebuilt every time during bootup and most (verified) cases I've seen it behave like this is due to the underlying system is not suitable for sqlite or the database has some very very strange data/broken data. Who knows what your problem is? By posting a “me too” without any useful information or lengthy rants that essentially claim you’re the only one who knows how to run a project, while everyone else is just idiots?
Nobody has paid me a cent for all the works I spent to Jellyfin. And you can see how those money are spent right on the Open Collective page. We even told people that we don't need more money at the moment. Do you really think we have full-time paid developers?
@slimshizn commented on GitHub (Feb 13, 2025):
Logs have been posted by the OP of the issue long ago.
@joshuaboniface commented on GitHub (Feb 13, 2025):
There is a lot going on here.
First, the original underlying issues. I'm sorry to say but if it's happening to just you and not everyone - which it's not - then there's something wrong with your particular setup. SQLite is incredibly finicky with its storage. Some filesystems (including virtually all remote filesystems) do not work well with SQLite. This includes at least that we know of, CoW filesystems like ZFS and BTRFS, and NFS. Beyond that, we really need some more detail to be able to help. Enable debug logging and possibly
stracecapture the entire startup sequence on a fresh DB.Second, project criticism. As you can see, we're perfectly willing to accept criticism. No one is perfect. 6 years ago I had no idea what I was doing when I started the project, but yet here we are. Jellyfin is a purely volunteer project. I'm sorry but we do not have any obligation to anyone. I don't care how big the project is or gets, we're here to scratch our own itch. If you find it useful, great. If not, feel free to vent, use something else, fork it, whatever you want. But it won't make you friends unless you're willing to also help put in the work. Screaming "it sucks" from the rooftops does nothing to help it get better. Re: Money, as is clearly stated all over the donation page, the money helps us cover recurring costs (hosting our repo, forums, demo instances, our domain, possible admin expenses, etc.) and a one-time optional "if you need some hardware in a pinch submit the expense" policy for team members. That's it. It does not pay developers for development, and it never will. Every developer and everyone on the core team (including yours truly) makes nothing monetarily from Jellyfin, by design to keep it purely community-focused. We collect donations only because people kept begging to throw money at us, not because we wanted to, and because the costs to run the infra to distribute Jellyfin grew too large. Our "sponsorships" from Jetbrains and Digital Ocean are also minimal (a few free licenses, and a few hundred dollars per year of credit, about 1 month of costs, respectively). So, frankly it doesn't matter to us how many people have donated - donations are voluntary, are entirely separate from the code, do not influence the work actually done on the project at all, and do not in any way obligate anyone involved to do anything.
So, wrapping all this up to get back to the topic at hand: advice was provided early that something is likely wrong with the underlying setup of the systems in question, not with the code which works on 99+% of systems we know about. And exactly what has not been clear from the information provided. So we will need more (useful) information and specifics to be able to pinpoint this. Further, we are also very actively working on a complete DB refactor which hopes to significantly improve DB performance and get us off SQLite as a requirement. While only part of that work has been merged to
master, it might be worthwhile to test out the unstable builds to see if things are better, worse, or about the same there, as another data point.@OdinVex commented on GitHub (Feb 13, 2025):
For the last time, this occurs on DBs that have NEVER been on a remote filesystem (edit: or COW-filesystems). I posted logs. "Jellyfin (paraphrased): Not a Jellyfin issue." It's a Jellyfin issue. I built tools to manually dump the database before and after, it contained duplicate
GUIDs and nulls inNOT-NULLs (which trigger a rebuild because the DB is corrupt at that point), it's a Jellyfin issue. Duplicate GUIDs which are supposed to be impossible on a single-threaded operation happening...suggests it's a thread-access (read/write/whatever) issue. Later testing using atmpfson a direct install (so there's no drive/filesystem "corruption" to even be considered, used plain and simple ext4) and it still happened. Multiple different servers used to test it happening. Not always, just sometimes. My guess is a threading issue in Jellyfin, perhaps obscure. SQLite is 100% NOT friendly to multithreaded-access and explicitly states it over and over. I withdrew from the issue because it was obviously and clearly communicated Jellyfin has no plans to figure out what it is or nail it down, so **** it. Either fork or dump Jellyfin and go back to task schedules and scripts to do the same thing. Unsubscribing to prevent further notifications.Edit: As for anyone mentioning donations/money, no idea what's going on there. I never brought anything like that up and I don't expect anything from Jellyfin except something other than a dis-acknowledging, unhelpful response such as "not a Jellyfin issue" (given situation).
Edit: But for what it's worth: I'd rather have an external database, real SQL, maybe even some Redis caching for pages.
Edit: Regarding the DB refactor, getting off of SQLite would be wonderful and hopefully this eliminates (or at least highlights the reason for) the DB corruption issue(s).
@cvium commented on GitHub (Feb 13, 2025):
It is indeed quite well-documented how to corrupt the SQLite DB. If you take the time to read this document, you'll also see that there's realistically only 1 scenario in which Jellyfin could be at fault (Section 7. SQLite Configuration Errors). However, we do not use the "dangerous" configuration values.
This is just arguing in bad faith. Multi-threading is very much safe and supported, assuming you adhere to their recommendations, which we very much try to. If many people had the "slow startup" issue, I'd buy into it possibly being a misuse of the SQLite connection or its objects, but it's not common at all, which points more in the direction of it being a "hardware" issue.
That is not to say that it can't be a Jellyfin issue, but there's only so much time we can spend on one single hard-to-reproduce issue that affects a handful of people.