My experience was also that Plex is better at working around the limitations of Apple TV, compared to Jellyfin.
E.g., the native, built-in Apple TV video player expects everything in an MP4 container, and Plex appears to seamlessly repackage content into MP4, whereas you need to dig into the settings to force an MP4 container and force the native player on Jellyfin. Otherwise, Jellyfin will try to use VLCKit or some other player that has other issues (like with HDR content).
Apple TV also has very limited subtitle support in the native player, and Plex does better at accommodating that.
I'm assuming Jellyfin is closer to parity on hardware with more capable native players, though.
If you haven't already, check the CPU usage on your Jellyfin server during laggy playback. It could be transcoding on the fly, which can noticeably affect performance.
I recently used an old thin client with Kodi as Jellyfin client. Controlling it with a remote feels horrible, I’m now planning to reinstall some plain linux (instead of LibreELEC) and try out the jellyfin desktop client (it supposedly has a TV setting), surely it can’t be worse.
I've done both and the only major reason I used jellyfin was to track watching on that server rather than Infuse as for some reason Infuse tends to reset stats on me.
Of course transcoding on the fly and stuff is there but I just want to stream the direct file to the tv so none of that matters.
*edit: For anyone else reading down here the parent's solution was better for me vs Plex. Simply streaming through infuse via SMB removes all the issues I don't want to deal with with Plex as it has gotten worse over the years and has invaded my purely local setup more and more. As long as it is encoded in a way Infuse can read it works great. I didn't have a file Infuse couldn't play.
You're tied to the iOS family using the built-in library, and tags and indexing get to be a big burden if your collection grows into the 10+ TB. Jellyfin gives you location, platform, and codec independence.
My understanding is that Plex&Co's big selling point is transcoding, which might be valuable if your clients have less horsepower and don't natively support whatever format your content is in.
The plex app refuses to play 4K and says my connection is too low. Doesn’t matter if it’s wired or an ap 5 feet away, my iPhone gets almost 850mb down in the same room. If I play the 4K file directly from VLC I see absolutely 0 stuttering or glitches. I really don’t understand why plex has issues direct streaming (no transcoding). The computer has 16 cores so it’s not the problem, only plex has the issues.
Plex has a much better UI and UX than anything else.
Plex has a configurable bit rate limit on the server side for streaming. The default is too low for a lot of 4k media. Just set it to something stupid high.
That sucks, I was about to do the same (Jellyfin on the Synology, Apple TV for streaming). Seems there is no alternative to Plex that works with an Apple TV...
Actually, the Infuse app is almost perfect for Jellyfin on Apple TV. Highly recommended if you don't mind the small subscription. It takes load off of the server by playing a wide variety of codecs natively. There's also an OSS app in fast development that looks promising, but only Android/iPadOS for now.
The lack of a corporate entity behind Jellyfin makes a big impact on the quality of their streaming clients. On some platforms they work flawlessly, on others there are excellent third party clients, and some platforms don't attract a lot of open source devs and provide a lacking experience.
Jellyfin split off from Emby years ago, which does have a commercial side. Perhaps their Apple TV client is better?
Seconding Jellyfin. I installed Tailscale on all my devices, including the media server, to achieve free remote playback without worrying about trying to secure the server against the whole Internet.
I'm no expert, but I think you could have the reverse proxy node be the only thing whitelisted to be accepted by the nodes you want behind the proxy (via their tailscale IP). I believe this would all be done in their ACL JSON.
Tailscale offers a super basic reverse proxy called tailscale serve. Or baring that caddy has built in support. They have the full details for both in the tailscale docs.
My biggest frustration with Jellyfin is mostly related to using custom or self-signed SSL certs, the clients can't seem to cope with that (especially on Google TV).
As plex got more annoying to use I switched to Jellyfin on my Roku. It's been great. Reading the threads it does seem like there's some issues on the Apple TV, but on the Roku it's great.