Discussion Unload your server: Media compliant files.

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#1
Lately I see a lot of posts about transcoding, serverload bandwidth, should I go remote or stay local,.... Local machines have their advantages over remote machines. Then again, they have their flaws too. Running a machine truly local and then I mean storing the files locally too, takes away bandwidth issues. You can littarly stream any file locally without hastle. And to me, that's about it. Local machines are and can be expensive, especially if you need to provide all the storage for your so loved files. Especially if those files are big and chunky for example bluray disks, bluray remuxes, encodes containing HD audio. Those machines tend to be on a lot. I imagine someone wanting to watch a movie doesn't feel the need to take a little strole towards their mediaserver, powering it in, sit and wait till the thing decides to show up and then can start watching. Always on machines are key for meida servers. Now an always on machine is expensive in terms of power consumption. And there's the fact that thing trying to watch your media on the road. You'd need a decent internet connection to guarentee the upload and a powerfull server to do the hard needed transcoding. Let's face it, local isn't all fine and dandy.

So that brings is to remote servers. Well as we all know, there are some good machines and even VPS ones that do a hell of a good job performance wise. They are attached to decent internet connections and are always on. Hey, sounds like a good deal right? Well yes but there's still those bulky files that consume a lot of bandwidth and there's maybe still the element of transcoding. Let's talk about that last one for a second here.

Transcoding can eat a system alive, aspecially if you're doing video transcoding. Let's face it, plex does a poor job transcoding. Quite frankly, I hate it. The pre-defined bitrates and resolutions, they just don't match certain needs. Yes, I hear you think: but we need that transcoding to make our stull play. Well there's a couple of ways around that if you're willing to take some time to get your media ready. Why should this be handy? Well first of all, it will make your media files smaller so you'd have less bandwidth consumption. Second, less heavy dudty transcoding on your server because let's say you're playing a file that is just not mediafriendly to your device. Here are a couple of things to consider. Things that in my experience over the last couple of years preparing media for streaming make a lot of difference. Yes that's right, let's make our media stream ready.

1. Audio
Now I can imagine we all love our pounding audiotracks. DTSMA audio is simply amazing and the more channels we have around us, the better. Sad truth about these audiotracks is that they are very bulky. A DTSMA audiotrack can range up to a filesize of 3 GB for 7.1 audio. Also, there are some players that do not support DTSMA. DTS is a licensed product and it doesn't come cheap. Now I'm not talking about receivers. They can do the job. I am talking about players. Systems that run PMP. Not all of those handle these audiotracks well. Most of them support passtrough but then you're still ending up with a bulky and not so stream friedly file. Now there are a bunch of codecs you can conver your audio to. Most of them do degrade your audio quality. AC3 is such a codec. It is horribly outdated but due to the native support, still very popular. But, there's a fantasctic ompetitor that doesn't get the recognition it deserves. Yes folks, I'm talking about AAC. AAC is a very native supported audioformat. it is suppoted on mediaplayers over very different ranges of devices, including phones and game cosoles. In fact most of the audio you hear comming out of your favourate web player, is AAC encoded. What is so wonderful about this codec? It supports up to 7.1 audiochannels. Encoded with coreaudio (you need itunes installed for that) and the LLC profile at highest quality setting, it mesures up to DTS quality. Yes, it has been tested by very demanding audiophiles. Most importantly, it is small. A 7.1 AAC audiotrack takes up a filesize between 500-800 MB, depending on the dynamics of the audio and duration of the movie or TV show. For TV shows, most audio is 5.1 channels and they're shorter. Expect that that filesize to stay below 300 MB for sure. Now there you have it folkes, you just made your mediafile a hell of a lot streaming friendly and smaller in filesize. But we're not there yet.

2. Container
Most of of the stuff people have comes in the .MKV container. While MKV is getting more and more native support, there are still a lot of players that do not handle it very well. The player for Xbox One for example doesn't. MKV isn't stream friendly either. It is not compliant with a lot of streaming protocols. The two most compliant protocols in this area are in fact .MP4 and .TS. TS is widely used. Digital broadcast systems, still use TS as their means of video and audio container for transport and is supported by all major streaming protocols such as RTMP, RTSP, HTTP and so on. Another big competitor is MP4. This container gained it's popularity thanks to apple. m4a is the must go container for AAC audio. MP4 is the perfect package for audio, video and subtitle tracks. There are some drawbacks. This container doesn't support our beloved DTSMA and TrueHD audiotracks. Wait, but we were just getting rid of those right? Ah, no biggy then. Second drawback is the support for subtitles. It only supports text based subtitle files such as SRT and webvTT. Graphical subtitle support is limited to idx+sub. If you have files including the original bluray .sup files, you will need to fine a replacement. This brings me to the next point of interest.

3. Subtitles
Now those text based subtitle files are a hell of a lot smaller then those graphical ones. Those concist of .BMP images for every text line. Now since we are trying to get our files smaller, it's a good idea to toss those too. Also, stick to the languages you only wish to keep. Mind you that all those subtitle tracks when they are muxed in the stream, are being streamed, eventhough you don't use them. So, when it comes to subtitles, I have these rules:
- use text based subtitles
- only keep the languages you wnat
- keep them external;
Yes that's right, put them next to your audio/video container. Don't mux them in. That way, only the subtitle track that you wish to view is being loaded and streamed.

So we got our files a hell of a lot smaller with audio compression saving us about 2 GB and subtitle use saving us 30 40 MB. I know that last one isn't all that much but it is something. And, we made our file a lot stream more stream friendly and device compliant. Now, we haven't talked about the most important thing yet. Something that is a bandwidth eater on itself.

4. Video
Most of us love quality right? And yes, with all that 4K floating around and even HD Bluray videostreams are bitrate eaters. They can range up to 20-to 30 mbps or sometimes even more depending if you're watching 4K or HD. Yes, we can let plex transcode our video for us, but then we lose our precious video quality since it's doing a horrible job and above all, transcoding is a CPU eater. Now getting your video ready is a time consuming part but for those of you that want to go for the whole stream friendly, low bandwidth package worth looking into. Keep maximum bitrates for your video low. You do not want to create bitrate peaks. 25 mbps is high enough. Yes, I am talking about maximum bitrate. Rest asure only a couple of frames will ever get that. Go as low in bitrate as you can. streams below 10 mbps stream a whole lot better and are very low in filesize. Ok but what about quality you ask? Yes it can be done.L You can create below 10 mbit files that are bluray transparant. I've been doing it for a couple of years now, finetuned x264 settings to make it streaming compliant. Don't be fooled by huge bluray bitrates. The reason those are used is studio's have a 50 GB size disk to store on so in most cases bitrate doesn't even matter. X264 has a lot of tricks up its sleave to achieve good video compression. You just need to learn about settings.

Now that video thing, I'm leaving that one right up there in the middle as I do understand most people can't be bothered about it since it's too time consuming. For me, it is important.

So there you have it. You not only created your device friendly file, perfectly sized for streaming bandwidth and above all less server demanding but saved yourself a lot of dosh being able to run a remote server. If you guys wish to try one of those files out, feel free to PM me and we can arrange something.

peace out!
 

bubbadk

PG Captain
Tech Lead
Mar 18, 2018
334
93
28
44
Denmark
Reactions
93 0 0
#2
awesome article (y)(y)(y)(y)(y):):)
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#2
thnx man!
 

zukato

PG Specialist
Jul 13, 2018
22
7
8
Reactions
7 0 0
#3
Excellent post, thanks!

I used to use m4v/mp4 files but I noticed it is sometimes hard to find good subtitles, and checking each time if the srt is correct can be a pain. So I am now keeping my mkv files with vobsub subtitles. I wish I could find a good way to convert them into text files but I haven’t yet (used subler but more often than not there are some glitches).

Regarding your settings to encode with x264, would you care sharing some advice?

And last but not least, why aren’t you encoding with x265?
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#3
For all your subtitle needs and conversion:
https://subtitletools.com

My x264 settings are like this:
Code:
--level 4.1 --crf 21.5 --deblock -3:-3 --keyint 48 --min-keyint 24 --bframes 2 --b-adapt 2 --b-pyramid none --no-weightb --scenecut 99 --ref 5 --qpmax 50 --qpstep 2 --ipratio 1.1 --pbratio 1.1 --chroma-qp-offset -1 --vbv-bufsize 25000 --vbv-maxrate 20000 --no-mbtree --merange 48 --me umh --subme 9 --trellis 2 --no-dct-decimate --no-fast-pskip --colormatrix bt709 --aq-mode 3 --aq-strength 0.60 --deadzone-intra 10 --deadzone-inter 6 --sar 1:1
The ones I do have to mess around with depending on the video are --crf and --aq-strength. crf for the quality setting and the --aq-strength for bitrate distribution when I notice I have a serious lack in detail between lighter and darker areas in a frame. As you can also notice, a lot of the retention settings are turned off. It has come to my attention that a lot of these are actually bitrate eaters and can not be used in lower bitrate encodes. I also keep my --ipration and --pbratio low and at the same level. This prevents the quantizer from fluctuating too much over the different frametypes. The quantizer determines the quality of a frame. The higher that number is, the lower the quality. Also, I use a lot of ref frames and very low b-frames. B-frames are the worst looking frames in an encode. I know they're the most compressed ones but it has it's price. So swisth quantizers not fluctuating too much over all thos frames and b-frame count to a low, they still end up looking pretty good. I also use a non-conventional build of x264. I will upload it later on. It's an old version but for these encodes sit stands like a rock.

regarding x264. Good question I must add. For me it has to do that a lot of webplayers can't handle x264 yet. Also, players like plex for ps3. It is why I stick with x264 for 1080p. Haven't gotten round to doing 4K x265 since I have no means of testing properly. Now a lot of these settings can be implemented in x265. Maybe if you could spare a moment to test some things out, that would be cool. Who knows, maybe we could build ourselves a nice encoder script or something?
 

mythbuster

PG First Class
Sep 23, 2018
3
1
3
Reactions
1 0 0
#4
@SirTranceALoT - Excellent write-up, thanks for sharing! Do you have a recommendation for conversion software?

The beauty of Plexguide for me is the automation. I'd want to eventually try to incorporate preset settings into an automatic conversion process of the files (preferably after they download but before they upload to cloud storage). Any specific setting-tweaks you can share would be greatly appreciated!
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#4
The problem with conversion when you consider size and quality is that it can diverse in some cases. There are general settings that can stay the same though but it's not always a àne-click process. For the settings I use, see my comment above.
 

Admin9705

Administrator
Project Manager
Jan 17, 2018
4,680
1,984
113
Reactions
1,984 0 0
#5
stuck the thread. great write up
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#5
Thnx man!
 

bodgeup

PG Master Sergeant
PG Donor
Freelance Sponsor
Aug 12, 2018
64
21
8
Reactions
21 0 0
#6
Ive started to download H265 rips for 4k. The bitrate is much lower and the quality is just as good. Plex now seems to play H265 fine.
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#6
Yes it does. On most devices now even HDR is supported.
 

bodgeup

PG Master Sergeant
PG Donor
Freelance Sponsor
Aug 12, 2018
64
21
8
Reactions
21 0 0
#7
@SirTranceALoT i see your using quickbox.io script, how you getting on with that. i looked at that a while ago but it hadnt been updated in ages plus i found it doesnt give you true multi user access does it?? Also looked at openflixer, but none of those seemed to be as streamlined / automated as Plexguide is. But if it works for you it works.....
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#7
Yes true. i used quickbox for my seedboxing needs. That runs on a seperate server as plexguide. I'm very satisfied with quickbox. I mostly use deluge on there, axcept when I have to cross seed things. the beauty is that in the quickbox script, deluge is setup as remote client too, so I can use my deluge client at home to shuv torrents on that thing; I am using my server for solo seeding so no others are on there but I do believe you can add users to it trough the commandline. The pro version is completaly web based and even has a docker install now. Actually, the perfect mix would be plexguide and quickbox.
 

Datamonkeh

Data Hoarding Primate
Mod | Tech Lead
PG Donor
Jan 20, 2018
550
430
43
Reactions
430 0 0
#8
Some good points well made:)

A few bits to consider:-

Modern systems have sub 30w idles, my (ancient) LGA2011-3 x99 builds idle around 64w, Ryzen 1700 a little less, and an ancient i5 2500 (iGPU) around 55w from memory. Intel are the kings of power gating, if a core isn't used, it's draw is incredibly small.

In general (excluding devices where you are say on 4G or a bandwidth limited connection), if you're having to transcode, you're probably doing it wrong.

That said, Plex doesn't do a bad job of transcoding, emby and JellyFin use the same back-end, it's problem is that the output is h264 at whatever quality setting you choose and sometimes that's not ideal depending on the hardware used and/or the source media. This leads to several issues:

Software transcoding requirements (source)
  • 4K HDR (50Mbps, 10-bit HEVC) file: 17000 PassMark score (being transcoded to 10Mbps 1080p)
  • 4K SDR (40Mbps, 8-bit HEVC) file: 12000 PassMark score (being transcoded to 10Mbps 1080p)
  • 1080p (10Mbps, H.264) file: 2000 PassMark score
  • 720p (4Mbps, H.264) file: 1500 PassMark score
Do NOT transcode HDR to SDR, it's pointless unless you like horrible washed out garbage as tone mapping is broken. Kodi nightly builds got a first look at a fix a while back, but don't hold your breath.

4K SDR transcoding is 12k sized CPUMark performance hit (potentially less at lower BR), now that's a pretty tall order for many of todays CPU's, either way transcoding 4K is a bad idea, you are better off keeping a 1080 version and only sharing 4K content with those clients that can actually direct play it. Separate the libraries and use Tautulli to lock down transcoding 4K.

That brings us to PlexPass users, PlexPass users can make use of HW encoding to use a GPU to do the heavy lifting. You have three main players:

AMD - Nothing released up to 2018 should be considered suitable for HW Encoding, iGPU or dedicated.

Nvidia - 2 streams maximum on consumer GPU's, Quadra P2000 or better has 'unlimited' streams (and is a 75w card). Using the modified Ubuntu Nvidia drivers opens up 'unlimited' streams on consumer cards.

Intel - 'Unlimited' streams and decent support, often criticised due to old VAAPI driver issues, the 2nd gen iGPU was rough, it evolved over the generations and realistically 5th gen is where Plex say you should consider optimum, 6th gen is basically identical other than hw decode for HEVC. If you want low power, no dedicated GPU and efficient, intel is the better option at this stage.

Media choice is vitally important (as you rightly say), but that's not where people need to focus, consideration needs to be given to client choice. If you have the right client, then you choose the right media for it and transcoding is rarely needed. Some devices are a little quirky (Roku!) and some will play pretty much everything, but are massive overkill as media playback devices (eg Nvidia Shield TV). Other devices such as a Pi3 may struggle with 1080 h265 for example. The take away from this should be that the client dictates the media, not the other way round. With the proper media choice for the client, it removes the need to transcode. Personally I put clients on Roku devices, they are easy to manage/pair and work well, especially for the price. Ideally i'd choose wired devices, but both Amazon and Roku are seemingly in a race to the bottom in terms of price and that means cutting non-essential interfaces on the budget lines and the Roku Express is stupidly cheap/works well with a good remote, so I take them over FTV's.

Personally I run local servers with remote content, it works for me and my uplink is sufficient for my remote clients.
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#8
Great add there and a lot of valid points too. I also agree that the client dictates the media used. That was actually my intention why I wrote this.L Sorry if this didn't come across like that.

As for the encoding engine of both plex and emby. True. The beauty is that emby does let you alter certain settings. It makes it a bit more dependable on quality where plex is just one setting and then that's it.

Totally agree on transcoding 4K. It is pointless. It eats your system. Better to have a 1080p version indeed.

regarding HDR to SDR tonemapping. That's a very tricky one. The only encoder that can handle it properly is ffmpeg. Haven't tried it yet but the results look good. It's on my plans for testing though since some tonemapped 4K to 1080p looks better then a genuine 1080p bluray release. The remastered versions of The Matrix are living proof of that.

This is the type of discussion I love. Very constructive. Transcoding and mediasources are areas that are way too much overlooked and not considered enough. Thanks for your reply. A really great addition.
 

Dakpan

PG First Class
Jan 10, 2019
2
1
1
Reactions
1 0 0
#9
Hi!
I noticed your foot-text, and I am wondering 2 things;
1. Why have a seed server? what is your bandwidth?
2. Can you stream 4k?

Referencing too:

Server 1:
Plexdump:
Intel Core i7-3770
2x HDD SATA 3,0 TB
2x RAM 8192 MB DDR3
Host OS: ubuntu 16.04
1 VM running windows 7 Ultimate.
PG version: 7.2.4

Server 2:
Seedbox:
Intel Core i7-3770
2x HDD SATA 3,0 TB Enterprise
2x RAM 8192 MB DDR3
Host OS: Ubuntu 16.04 with quickbox.io script
1 VM running windows 7 Ultimate
 

MrDoob

PG Captain
Mod | Tech Lead
Mega Sponsor
PG Donor
Aug 4, 2018
302
108
28
31
GERMANY
Reactions
108 0 0
#9
Yes well , i can stream 4k direct play.
Audio decode no problem.
I test it with 4 aktiv streams.
The seedserver has more then 10Gb.
I can load over 1.ooo Mb/ps each file.
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#9
The reason I have a seperate seedbox is because I want to have my torrent traffic seperated from my plex server. I run 80+ torrents at the same time sometimes and they include full disks or remuxes to encode my plextreams from. Then there wouldn't be traffic jams. 4K streaming is no problem. I have it tested. Both servers have 1 gbps bandwidth.
 

Jstrawsine

PG First Class
PG Donor
Jan 8, 2019
8
4
8
Reactions
4 0 0
#10
So is there a way within Radarr, Plex, Sonarr... to get these files into a user/server-friendly format? Or is another application required?

Details would be appreciated! Great article.
 

SirTranceALoT

PG 2nd Lieutenant
Tech Lead
Jun 14, 2018
190
93
28
Reactions
93 0 0
#10
Actually I do my encoding manually since I download full disks or remuxes for movies. That process is a bit of work since I do comparescreens to determine the most optimal quality. For WEB_DLs of TV shows, I do it automated by implementing a script. See this post for more details.