Tweaking4All.com - HandBrake - Optimizing Video Encoding (h264) (2024)

Optimizing video encoders is not easy with all the video lingo, trying to get a file as optimized as much as you can. With optimizing I mean finding a balance between file size and video quality.

In this article a few tips and tricks on how to optimize your encoder settings in HandBrake.

HandBrakeis freely available on multiple platforms (Windows, MacOS X, Linux) and is probably one of the better encoders out there.

Optimizing Video Encoder Settings

Optimizing or fine tuning of video encoders comes basically down to:

• What file size do I want?
• What quality do we want?
• How fast should encoding go?

Personally I believe that fine tuning should be a balance between file size and quality.

The time to encode a movie should be irrelevant – sometimes the time difference is really not worth it, for a few minutes extra you will get a better quality and/or smaller file size. The times that encoding a movie from DVD took 8 hours are long gone, most of my computers can convert a DVD easily under 15 minutes.

So based on personal experience (note that I’m not an expert)a few pointers:

The chosen file format, or “container”, has hardly any effect on file size or quality (MP4, MKV, AVI).

Since the “container” file has a minimal or even negligible impact on file size or quality. Therefor I’d recommend using the container that is most compatible with your devices and/or software. AVI is widely supported but MP4 is definitely the strong upcoming format to use. MKV is (in my humble opinion) better, but not as widely supported.

The common containers (AVI, MP4, MKV) typically support a variety of so called codecs (enCoder/Decoders)and the selection of the codec is what influences quality and file size.

Preferred setting in HandBrake: MP4.

A great codec is not a guarantee for quality but has a great influence on file size.

Some codecs (enCoder/Decoder) are more efficient than others when it comes to compressing and storage.
The compression settings can make it that an old MPEG-2 codec can actually out perform a modern codec like h264. These settings or options can also be the root of all problems. For example: older devices that are h264 capable might not yet support the newer “options” of this codec, resulting in crappy or even unplayable files.

Preferred setting in HandBrake: h264.

High bitrates do not guarantee quality but seriously impact file size.

Although in general there is a good correlation between quality and bitrate, a higher bitrate is not a guarantee for quality.

For example when transcoding a existing MP4 file with a low bitrate, to an MP4 with a high bitrate is total nonsense. It will only increase the file size but most certainly not increase quality – and probably even make the quality worse.

Note that so called “single pass” encoding can result in a less good quality or less efficient compression when compared to “multi pass” or “double pass” encoding – but that doesn’t have to be the case. The good thing of “double pass” – and that is just my opinion – is that it does a pre-flight analysis of your video source, allowing it to better anticipate changes in the video. “Single pass” encoding however has become significantly better these days.

Rule of thumb: When convertingnever use a bitrate higher than the source file.

Quality Setting for “Constant Quality” can be overdone.

HandBrake has a slider at “Constant Quality” allowing you to set an RF value, between 50 (low quality) and 0 (highest quality).

The default value is set to “20” which can be considered adequate for a DVD copy. For HD video (720p and up) an RF of 21 – 23 is recommended, and some even recommend “30” – but you’ll have to play a little with this to find what works best for you.

Also keep in mind that RF=0 means NO COMPRESSION at all, which is … useless. You’d end up with a gigantic file, larger than the original file, and with zero improvements on the picture quality. You will have to keep in mind that DVD’s and even Blu-Ray discs are compressed in such a way that they already come with “loss” – i.e. it kind-a has it’s own RF value already. Going below that value will not add improvements to your copy, and will only increase the file size unnecessarily. (reference)

Rule of thumb: DVD RF 20, BLU-RAY RF 22

Doing things in a hurry gives half-assed results.

This rule goes for pretty much anything in life: the difference between doing it fast and doing it well. The same goes for converting movies/video. The faster you try to encode a movie, the less “good” (in size and quality) the result will be.

Settings like “fast” basically means: we’re doing our best to maintain a reasonable quality, but speed of conversion is our main goal. So the slower we go, the more attention your encoder will pay to detail!

The Placebo setting goes totally nuts on quality, and takes forever to complete.

Preferred HandBrake setting: Medium (which gives a nice balance between speed and quality).

Sizing Video: Do crop the black borders, but do not “upscale” video.

Like with increasing bitrates from a low bitrate source, upscaling (or enlarging) the video resolution is probably one of the worst things you can do. First of all, you’re not adding any quality improvements, instead you’re adding only more useless data which results in a larger file for no good reason. Second of all: Your playback devices is typically much better at “upscaling” if needed.

Under normal circ*mstances, HandBrake will do a fine job in determining size, aspect ration, and cropping – it bases these settings on the information it gets from the video source you’re using. If you’d like though, you can look under the option “Picture Settings” and tweak cropping, size and aspect ratio. Just do not make the size more than what HandBrake suggests by default.

When it comes to cropping, HandBrake will remove the excessive “data” automatically, by removing the black borders often seen in movies (the gray area in the image on the right).

Please note: Cropping the black bars doesn’t really save a lot of disk space. It will however, as correctly noted by Thomas, have a potential negative side effect when it comes to playback. The cropped video (removing the black bars) will make it harder for devices to determine what the correct aspect ratio and/or resolutions are and as a result in possibly distorted playback.

Preferred HandBrake settings: Leave Sizing, Aspect Ratio.

Use filters or optimal settings for the type of video you’re working with.

Some encoders, like h264 in HandBrake, offer special settings and filters geared towards the type of movie or video you’re working with.

Let’s compare a regular movie with a cartoon as an oversimplified illustration.

What do we notice?The first thing you’d notice is that cartoons appear to be using a smaller palette of colors, shapes are simplified and filled with one color and there are hardly any shadings or patterns when compared to a regular movie. In the example above: simply look at the difference in fine details when you look at the hair – the picture on the left contains MUCH more data than the cartoon on the right.

These special settings use these characteristics to compress better and more efficient.

Preferred HandBrake setting: Match your video type or none if unsure.