High Quality H264 Encoding Guide
Posted: May 4th, 2012, 1:13 pmHello. After being a longtime devotee of XviD for converting DVD-quality source video down to a smaller and more manageable file size, I've eventually taken the leap to x264 MPEG-4 AVC encoding. 
Why have I waited so long? Because I was never entirely satisfied with the results I achieved with earlier versions of x264 unless a relatively high bitrate 'brute force' approach was used. The x264 developers joined the Google Code-In project last year meaning that updates and bugfixes have been coming in thicker and faster than ever before, and I'm now more than happy with the results I can achieve at sensible bitrates.
What do I mean by 'high quality'? Well, there are many GUIs/front-ends available for converting video to H.264 (HandBrake and SUPER to name two of the most popular), but most use outdated versions of x264 and don't give full access to all of its advanced features, meaning that only suboptimal results can ever be achieved. This guide aims to show you how to achieve the best quality possible for a target size of 10MB per minute of source material and is tuned primarily for SD sources. Higher quality encodings are possible with x264, but not at only 1GB of storage space used per every 100 minutes of source material.
Why a custom preset? The standard presets don't even come close to exploiting the full potential of the x264 encoder. They merely provide 'safe' defaults primarily for older hardware players. The numerous tweaks made in this custom preset still stay within the constraints of an official 'High 3.1' profile, and the resultant files will play back on almost any modern H.264 compatible consumer device from an entry-level Android tablet upwards. They also play back very nicely indeed on a PC using the third-party MPC-HC or VLC media players.
How long will encoding take? Roughly 6 hours per hour of interlaced source material on a C2D CPU running at 3.3GHz. Expect considerably shorter encoding times with a fast i5 or i7 CPU. Progressive source material (ie, non-interlaced) will convert much more quickly.
Why so long? High quality, high compression ratio video conversion can't be rushed. For further details, please read the 'For Geeks' section at the bottom of this guide.
Can this method be used to convert commercial DVDs? Legally, no. MediaCoder won't handle encrypted files for starters, so you'd need to provide it with decrypted DVD source material. It won't always successfully stitch files together either, so the separate 1GB chunks usually found on commercial DVDs would also have to be stitched together before conversion. Both problems are easily circumvented with free software, but due to the illegality of the subject, I feel it would be unwise to discuss this in open forum.
Getting Started
What software will you need? You'll need the latest version of MediaCoder from HERE (32- or 64-bit to suit your OS), the latest 8bit-depth version of x264 from HERE (32- or 64-bit version to suit), and my custom preset from HERE (32-bit), or HERE (64-bit).
Should you go 32- or 64-bit? If you know that you have a 32-bit version of Windows, go for 32-bit. If you know that you have a 64-bit version of Windows, either will be fine although the 64-bit version may run slightly faster.
Installing
1/ Install MediaCoder first.
2/ Now copy your freshly downloaded 'x264.exe' into the MediaCoder 'codecs' folder accepting the option to overwrite the existing version. This folder is called 'codecs64' in the 64-bit version of MediaCoder.
3/ Now copy my custom preset file into the main MediaCoder folder.
Setting up
1/ Launch MediaCoder and tick the box on the webpage that opens telling it not to open the webpage again. You'll see what I mean the first time you run MediaCoder.
2/ Now select 'File', 'Load Preset' and navigate your way to the main MediaCoder folder and load my custom preset.
3/ Select an output folder in the top-right corner of the MediaCoder main window.
Encoding
1/ Select 'File', 'Add FIle' to load up your original video file.
2/ Click on the 'Start' icon button to start the encoding process.
That's it.
For Geeks
Why does this method take so long to encode then? Well, it's a combination of several factors which I'll list below for clarity...
1/ CPU encoding always takes longer than GPU encoding. This is partly because of the quick-and-dirty nature of x264 GPU encoding where the more advanced features of x264 simply aren't available for the GPU to use yet. This will hopefully change as time passes, but CPU encoding is currently superior by a fair margin in terms of encoding quality versus file size. As the primary aim of lossy encoding is to save space, GPU encoding makes less sense than CPU encoding at this time (assuming that you're in no hurry) as the resultant files have to be larger to maintain the same level of picture quality.
2/ Interlaced source material must be deinterlaced well before feeding to x264 in order to obtain maximum efficiency, and good deinterlacing is a very processor-intensive task. The motion-compensating deinterlacer used by this preset is amongst the 'hungriest' there are, but it's near-unbeatable in terms of results and produces visibly superior output compared to any other method available in MediaCoder. The deinterlacer will automatically detect whether or not the source is interlaced and disable itself when not required, so there's no tweaking necessary to make the most of either interlaced or progressive source material with this preset.
3/ The encoding method used is 2-pass encoding which takes considerably longer than single-pass encoding. This allows the encoder to generate a look-up table during the first pass to decide how best to distribute the bitrate on the second pass where the actual encoding takes place. More bitrate will be assigned to complex scenes and less bitrate to simple scenes, thus quality remains almost perfectly constant throughout the encoding regardless of scene complexity. Similar results can be achieved in a single pass with the encoder running in constant quality mode, but the final filesize would be almost entirely unpredictable. 2-pass encoding allows us to achieve both constant quality and a predictable output file size.
4/ Some of the advanced x264 settings used also add to the encoding time but increase picture quality noticably through improved compressibility, all giving a small but perceivable improvement whilst staying within the technical limits of a 'High 3.1' profile to maintain a good degree of playback device compatibility.
What does MediaCoder combined with this preset do that other encoding methods don't necessarily do by default? This preset aims for the highest level of compression possible within the bounds of a 'High 3.1' profile whilst not leading to absurdly long encoding times that would show little or no further improvement in perceived picture quality for the given target bitrate of 1.33Mbps. The tweaks are numerous, and the most influential ones are listed below...
1/ The source video, if interlaced, is precision motion-compensation deinterlaced to a very high standard. This can improve compressibility and reduce artifacting significantly, especially in high-motion scenes.
2/ Full-range correction is applied to the source to ensure that black is absolute black and white is absolute white. As long as your viewing device is correctly calibrated, you no longer need to worry about whether its full contrast range is being taken advantage of when viewing video generated with this preset.
3/ A very small amount of x264's internal noise reduction is applied to improve compression with little if any discernable loss of fine detail.
4/ x264's internal deblocking filter is fine-tuned to maintain slightly more fine detail than usual. This helps to preserve original film grain.
5/ CABAC entropy coding is enabled to improve compression by around 10-15%.
6/ All macroblock types are enabled to improve compression.
7/ The number of reference frames is increased to improve compression.
8/ The maximum number of B-frames is increased to improve compression, and the B-frames decision method is set to optimal.
9/ Weighted prediction for B and P frames is enabled for improved B and P frame temporal precision.
10/ All motion estimation and search settings are set to as high a precision as makes any visible difference.
11/ Psychovisual optimisations are more accurately tuned than standard for typical movie sources and are enabled during all mode decisions.
12/ Adaptive quantisation is tuned more accurately than standard for typical movie sources and is set to adapt on a frame-by-frame basis.
What about sound quality? The soundtrack is encoded with the Nero AAC encoder in HE-AACv1 at 64kbps. This may seem a little miserly to some, but it still maintains sufficient information to give the impression of a full 20kHz audio bandwidth and for fairly accurate surround sound reproduction if the original soundtrack contained Dolby ProLogic encoded 5.1 surround information. Don't knock it until you've heard it.
Enjoy!
Cheers, Slipstreem.
Why have I waited so long? Because I was never entirely satisfied with the results I achieved with earlier versions of x264 unless a relatively high bitrate 'brute force' approach was used. The x264 developers joined the Google Code-In project last year meaning that updates and bugfixes have been coming in thicker and faster than ever before, and I'm now more than happy with the results I can achieve at sensible bitrates.
What do I mean by 'high quality'? Well, there are many GUIs/front-ends available for converting video to H.264 (HandBrake and SUPER to name two of the most popular), but most use outdated versions of x264 and don't give full access to all of its advanced features, meaning that only suboptimal results can ever be achieved. This guide aims to show you how to achieve the best quality possible for a target size of 10MB per minute of source material and is tuned primarily for SD sources. Higher quality encodings are possible with x264, but not at only 1GB of storage space used per every 100 minutes of source material.
Why a custom preset? The standard presets don't even come close to exploiting the full potential of the x264 encoder. They merely provide 'safe' defaults primarily for older hardware players. The numerous tweaks made in this custom preset still stay within the constraints of an official 'High 3.1' profile, and the resultant files will play back on almost any modern H.264 compatible consumer device from an entry-level Android tablet upwards. They also play back very nicely indeed on a PC using the third-party MPC-HC or VLC media players.
How long will encoding take? Roughly 6 hours per hour of interlaced source material on a C2D CPU running at 3.3GHz. Expect considerably shorter encoding times with a fast i5 or i7 CPU. Progressive source material (ie, non-interlaced) will convert much more quickly.
Why so long? High quality, high compression ratio video conversion can't be rushed. For further details, please read the 'For Geeks' section at the bottom of this guide.
Can this method be used to convert commercial DVDs? Legally, no. MediaCoder won't handle encrypted files for starters, so you'd need to provide it with decrypted DVD source material. It won't always successfully stitch files together either, so the separate 1GB chunks usually found on commercial DVDs would also have to be stitched together before conversion. Both problems are easily circumvented with free software, but due to the illegality of the subject, I feel it would be unwise to discuss this in open forum.
Getting Started
What software will you need? You'll need the latest version of MediaCoder from HERE (32- or 64-bit to suit your OS), the latest 8bit-depth version of x264 from HERE (32- or 64-bit version to suit), and my custom preset from HERE (32-bit), or HERE (64-bit).
Should you go 32- or 64-bit? If you know that you have a 32-bit version of Windows, go for 32-bit. If you know that you have a 64-bit version of Windows, either will be fine although the 64-bit version may run slightly faster.
Installing
1/ Install MediaCoder first.
2/ Now copy your freshly downloaded 'x264.exe' into the MediaCoder 'codecs' folder accepting the option to overwrite the existing version. This folder is called 'codecs64' in the 64-bit version of MediaCoder.
3/ Now copy my custom preset file into the main MediaCoder folder.
Setting up
1/ Launch MediaCoder and tick the box on the webpage that opens telling it not to open the webpage again. You'll see what I mean the first time you run MediaCoder.
2/ Now select 'File', 'Load Preset' and navigate your way to the main MediaCoder folder and load my custom preset.
3/ Select an output folder in the top-right corner of the MediaCoder main window.
Encoding
1/ Select 'File', 'Add FIle' to load up your original video file.
2/ Click on the 'Start' icon button to start the encoding process.
That's it.
For Geeks
Why does this method take so long to encode then? Well, it's a combination of several factors which I'll list below for clarity...
1/ CPU encoding always takes longer than GPU encoding. This is partly because of the quick-and-dirty nature of x264 GPU encoding where the more advanced features of x264 simply aren't available for the GPU to use yet. This will hopefully change as time passes, but CPU encoding is currently superior by a fair margin in terms of encoding quality versus file size. As the primary aim of lossy encoding is to save space, GPU encoding makes less sense than CPU encoding at this time (assuming that you're in no hurry) as the resultant files have to be larger to maintain the same level of picture quality.
2/ Interlaced source material must be deinterlaced well before feeding to x264 in order to obtain maximum efficiency, and good deinterlacing is a very processor-intensive task. The motion-compensating deinterlacer used by this preset is amongst the 'hungriest' there are, but it's near-unbeatable in terms of results and produces visibly superior output compared to any other method available in MediaCoder. The deinterlacer will automatically detect whether or not the source is interlaced and disable itself when not required, so there's no tweaking necessary to make the most of either interlaced or progressive source material with this preset.
3/ The encoding method used is 2-pass encoding which takes considerably longer than single-pass encoding. This allows the encoder to generate a look-up table during the first pass to decide how best to distribute the bitrate on the second pass where the actual encoding takes place. More bitrate will be assigned to complex scenes and less bitrate to simple scenes, thus quality remains almost perfectly constant throughout the encoding regardless of scene complexity. Similar results can be achieved in a single pass with the encoder running in constant quality mode, but the final filesize would be almost entirely unpredictable. 2-pass encoding allows us to achieve both constant quality and a predictable output file size.
4/ Some of the advanced x264 settings used also add to the encoding time but increase picture quality noticably through improved compressibility, all giving a small but perceivable improvement whilst staying within the technical limits of a 'High 3.1' profile to maintain a good degree of playback device compatibility.
What does MediaCoder combined with this preset do that other encoding methods don't necessarily do by default? This preset aims for the highest level of compression possible within the bounds of a 'High 3.1' profile whilst not leading to absurdly long encoding times that would show little or no further improvement in perceived picture quality for the given target bitrate of 1.33Mbps. The tweaks are numerous, and the most influential ones are listed below...
1/ The source video, if interlaced, is precision motion-compensation deinterlaced to a very high standard. This can improve compressibility and reduce artifacting significantly, especially in high-motion scenes.
2/ Full-range correction is applied to the source to ensure that black is absolute black and white is absolute white. As long as your viewing device is correctly calibrated, you no longer need to worry about whether its full contrast range is being taken advantage of when viewing video generated with this preset.
3/ A very small amount of x264's internal noise reduction is applied to improve compression with little if any discernable loss of fine detail.
4/ x264's internal deblocking filter is fine-tuned to maintain slightly more fine detail than usual. This helps to preserve original film grain.
5/ CABAC entropy coding is enabled to improve compression by around 10-15%.
6/ All macroblock types are enabled to improve compression.
7/ The number of reference frames is increased to improve compression.
8/ The maximum number of B-frames is increased to improve compression, and the B-frames decision method is set to optimal.
9/ Weighted prediction for B and P frames is enabled for improved B and P frame temporal precision.
10/ All motion estimation and search settings are set to as high a precision as makes any visible difference.
11/ Psychovisual optimisations are more accurately tuned than standard for typical movie sources and are enabled during all mode decisions.
12/ Adaptive quantisation is tuned more accurately than standard for typical movie sources and is set to adapt on a frame-by-frame basis.
What about sound quality? The soundtrack is encoded with the Nero AAC encoder in HE-AACv1 at 64kbps. This may seem a little miserly to some, but it still maintains sufficient information to give the impression of a full 20kHz audio bandwidth and for fairly accurate surround sound reproduction if the original soundtrack contained Dolby ProLogic encoded 5.1 surround information. Don't knock it until you've heard it.
Enjoy!
Cheers, Slipstreem.
