Online video has always been very hot. While embedding Instagram and Youtube videos on a page is easy, there are more and more needs—such as many e-commerce scenarios—requiring custom video delivery methods.
When setting up video processing and transmission pipelines, the first thing to consider is the video format to be served. User experience, support (browser and system), compression efficiency, or encoding speed may be relevant to this choice.
Based on my web business media optimization experience , I will try to highlight the main aspects that need to be considered. If you are looking for simple transcoding and optimization options for using ffmpeg, you can also check out this article .
Container and codec
It is very important to be aware of the difference between containers and coding standards compared to the usual image formats . The file extension can only tell us which container it belongs to, not which codec to use. The coding standard that is followed determines whether the browser or system supports it.
For example, although web video formats are typically encoded using the mp4 container and the H264 standard, not every mp4 file is generally supported because it may use a different standard encoding, such as H265.
It even becomes more complex in terms of adaptive bit rate (ABR), which is the best way to respond to users’ network and device capabilities.
Let’s take a look at the main combinations of containers, coding and delivery standards, and their differences in support, compression efficiency, coding speed, and user experience.
The king of the video format uses an mp4 container with H264/AVC encoding. Sometimes you will also see it in the m4v container (the default format in Handbrake), an mp4 derivative developed by Apple for DRM-protected H264 video.
This format is supported by every browser and system — as well as native applications in iOS and Android. This is a safe choice to avoid compatibility issues.
In addition, hardware acceleration for H264 is supported on almost all desktops and mobile devices. The codec is very fast. ,
All in all, coding and using this format is very simple. Just like an image, you just need to insert a video link in HTML5 and you can use it in any browser.
A delay of approximately 2000 kbps and more than a few seconds may affect visual quality. Pause and re-buffering may occur when viewed over a mobile network or during peak hours. If a scheme that reduces image quality is used, artifacts such as blur, flying mosquitoes or blocks will be produced.
This is a powerful video format that uses the same container and is encoded with H265 HEVC, which results in higher compression efficiency ( approximately 50% reduction in volume ) and much less than blurring. The main problem with this format is that support is limited to Apple devices, including high royalties in their prices. It is only available to Safari and iOS apps. If you have a lot of iPhone or Mac users, you can use it as a backup for H264. Their experience will be better.
Even with the use of hardware acceleration (almost available only on Apple devices), the higher complexity of this format means that the coding speed is significantly slower, so generating the delivery file requires more computation and time.
This is a free, open source video format from Google. It uses the webm container instead of mp4, which is basically a mkv container, but sets the encoding standard to VP8 or VP9. Using H265 can also bring similar benefits , perhaps less efficient but still much more than H264. Again, it allows for a reduction in size, with artifacts other than blurring being much smaller. The encoding speed is similar to H265, which is very slow.
Note that although the previous version (VP8) also has the same support, we don’t recommend it at all because it doesn’t bring any benefits to H264 that is already widely supported. Webm can only be used with VP9 encoding.
Of course, support for webm is limited to the world of Google. This means that only Chrome and Android will support it.
The first stable version of the standard was released in March 2018, which contains mappings for MP4 and MKV containers. Compared to the H265, it offers similar or slightly higher compression efficiency gains, while licensing is free. Compared to the H265, the final implementation also increases the decoding speed , and AV1 is a compelling alternative to web video transmission.
The Open Media Alliance, which is involved in creating this format, promises to provide broad support in the near future.
Participating in AV1’s Open Media Alliance partners
However, the currently available implementation should still be experimental, and the bottleneck is still the encoding speed. The lack of hardware acceleration is clearly a problem and it is expected that there will be a first solution by the end of this year.
The committee responsible for H264 AVC and H265 HEVC is rapidly tracking the new standards and is expected to be released in 2020. Preliminary testing of the currently considered methods has shown a significant increase in performance compared to H265 and AV1. I included it as a futuristic possibility here, just to show that the competition for video coding seems far from over.
Adaptive bit rate (ABR)
This is a very interesting alternative to the progressive format. It is built on top of HTTP-based media streaming communication protocols. This method provides the video as a master playlist. Playlists offer options with different resolutions and bitrates to accommodate different viewport sizes, network bandwidth, and devices.
In addition, the video is divided into segments or blocks so that the client can jump from one quality level to another. It adapts to the user’s current conditions, ie network speed, and also to the viewport size – such as switching to full screen.
ABR brings tremendous advantages to optimizing the user experience of mobile devices, avoiding pauses or rebuffers under mobile networks. If you are looking for truly responsive behavior, this is obviously the approach you should take. It has two main standards: HLS and MPEG-DASH.
Although it is widely believed that ABR only makes sense for very long videos, in my experience, quite a few short clips can benefit from this approach in many cases.
Developed by Apple, this ABR protocol relies on different renderings that are split in mp4 format. Originally using H264, H265 is now also supported. But as a compromise, I recommend adhering to H264 encoding for HLS because it enables better compatibility in a variety of client cases.
One focus of this standard is the support of recent Apple devices. For clients other than Safari or native iOS apps, you need a viewer. But this is not a big problem because there are good open source options like videojs. Or you need to put some effort into customizing it and using it for your own front-end programs. It also offers great transcoding and delivery services to make it easy for you to do all of this.
Since each play should be encoded at a constant bit rate, I recommend using HLS with per-title encoding. That is, the bit rate of the playback is selected based on the content of the video.
This is a codec-independent protocol for ABR, so in addition to H264 and H265, it can be coded with VP9, and even new alternatives such as AV1 can be used. The downside is that it’s relatively young, which means less support than HLS. That’s why we don’t recommend it to most web businesses.
to sum up
The advantages of H264 AVC compression over the years are clear, and new approaches are adding momentum to web video. Competition in display size and resolution has facilitated the development of new formats that offer more content at the same bandwidth.
VP9 in webm has a significant improvement in compression efficiency (about 30%), no copyright issues, and is supported by Google solutions (Chrome, Android). Furthermore, compared to H264, H265/HEVC achieves considerable subjective quality with only half the bit rate. Since they are not universally supported, H264 is still needed, at least as a backup.
Adaptive bitrate is a compelling alternative that delivers an unparalleled user experience. In this regard, HLS is widely supported with the help of open source viewers. It may be the best choice for medium-sized networks. Due to open source initiatives such as videojs and the ability to offer third-party services at competitive prices, the complexity of the viewer’s needs has been significantly reduced. If you use the last technical route, be sure to ask for per-title encoding.