Video Codecs and Encoding: Everything You Should Know (Update)

Without video codecs, “Netflix and chill” wouldn’t have ever been coined. These two-part compression tools allow distributors to condense a video file for delivery across the internet via a process called video encoding. Codecs are the reason we can so easily conduct business meetings over Zoom and binge TV shows on our phones — even with limited bandwidth.

Thanks to codecs, Netflix manages to stream more than 404,000 hours of content every minute. And in order to get those streams to the various devices that end-users are tuning in from, Netflix must use both new and tried-and-true codecs.

What does video encoding involve, and how do video codecs work? We’ll dig deeper below and cover our list of the best video codecs for streaming.

Table of Contents

• What Is Video Encoding?
• What Is a Codec?
• What Is a Container Format?
• Video Codecs vs. Containers: What’s the Difference?
• Best Video Codecs for Streaming
  • H.264/AVC
  • VP9
  • H.265/HEVC
  • AV1
  • H.266/VVC
• Encoding Best Practices
• Encoding vs. Transcoding
• Multi-Codec Delivery

What Is Video Encoding?

Video encoding refers to the process of converting raw video into a digital format that’s compatible with many devices. When it comes to streaming, videos are often compressed from gigabytes of data down to megabytes of data. Video encoding is essential to live streaming, helping to ensure quick delivery and playback.

Encoding can occur within a browser or mobile app, on an IP camera, using software, or via a stand-alone appliance. We offer the Wowza Clearcaster live encoding solution, which enables production-quality encoding with unmatched cloud control. Popular software options include Vmix, Wirecast, and the free-to-use OBS Studio.

In order to compress the raw video into a more manageable size, encoders use video and audio codecs, which apply algorithms to shrink the bulky video for delivery. To put it more simply: encoding describes the process of compression, whereas codecs describe the means for doing so.

What Is a Codec?

Content distributors use a video compression technology called a codec to shrink a video into a streamable size. Codecs allow us to tightly compress bulky streams down for delivery and storage.

Literally ‘coder-decoder’ or ‘compressor-decompressor,’ codecs apply algorithms to the video and create a facsimile of it. When it comes to streaming, codecs employ lossy compression by discarding unnecessary data. The video is shrunk down for storage and transmission, and later decompressed for viewing.

Streaming requires the use of both audio and video codecs. H.264, also known as AVC (Advanced Video Coding), is the most common video codec; AAC (Advanced Audio Coding) is the most common audio codec.

But wait — why does H.264 also go by AVC? And how are we to sort through so many confusing acronyms? Streaming codec expert Jan Ozer explains it:

“H.264/AVC and H.265/HEVC have two names because each codec was standardized by both MPEG and the International Telecommunication Union (ITU). Versatile Video Coding, or VVC, is also H.266 for the same reason.”

For the sake of clarity, I’ll be including both names at first mention of each codec within the subsections of this article.

What Is a Video Container Format?

Once compressed, the components of a stream are packaged into a wrapper or file format. These files contain the audio codec, video codec, closed captioning, and any associated metadata. Common containers include .mp4, .mov, .ts, and .wmv.

Containers can often input multiple types of codecs. That said, not all playback platforms accept all containers and codecs. That’s why multi-format encoding is crucial when streaming to a wide range of devices.

For example: a .mov file and a .wmv file might have the same data and codecs inside. But the .mov file would be used for playback on Macbook’s QuickTime player, while the .wmv file would be used for playback on a PC’s Windows Media Player.

Video Codecs vs. Containers: What’s the Difference?

A codec acts upon the video, both at the source to compress it and before playback to decompress it. This is done through lossy compression, during which any unnecessary data is discarded.

Lossy compression is a lot like Wonkavision in Charlie and the Chocolate Factory. It makes a large collection of data smaller for transport to your screen:

On the other hand, a video container format stores the video codec, audio codec, and metadata such as subtitles or preview images. The container holds all the components together and determines which programs can accept the stream.

Best Video Codecs for Streaming

Delivering video over the internet to a variety of devices starts with encoding across a variety of codecs. Next-generation codecs improve encoding efficiency and quality, while legacy codecs enable playback on outdated machines.

Take it from the largest distributor of video streaming around: Netflix.

“Netflix says it utilizes a deep toolbox of codecs, which can be called upon to stream compatible formats to display devices. Although Netflix continually adds new and improved codecs, it has never abandoned one — it continues to support the VC1 codec it started with in the first Netflix streaming device, a 10-year-old LG Blu-ray player.”

Our video codec list below includes both the old and the new. While industry leaders continue to refine and develop the latest compression tools, they also employ older codecs like H.264/AVC for delivery to legacy devices.

Let’s look closer at the most common encoding technologies in 2021.

1. H.264/AVC
2. VP9
3. H.265/HEVC
4. AV1
5. H.266/VVC

H.264/AVC

The majority of encoding output today takes the form of H.264 files, also referred to as AVC (Advanced Video Coding). This widely supported codec was developed by the International Telecommunications Union and the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group — wow, what a mouthful.

H.264 also has significant penetration into markets outside of streaming, such as Blu-ray disks and cable broadcasting. It is often incorporated with the AAC audio codec and can be packaged into .mp4, .mov, .F4v, .3GP, and .ts containers.

H.264 plays on virtually any device, delivers quality video streams, and comes with the least concerns surrounding royalties. That’s not to say that it isn’t royalty-bearing, only that content publishers know what to expect — which isn’t always the case. Because of its widespread device support, H.264 remains the most commonly used option. In Bitmovin’s 2020 Video Developer Report, a whopping 91% of those surveyed indicated that they use it.

That said, although H.264 works great across all the primary markets for video consumption (browser, mobile, Smart TVs), it isn’t suitable for 4K video or high dynamic range (HDR) content.

Rather, as the speediest codec detailed in this blog, H.264 is better applied to low-latency streaming than 8K video delivery. Streaming Media’s Jan Ozer explains:

“H.264 excels in live origination and transcode because of encoding speed; all other codecs need hardware support for both functions. Understandably, H.264 has received the lion’s share of attention in low-latency applications, whether HTTP-based or WebRTC.”

H.264 is also the oldest video encoding format on this list. Many predicted it would be obsolete by now. But when you factor in its processing power and cost, H.264 is hard to beat. Plus, the number of devices that can encode and decode H.264 can’t be beat — including IP cameras, set-top boxes, mobile devices, and low-power devices.

TL;DR

H.264 is an efficient and widely adopted video compression technology used to contribute, distribute, and deliver streams. It lends especially well to low-latency workflows.

VP9

Google developed VP9 as a royalty-free, open-source alternative to H.265. The Google-owned YouTube platform and Chrome browser support VP9, as well every Android phone, Mozilla’s Firefox, Apple’s Safari, and all new iOS devices. This codec also makes an appearance in many WebRTC workflows, with more than 90% of Chrome-encoded WebRTC video using VP9 or its predecessor VP8.

VP9 was released in 2013, which puts it in the middle of the pack as far as age goes. Even so, it’s a better option than most for several reasons. For one, VP9 performs about the same as H.265/HEVC. This makes it well suited for 4K video, especially when publishing to YouTube.

Beyond that, VP9 ranks second only to H.264/AVC in terms of compatibility across browsers and devices. Samsung, Sony, LG, Roku, and many other household names support it. Plus, Google’s implementation of the codec in YouTube and Netflix’s use of it for some content will continue to drive this trend.

Think of VP9 as AV0, or an earlier version of AV1. Both are open-source, and both claim to be royalty-free (although there is some doubt surrounding this). For the time being, VP9 is also a better alternative to AV1, since more devices support it.

While we’ve listed VP9 as second on this list, it remains neck-in-neck with H.265/HEVC, which we’ll discuss next.

TL; DR

VP9 is a more advanced and higher-quality compression technology than H.264/AVC that enjoys greater compatibility than many of its alternatives and works well for 4K streaming.

H.265/HEVC

The ISO/IEV Moving Picture Experts Group developed H.265 as the successor to H.264. Also called HEVC (High Efficiency Video Coding), this codec aims to improve compression efficiency and support 8K resolution. It generates smaller files than H.264, thus decreasing the bandwidth required to view these streams. This makes it an ideal codec for high-resolution streaming.

That said, only about 10 percent of encoded files take the form of H.265. Uncertainties about royalties have stifled adoption. Specifically, content distributors are frustrated by the lacking transparency into what they’ll have to pay when using this codec.

The patent and royalty drama surrounding H.265 directly resulted in the development of the AV1 codec (which we’ll cover next) by the Alliance for Open Media, as well as its lacking compatibility for browser-based playback. Why? Industry leaders like Google, Microsoft, and Mozilla had no interest in adding support for the costly codec to Chrome, Edge, and Firefox. As a result, only ~18.08% of browsers accept H.265-encoded video.

One place where H.265 remains a better option than VP9 or H.264 is 4K and HDR video delivery to living room devices, as it’s almost universally supported on Smart TVs.

TL;DR

If you’re delivering premium OTT content to living room devices, H.265 is your best bet — but be prepared to pay royalties.

AV1

Frustrated about the royalties associated with H.265, Amazon, Netflix, Google, Microsoft, Cisco, and Mozilla formed the Alliance for Open Media. The goal? Create an open-source, royalty-free alternative called AV1.

While the codec has been finalized, this initiative to democratize high-quality video delivery and playback is still playing out.

According to Johnathan Rosenberg, CTO of the Collaboration Technology Group at Cisco, “The creation of an advanced, royalty-free video codec is paramount to the ongoing success of collaboration products and services. This is why Cisco joined AOMedia as a founding member, and why Cisco has invested in making AV1 both efficient and accessible to the internet community.”

AV1 touts itself as being 30 percent more efficient than H.265, but these claims still need to be verified by independent sources. It will also take some time before AV1 hardware decoding capabilities are integrated on a mass scale. Even Apple devices lack support for the codec, despite the fact that Apple joined the Alliance back in January of 2018.

In other words, the industry is still in flux when it comes to AV1.

“The one disadvantage at this point is that [AV1] is just new,” explains Anne Aaron, director of encoding technologies at Netflix. “H.264 is a really good codec that’s been developed over more than ten years — and AV1 is new, so there are still kinks in implementations.”

It will take some time before AV1 hardware decoding capabilities are integrated on a mass scale. While leaders at Netflix, Facebook, and more are planning to move to AV1, playback limitations can’t be ignored. The AV1 codec also requires lengthy encoding times, and time is money. For this reason, it’s only a cost-effecting solution when encoding videos for mass consumption.

In a recent deep-dive blog on the codec, Jan Ozer concluded, “even if you’re YouTube, if a video doesn’t project to top out over several million views, the increased encoding cost isn’t likely to be recouped in the form of bandwidth savings.”

TL; DR

AV1 is the latest and greatest open-source video encoding technology. Still, it’s far too soon to tell how adoption will play out — and lengthy encoding times currently translate to high encoding costs.

H.266/VVC

As the newest kid on the video compression block, the H.266/VVC (Versatile Video Coding) specification was only just finalized in 2020. While intended to usurp H.265 and H.264, it has the same royalty issues as its predecessors.

Back in 2008, Beamr’s chief technology officer Dror Gill explained, “It’s okay to pay royalties as long as you know how much you need to pay and when. With H.264, it was very clear how much you need to pay, there was one body collecting all the royalties, and this became the world’s most prominent video codec. The same can happen with VVC, if they get their act together before releasing the standard.”

And yet, the H.266/VVC royalty rate remains a mystery at this time. Unforeseen challenges on the licensing front are up in the air, and we’re also waiting to see how adoption of the codec pans out. 

To put it in a nutshell à la codec expert Jan Ozer:

“Overall, the individual H.266/VVC codec developers have made great strides in delivering the promised bandwidth savings, though final performance won’t be known until the royalty policies are set and we know which tools are in which profiles. Beyond that, given the diverse range of other factors in play, it’s impossible to know at this time whether VVC will ever reach critical mass.”

TL; DR

Still a nascent technology, H.264/VVC won’t even be a consideration for most content publishers until 2022 at the earliest, with too many unknowns surrounding the codec to make any predictions.

Encoding Best Practices

Encoding best practices go way beyond what codec you select. You’ll also want to consider frame rate, keyframe interval, and bitrate.

Luckily, a live stream can always be transcoded into another format once it reaches the server. This can be done using streaming software and your own servers, or in the cloud for professionally managed delivery.

Encoding vs. Transcoding

So what exactly is transcoding? Transcoding involves taking an encoded file and decoding it to alter it in some way. This can take the form of re-encoding the data into a more common codec, transizing the video into a lower resolution, transrating the file into a different bitrate, or transmuxing it into a more scalable protocol.

Once the process is complete, the media server recompresses the manipulated file for delivery. 

Transcoding gives you the freedom to change an H.264-encoded video into a VP9- or H.265-encoded video. That way, your content would be optimized for 4K streaming to end users — while also benefiting from speedy video contribution.   

Think of encoding and transcoding as the steps involved when packing for a vacation. Initially, you compress (encode) your clothing into a bag for easy transportation to wherever you’re headed. Once you arrive at your destination, you unpack, discard items that you no longer need, add trinkets gathered during your travels, and then repack your bag for the next step of your journey. This is transcoding.

Multi-Codec Video Delivery

There you have it. Video codecs are what allow us to take the boundless world, capture a slice of it through the lens of our camera, and compress it down for delivery over the internet.

Because proprietary codecs and video containers exist, it’s essential to deliver multiple different versions of your live streams to viewers.

Luckily, we offer the Wowza Streaming Engine software for multi-codec video delivery using your own servers — whether they’re on premises or in a third-party cloud platform. For those who want to get up and running quickly without any hassles, the Wowza Streaming Cloud service might be a better fit.

That way, you can convert streams as needed by transcoding the data into new codecs or transmuxing into different video encoding formats.

About Traci Ruether

As a Colorado-based B2B tech writer, Traci Ruether serves as Wowza's content marketing manager. Her background is in streaming and network infrastructure. Aside from writing, Traci enjoys cooking, gardening, and spending quality time with her kith and kin. Follow her… View more