🎲 Which video codec to use in your WebRTC application? 🎲

Picking the right video codec for a WebRTC application is tricky. Should you use VP8? H.264? VP9? Go with AV1? What about HEVC?

WebRTC video codecs - a quick reminder

WebRTC was once easy. You had VP8, Opus and G.711. G.711 is striked through because I don’t want you to use it. There’s really no reason to. Later on, H.264 was added as a mandatory to implement video codec. And all was well in the world of WebRTC.

Google then decided to introduce VP9 in Chrome. As an optional codec. Mozilla added VP9 to Firefox as well. Microsoft? They got it for “free” when they switched Edge to Chromium. And Apple… well… Apple. VP9 should be in their Technology Preview for Safari, but mainly because of Google Standia which uses VP9 - surprising as this may sound.

Oh, and Apple decided to add HEVC as an optional codec of their own to WebRTC - just for good measures. And to confuse us all even further.

Then there’s AV1. The next gen bestest video codec. For the time being. At least once it gets added to Chrome (in version 90 that is). And used by developers.

Video codecs support across WebRTC browsers

The diagram below is taken from my recent workshop on trends in WebRTC for 2021. It shows the current state of video codec support in web browsers.

To sum things up:

• VP8 and H.264 are ubiquitous across browsers, and yes, there are some issues with both of them
• VP9 isn’t adopted as much after years of being available to developers, and is coming to Safari “soon”
• HEVC is Apple
• AV1 is new

Video codec performance in WebRTC

Last week, I sat down with Philipp Hancke for our WebRTC Fiddle of the Month. In this month’s fiddle, Philipp suggested we look at video codec performance, so he wrote a… fiddle.

You can watch the whole fiddle here: measuring video codecs performance

The results were quite interesting and sometimes surprising. What’s nice here is that you don’t need to take our word for it - you can take the code and use it yourself. Also make sure to use it in the scenario you have and not the simple one we’ve shared, as your mileage may vary.

VP8 or H.264 for your WebRTC application?

Today? You’re probably using VP8 or H.264 - or should use VP8 or H.264.

Is there any real difference between the two? No. Not really. They produce similar video quality for a given bitrate.

That said, there are some nuances between them:

• Google doesn’t really use H.264 in WebRTC. So VP8 is the more maintained video codec out of the two. For example, H.264 didn’t support simulcast in Chrome for many years (it does now)
• VP8 has virtually no hardware acceleration, so it will eat up more CPU in some cases
• H.264 has hardware acceleration. On Apple devices. Sometimes on PCs. Sometimes on Android. Sometimes though, you won’t have a H.264 implementation in WebRTC, because the hardware isn’t accessible and the software implementation isn’t there (royalties and stuff)
• Temporal scalability is only available in VP8. Not in H.264

Our own quick tests suggest that the H.264 decoder is better than the VP8 one - with or without hardware acceleration on H.264. Definitely something to think about.

Which one should you use? Throw a dice… 🎲 or two 🎲🎲

VP8/H.264 or VP9 in WebRTC?

Here’s a real question. Should you go for VP9? Last year I suggested it might be time to use VP9. Little has changed - no real adoption that I can see to it.

Except from Google, no one uses it.

In our tests, it was close to VP8 in its CPU use. That was quite surprising. It is probably why Google is using it in Google Meet.

The best thing about VP9? It also supports SVC (in an undocumented munging kind of a way).

The challenge? Apple. Doesn’t really have it yet. Should be getting there. Question is when.

When to use HEVC in WebRTC?

This one is simple enough to answer - never.

That said, if you have calls that take place only between Apple devices, then HEVC might be a good option.

Is the time right for AV1?

No. Maybe. Yes.

From our own testing, AV1 is considerably worse than all other codecs when it comes to performance. It takes twice or more of the CPU it takes to encode and decode any of the other video codecs we tried.

AV1 should offer better quality than the other codecs, so you may actually want to pay that extra CPU. As far as I can say, there are two reasons for using AV1 today:

1. To handle specific scenarios such as very low bitrate, where CPU isn’t the bottleneck but bandwidth is
2. When you are decoding only and the encoder is in the cloud - a place where you control the hardware. You’ll pay for it in compute costs though
3. It is rumored to be good at decoding thumbnails

Welcome to a multi codec WebRTC world

WebRTC started without many options. VP8 and H.264. That’s about it. Now? We’ve got 4-5 video codecs to choose from.

Most of us end up using VP8 just because. Some pick H.264, mainly because of performance considerations. The rest are mostly talked about but almost never used.

The newer video codecs are really promising - VP9, AV1 and even HEVC have real potential in a WebRTC application. The challenge though as that they come with some big challenges - mainly CPU and availability across browsers.

To use them, a new approach is needed. One where more than a single video codec is used by an application, at times within the exact same session.

Here are a few suggestions for you to explore:

• Support higher complexity codecs on 1:1 calls only, dynamically switching to other video codecs if and when a call grows beyond two participants
• Dynamically switch to a higher complexity codec on low bitrates
• Enable decoding in as many codecs as possible in parallel on a device, and then deciding what the encoder should send based on its CPU capabilities
• Using multiple video codecs in simulcast - for example using AV1 with very low bitrate and next to it use VP8 or VP9 at higher bitrates. Simulcast doesn’t support this (yet), but you could open two separate peer connections with different codecs and bitrates to achieve a similar outcome

Is it worth it? Maybe. You tell me if enhancing video quality in your application is important. Venturing into the multi video codec realm in WebRTC is about the 80% effort that yields the last 20% improvements. Go there once you’ve finished pursuing all other simpler optimizations.

If that's not headache enough, Google and Microsoft are throwing in Lyra and Satin, brand new AI-powered audio codecs. Things are getting more interesting (and complicated).

WebRTC trends in 2021

Last month I hosted a workshop about WebRTC trends in 2021.

I covered optimizations of a single video call, voice suppression, background blurring, E2EE and video coding aspects. The challenge of which video codec to choose was there as well.

The sessions have been recorded and are now available as an online course on my website. If you are interested, you can register for it.