WebRTC & HEVC – how can you get these two to work together

Getting HEVC and WebRTC to work together is tricky and time consuming. Lets see what the advantages are and if this is worth your time or not.

Does HEVC & WebRTC make a perfect match, or a match at all???

WebRTC is open source, open standard, royalty free, …

HEVC is royalty bearing, made by committee, expensive

And yet… we do see areas where WebRTC and HEVC mix rather well. Here’s what I want to cover this time:

WebRTC and royalty free codecs

Digging here in my blog, you can find articles discussing the WebRTC codec wars dating as early as 2012.

Prior to WebRTC, most useful audio and video codecs were royalty bearing. Companies issued patents related to media compression and then got the techniques covered by their patents integrated into codec standards, usually, under the umbrella of a standardization organization.

The logic was simple: companies and research institutes need to make a profit out of their effort, otherwise, there would be no high quality codecs. That was before the internet as we know it…

Once websites such as YouTube appeared, and UGC (User Generated Content) became a thing, this started to shift:

• Browser vendors grumbled a bit about this, since browsers were given away freely. Why should they pay for licensing codec implementations?
• Content creators and distributors alike didn’t want to pay either – especially since these were consumers (UGC) and not Hollywood in general

The new business models broke in one way or another the notion of royalty bearing codecs. Or at least tried to break. There were solutions of sorts – smartphones had hardware encoders prepaid for, decoder licenses required no payments, etc.

But that didn’t fit something symmetric like WebRTC.

When WebRTC was introduced, the codec wars began – which codecs should be supported in WebRTC?

The early days leaned towards royalty free codecs – VP8 for video and Opus for voice. At some point, we ended up with H.264 as well…

How H.264 wiggled its way into WebRTC

H.264 is royalty bearing. But it still found its way into WebRTC that was due to Cisco in a large part – they decided to contribute their encoder implementation of H.264 and pay the royalties on it (they likely already paid up to the cap needed anyways). That opened a weird technical solution to be concocted to make room for H.264 and allow it in WebRTC:

• WebRTC spec would add H.264 as a mandatory to implement codec for browsers
• Browsers would use the Cisco OpenH264 implementation for the encoder, but won’t have it as part of their browser binary
• They would download it from Cisco’s CDN after installing the browser

Why? Because lawyers. Or something.

It worked for browsers. But not on mobile, where the solution was to use the hardware encoder on the device, that doesn’t always exist and doesn’t always work as advertised. And it left a gaping headache for native developers that wanted to use H.264. But who cared? Those who wanted to make a decision for WebRTC and move on – got it.

That made certain that at some point in the future, the H.264 royalty bearing crowd would come back asking for more. They’d be asking for HEVC.

HEVC, patents and big 💰

HEVC is a patents minefile, or at least were – I admit I haven’t been following up on this too closely for a few years now.

Here are two slides I have in my architecture course:

There are a gazillion patents related to HEVC (not that many, but 5 figures). They are owned by a lot of companies and get aggregated by multiple patent pools. Some of them are said to be trickling into VP9 and AV1, though for the time being, most of the market and vendors ignore that.

These patents make including HEVC in applications a pain – you need to figure out where to get the implementation of HEVC and who pays for its patents. With regard to WebRTC:

• Is this the browser vendors who need to pay?
• Maybe the chipset vendors?
• Or device manufacturers?
• What about the operating system itself?
• How about the application vendor?

Oh, and there’s no “easy” cap to reach as there is/were with H.264 when it was included in WebRTC and paid for by Cisco.

HEVC is expensive, with a lot of vendors waiting to be paid for their efforts.

HEVC hardware

Software codecs and royalty payments are tricky. Why? Because it opens up the can of worms above, about who is paying. Hardware codecs are different in nature – the one paying for them is either the hardware acceleration vendor or the device manufacturer.

This means that hardware acceleration of codecs has two huge benefits – not only one:

1. Less CPU use on the device
2. Someone already paid the royalties of the codec

This is likely why Apple decided to go all in with HEVC from iPhone 8 and on – it gave them an edge that Android phones couldn’t easily solve:

• iPhone is vertically integrated – chipset, device and operating system
• Android devices have the chipset vendor, the device manufacturer and Google. Who pays the bill on HEVC?

This gap for Android devices was a nice barrier for many years that kept Apple devices ahead. Apple could “easily” pay the HEVC royalties while Android vendors try to figure out how to get this done.

Today?

We have Intel and Apple hardware supporting HEVC. Other chipset vendors as well. Some Android devices. Not all of them. And many just do decoding but not encoding.

For the most part, the HEVC hardware support on devices is a swiss cheese with more holes than cheese in it. Which is why many focus on HEVC support in Apple devices only today (if at all).

Advantages of HEVC in WebRTC

When it comes to video codecs, there are different generations of codecs. In the context of WebRTC, this is what it looks like:

There are two axes to look at in the illustration above

1. From left to right, we move from one codec generation to another. Each one has better compression rates but at higher compute requirements
2. Then there’s bottom to top, moving from royalty bearing to royalty free

If we move from the VP8 and H.264 to the next generation of VP9 and HEVC, we’re improving on the media quality for the same bitrate. The challenge though is the complexity and performance associated with it.

To deal with the increased compute, a common solution is to use hardware acceleration. This doesn’t exist that much for VP9 but is more prevalent in HEVC. That’s especially true since ALL Apple devices have HEVC support in them – at least when using WebRTC in Safari.

The other reason for using HEVC is media processing outside of WebRTC. Streaming and broadcasting services have traditionally been using royalty bearing video codecs. They are slowly moving now from H.264 to HEVC. This shift means that a lot of media sources are going to have available in them either H.264 or HEVC as the video codec – a lot less common will be VP8 or VP9. This being the case, vendors would rather use HEVC than go for VP9 and deal with transcoding – their other alternative is going to stick to using H.264.

So, why use HEVC?

• It is better than VP8 and H264
• Existence of hardware acceleration for HEVC that is more common than VP9
• Things we want to connect to might have HEVC and not VP9
• Differentiation. Some users, customers, investors or others may assume you’re doing something unique and innovative

Limitations of HEVC in WebRTC

HEVC requires royalty payments in a minefield of organizations and companies.

Apple already committed itself fully to HEVC, but Google and the rest of the WebRTC industry haven’t.

Google will be supporting HEVC in Chrome for WebRTC only as a decoder and only if there’s hardware accelerator available – no software implementation. Google’s “official” stance on the matter can be found in the Chrome issues tracker.

So if you are going to support HEVC, this is where you’ll find it:

• Most Apple devices (see here)
• Chrome (and maybe Edge?) browsers on devices that have hardware acceleration for HEVC, but only for decoding. But not yet – it is work in progress at the moment
• Not on Firefox (though Mozilla haven’t gotten yet to adding AV1 to Firefox either)

Waiting for Godot AV1

Then there is AV1. A video codec years in the making. Royalty free. With a new non-profit industry consortium behind it, with all the who’s who:

The specification is ready. The software implementation already exists inside libwebrtc. Hardware acceleration is on its way. And compression results are better than HEVC. What’s not to like here?

This makes the challenge extra hard these days –

Should you invest and adopt HEVC, or start investing and adopting AV1 instead?

• HEVC has more hardware support today
• AV1 can run anywhere from a royalties standpoint
• HEVC isn’t available on many devices and device categories
• AV1 is too new and can’t seriously deal with high bitrates and video resolutions
• HEVC won’t be adopted by many devices even in the foreseeable future
• AV1 is likely to be supported everywhere in the future, but it is almost nowhere in the present

Adopt VP9? Wait for AV1?

Where can you fit HEVC and WebRTC?

Let’s see where there is room today to use HEVC. From here, you can figure out if it is worth the effort for your use case.

The Apple opportunity of WebRTC and HEVC

Why invest now in HEVC? Probably because HEVC is available on Apple devices. Mainly the iPhone. Likely for very specific and narrow use cases.

For a use case that needs to work there, there might be some reasoning behind using HEVC. It would work best there today with the hardware acceleration that Apple pampered us with for HEVC. It will be really hard or even impossible to achieve similar video quality in any other way on an iPhone today.

Doing this brings with it differentiation and uniqueness to your solution.

Deciding if this is worth it is a totally different story.

Intel (and other) HEVC hardware

Intel has worked on adding HEVC hardware acceleration to its chipsets. And while at it, they are pushing towards having HEVC implemented in WebRTC on Chrome itself. The reason behind this is a big unknown, or at least something that isn’t explained that much.

If I had to take a stab at it here, it would be the desire of Intel to work closely with Apple. Not sure why, it isn’t as if Intel chipsets are interesting for Apple anymore – they have been using their own chips for their devices for a few years now.

This might be due to some grandiose strategy, or just because a fiefdom (or a business unit or a team) within Intel needs to find things to do, and HEVC is both interesting and can be said to be important. And it is important, but is it important for WebRTC on Intel chipsets? That’s an open question.

Should you invest in HEVC for WebRTC?

No. Yes. Maybe. It depends.

When I told Philipp Hancke I am going to write about this topic, he said be sure to write that “it is a bit late to invest in HEVC in 2024”.

I think this is more nuanced than this.

It starts with the question how much energy and resources do you have and can you spend them on both HEVC and AV1. If you can’t then you need to choose only one of them or none of them.

Investing in HEVC means figuring out how the end result will differentiate your service enough or give it an advantage with certain types of users that would make your service irresistible (or usable).

For the most part, a lot of the WebRTC applications are going to ignore and skip HEVC support. This means there might be an opportunity to shine here by supporting it. Or it might be wasted effort. Depending how you look at these things.

Learn more about WebRTC (and everything about it)

Which codecs are available, which ones to use, how is that going to affect other parts of your application, how should you architect your solutions, can you keep up with the changes coming to WebRTC?

These and many other questions are being asked on a daily basis around the world by people who deal with WebRTC. I get these questions in many of my own meetings with people.

If you need assistance with answering them, then you may want to check out these services that I offer:

• WebRTC courses
• WebRTC insights