CVE-2026-56208

A heap buffer overflow vulnerability was found in libaom, the reference AV1 codec implementation. A flaw in the AV1 encoder's Look-Ahead Processing (LAP) mode causes the first-pass stats ring buffer wrap-around guard to be bypassed when g_lag_in_frames is set to 1 or higher. This results in a 232-byte out-of-bounds write on every encoded frame after the second, corrupting adjacent heap objects. An attacker who can influence encoder configuration in a transcoding service or WebRTC session could exploit this to cause a denial of service (process crash) or potentially achieve code execution.

This vulnerability is rated as Important severity because a heap buffer overflow with attacker-influenced data can cause reliable denial of service and potentially lead to code execution, though the attacker has only indirect control over the written values (encoder-computed statistics). In Red Hat products, libaom ships bundled within Firefox and Thunderbird as a statically-linked dependency used for AV1 decoding and WebRTC encoding. The vulnerable code path requires the encoder to be configured with g_lag_in_frames >= 1 (Look-Ahead Processing mode). In Firefox's WebRTC implementation, the encoder configuration is controlled by the browser itself and not exposed to remote peers, which significantly limits the attack surface compared to standalone transcoding services. RHEL-AI 3.4 and Hummingbird 1 ship standalone libaom (aom) packages at versions within the affected range. Applications on those platforms that use the libaom encoder API with LAP mode and accept untrusted configuration input are vulnerable.

The CVSS score(s) listed for this vulnerability do not reflect the associated product's status, and are included for informational purposes to better understand the severity of this vulnerability.

There is no complete mitigation for this vulnerability. The following measures can reduce risk:

1. If using libaom as a standalone encoder library, avoid setting g_lag_in_frames to values >= 1 when processing untrusted input, or validate all encoder configuration parameters before passing them to the libaom API.
2. For Firefox and Thunderbird, ensure browsers are updated to versions that include the patched libaom (v3.14.0 or later).
3. For standalone libaom deployments (RHEL-AI, Hummingbird), restrict access to the encoding service to trusted clients only.
4. Apply network-level access controls to limit who can submit video for encoding.