Video Encoding using NVIDIA CUDA

Recently, someone sought help with implementing video encoding using NVIDIA CUDA. I asked why they didn’t consider using the NVIDIA Video Codec SDK, which features high-performance, hardware-accelerated H.264/HEVC encoders (NVENC) and decoders. They complained that NVENC was too slow, “not even real-time.” This is actually not an uncommon situation where people or companies, struggling to adopt cutting-edge technology, decide to take an alternative route without fully understanding the consequences.

In the early versions of NVIDIA’s Video Codec SDK, video decoding was handled by CUDA cores rather than dedicated hardware. This approach relied on the general-purpose processing power of CUDA cores for video decoding tasks, which proved to be less efficient compared to using specialized hardware. At that time, I was leading an engineering team at an Israeli startup specializing in personalized video content creation, and we were searching for ways to accelerate the decoding process. We decided to implement a complete H.264 video decoder from the specs using CUDA and PTX (the CUDA compiler’s assembly language). We even brought in a GPU optimization expert to extract the maximum performance from the hardware. Despite our efforts and some minor optimizations, the overall performance of our implementation was nowhere near that of the hardware-accelerated decoder that NVIDIA later introduced as part of their SDK.

So, if anyone is wondering whether it makes sense to write a video encoder with CUDA, the short answer is: No, it doesn’t. Here’s a more detailed explanation. NVENC operates on a dedicated encoder hardware chip located on the graphics card. This chip’s sole function is to encode frames, provided in any of the supported YUV formats, into a raw H.264 or H.265 bitstream. It doesn’t utilize CUDA cores at all, except in cases where the input frame is in RGBA format (which is possible with the latest versions of NVENC). In such cases, a CUDA kernel is used to convert the frame to the YUV color space—a step performed by the driver before passing the data to the encoder hardware.

The NVENC API is largely asynchronous and can run in its own thread without interfering with other GPU-related tasks in your application. If your frames are generated on the GPU, they can be fed directly into the encoder through low-overhead resource sharing between rendering and NVENC APIs, without the need for a round trip to system memory. This approach effectively keeps the GPU engaged throughout the entire encoding process.

A CUDA-implemented video encoder will be significantly slower than NVENC. Here are three key reasons why:

Serial Execution in Video Codec Algorithms: Some performance-critical components of video codec algorithms cannot be efficiently executed on a GPU because they require serial processing. This is inherent to the algorithm itself, particularly in areas like entropy coding and macro-block processing in H.264. Unless you’re planning to invent a new codec (good luck with that), this limitation is unavoidable.

Resource Sharing with Other GPU APIs: CUDA uses the same streaming multiprocessors (SMs) as other GPU-based APIs. This “unified architecture,” introduced with Fermi generation GPUs, allows the same resources (cores) to perform various computational tasks. However, if you’re running a rendering session with OpenGL or another API while also encoding with CUDA, you’ll likely be sharing GPU resources between these APIs. This sharing reduces the available SMs for each task, potentially leading to performance degradation. While having more SMs on your GPU can help with parallel computation, even the most powerful GPU has its limits.

CUDA Graphics Interoperability API Bottlenecks: Even if you manage to write such a codec, a significant performance bottleneck will be the CUDA Graphics Interoperability API. This API allows textures and buffers from rendering APIs like OpenGL, DirectX, or Vulkan to be shared with CUDA, but it’s notoriously slow. This is a major reason why high-performance rendering applications don’t use CUDA for general-purpose GPU (GPGPU) tasks; they use compute shaders, which operate within the same context and avoid the overhead of shared resource mapping. Moreover, you won’t be able to complete the entire encoding cycle in a single kernel invocation—this is both algorithmically and technically impossible. For example, with older architectures like Kepler, your GPU might run out of registers needed for execution, forcing you to split the workload into multiple kernel invocations. Even with newer architectures, overloading the GPU with encoder tasks will prevent it from efficiently handling other operations.

The above information is based on my personal experience. Another fact which is also based on my experience is this : if you use NVENC correctly, you can encode 1080p not only in real time (60fps), but much faster than that and in more than one concurrent encoding session. See NVENC performance statistics on NVIDIA website. They don’t lie. The API is hard, poorly documented, but if you take time to learn it and experiment, you will get encoding rates impossible with any other approach.