A Brief Survey of Modern Technological Solutions for Server-Side Vector Graphics Rendering

In a world dominated by web apps and armies of full-stack developers, a question like, “How do I render vector graphics?” is a trivial one. Every browser these days provides a JavaScript-driven 2D shapes drawing API (known as the Canvas API). Mobile development frameworks either expose the same 2D drawing functionality found in browsers or offer their own convenient high-level APIs similar to the JavaScript Canvas 2D.

 

However, there are cases where a software product cannot be designed using web technologies. One such case is when there is a requirement to render graphics content in real-time on remote servers, process many incoming requests in parallel, and send the results back to users—all in real-time and with minimal latency. Furthermore, if there is a need to interleave 2D graphics with 3D at maximum performance, a customized solution is usually required. This solution could be an existing third-party library or a “roll your own” approach that matches the specific needs of the application. In this post, I will provide a brief overview of the most common 2D scalable vector graphics rendering technologies, along with their pros and cons, based on my personal experience working on projects that required the implementation of such functionality.

Let me present a theoretical example of a niche application that could require high-performance vector graphics rendering: You want to develop cloud-based software capable of streaming live video with a 2D graphics overlay. This is a common application used in sports, entertainment, advertising, and other industries. As the tech lead in charge of developing this product, you must decide on the infrastructure your company will rely on to achieve this goal and deliver the best possible solution.

Abusing the Web Browser

Ideally, you would want a turnkey solution that provides 2D graphics rendering with minimal fuss right out of the box. A web browser like Chromium or Firefox, which can run even on a headless server, might seem like an appealing choice. Full-stack developers might lean toward using Node.js with browser-less canvas modules, issuing drawing commands, and downloading the results from the server as a bitmap. However, I won’t delve into the Node.js path in this article, as I don’t have any experience using it. That said, it’s clear that running vector graphics rendering via Node.js should be more efficient than doing the same with a web browser.

Pros:

State-of-the-art Vector Graphics Rendering API: A browser provides a sophisticated API with a convenient interface that supports any type of path drawing defined by SVG. This can save you the headache of developing and maintaining your own solution.

Cons:

• Limited Customization: There is little ability to tailor the solution to specific needs, and unique applications often come with specific use cases. For instance, if vector path rendering is part of a complex composition involving multiple graphics layers from various sources (video, images, 3D models), the implementation becomes challenging. It becomes even more difficult if there is a need for interaction between different layers, such as dynamic z-ordering, intersection, pixel-perfect collision detection, 2D/3D picking, etc. A browser returns a bitmap with all the shape data baked into it, making pixel-perfect hit-tests on 2D shapes nearly impossible. Animated shapes would require per-frame re-rendering through such an API, and then re-submitting those to your application to update the specific graphics element. If your app uses hundreds of highly dynamic animated shapes that interact with other graphics objects in a layered manner, including alpha blending, you’ll need to re-render any vector shape element that overlays another shape whose graphics state has changed. This complexity only increases when dealing with layered rendering that involves 2D graphics manipulated in a specific order.
• Scalability Issues: Consider how much memory Google Chrome consumes on your system with each new tab opened. While using Node.js on the server (which likely uses fewer hardware resources when calling different web APIs compared to running a headless browser) with dozens of cores and a ton of RAM might mitigate this to some extent, it’s not an ideal solution. This approach is inefficient for scalability and could lead to significant financial costs as you find yourself running more instances on AWS to manage the load.

In other words, if 2D graphics are used as a single monolithic overlay bitmap, this solution is acceptable—but forget about high performance.

Skia Library

Let’s dig deeper—beyond the user API level in a web browser like Google Chrome—and explore where the real action happens. We all know that the Canvas API is part of the JavaScript interface used by web developers to program applications running in the browser sandbox. It might not come as a surprise to many that web browsers today are still written in C/C++. What mediates between the JavaScript user interface and the native “back-end” of the browser is a JavaScript engine, such as Chrome’s V8.

While this layer isn’t the focus of our discussion, it’s worth noting that there is some overhead involved when using APIs like 2D canvas or WebGL. This is because these APIs require calls to be sent to native modules that are not part of the JavaScript VM. Additionally, although JavaScript engines have advanced significantly in performance over the last decade, they are still relatively slow compared to software compiled to machine code. This performance gap is particularly noticeable when executing programs that require access to 2D or 3D graphics rendering in browsers. This is one of the reasons why projects like WebAssembly and WebGPU have been introduced—to allow much faster communication with the graphics rendering library.

Rasterizing vector graphics, or any other type of geometry, remains a computationally intensive task even today. While it might be possible to implement a software rasterizer for vector paths in pure JavaScript, achieving 16ms per frame for anything more than a few simple primitives is unlikely. To handle this demanding task, Google Chrome and Mozilla Firefox use a C++ library called Skia.

Skia is a cross-platform 2D graphics rendering library originally developed and maintained by Google. It is now used by almost all major web browsers and mobile devices (with the exception of Apple and Microsoft, as far as I know) to render vector and raster graphics content. When you call JavaScript canvas drawing routines in Chrome or Firefox, the Skia library is responsible for performing the actual rendering work behind the scenes. Skia also takes advantage of hardware-accelerated low-level graphics APIs to speed up graphics processing, making it an attractive tool for high-performance rendering. Additionally, Skia project has some of its functionality exposed to JavaScript via compact WASM libs called ‘Library Modules’, one of which is called CanvasKit that allows GPU accelerated vector graphics drawing in the browser.

Skia is a fast and complex piece of software. It includes GPU-accelerated rendering support through heavily abstracted, low-level graphics APIs like OpenGL, D3D (via ANGLE), and Vulkan. From a quick inspection of the GPU module of the library, I couldn’t definitively determine whether the Vulkan backend supports path rendering—probably it doesn’t, as I would expect to see it reflected in the source code (Skia developers, please feel free to correct me if I’m wrong). Skia’s configuration code checks for the existence of GL_CHROMIUM_path_rendering or GL_NV_path_rendering extensions.

The first extension likely (again, Skia experts, please correct me if I’m wrong) uses the ANGLE backend, which provides WebGL and GLES interfaces that translate calls to D3D on Windows. DirectX offers a Direct2D library with its own implementation of GPU-accelerated path rendering. The second extension, NV_PATH, is an NVIDIA GPU-specific OpenGL extension that adds the ability to render vector paths using the OpenGL API. If you run Skia on an NVIDIA GPU, it’s likely that Skia will attempt to use this backend to rasterize 2D vector shapes.

Additionally, Skia provides its own GPU-accelerated path rasterizer. Based on my reading of the code in GrTriangulator and other Gr* files located in Skia’s GPU directory, this seems to be the case (though please correct me if I’m mistaken). Finally, Skia includes a complete software rasterizer implementation as a fallback for systems lacking proper GPU hardware.

Skia is written in a typical, Google-style, overengineered OOP manner. While its code may not be considered neat or minimalistic, it is clear that this design approach was intended to provide maximum flexibility for building higher-level API bindings on top of it. Remember, we are talking about middleware. As I mentioned earlier, the library heavily abstracts all the low-level backends behind a relatively simple-to-use drawing API, which is very similar to JavaScript’s Canvas interface. For instance, here is an example of how to integrate Skia with an external OpenGL rendering context.

One particularly advantageous aspect of Skia is that in GPU mode, the library does not create or manage the hardware rendering context. This design choice allows for easy integration of Skia into the host application.

Using Skia as a third-party library integrated into your application provides support for GPU-accelerated vector graphics rendering.

Pros:

• State-of-the-Art Vector Graphics Rendering API: Skia supports standards like SVG, which can save you the headache of developing and maintaining your own solution.
• Better Performance and Lower Resource Consumption: Compared to using a web browser as an interface, Skia offers much better overall performance and significantly less hardware resource consumption, as it operates independently of other applications like web browsers.
• Direct Integration: Skia can be integrated directly into the target application, providing seamless support for GPU-accelerated vector graphics rendering.

Cons:

• Complexity and Coupling: Decoupling unnecessary functionality from Skia can be challenging. Skia introduces its own types even for simple math data structures like SkPoint, SkRect, and SkScalar. It also manages dynamic memory allocation using memory pools with its own data structures, as well as its own types for reference counting (e.g., smart pointers). If you plan to extract the GPU path rendering functionality from the codebase, it will require significant effort. You’ll need to replace or include all the plain old data (POD) types, memory management routines, refactor almost every function signature, and possibly more. Additionally, functional modifications that your project may need won’t be easy, as the library is large and its more complex algorithmic areas lack documentation.
• Artifacts in Software Mode: Skia may produce noticeable artifacts for complex paths when running in software mode.

In the above illustration, the author (who is also the person behind NVIDIA’s NV_PATH extension) compares the results of different vector path rasterizers. As shown, Skia’s CPU and GPU (without NV_PATH rendering) implementations receive a grade of C, which is relatively low due to the noticeable artifacts produced.

In contrast, the NV_PATH OpenGL extension achieves a grade of A+ (perfect score), highlighting its superior rendering quality.

Such a level of visual artifacts is hard to ignore in real-world applications. If you want to learn more about using NV_PATH rendering directly, skip to the bottom of the article.

Qt SDK

Another option is the Qt SDK. Known for its robustness in rendering cross-platform desktop and mobile UI systems, Qt SDK can also be used in offscreen mode and offers its own cross-platform CPU/GPU internal vector graphics rendering functionality. In fact, native UI programming is just a small part of what Qt SDK can do today. You can use it to build a full-fledged 3D rendering engine with minimal effort. The latest version of the SDK provides vector graphics rendering via the C++ Graphics API or the QML Shape API, both of which render shapes at very high quality using the GPU. However, there is a catch: the Qt SDK codebase is so large and complex that extracting relevant pieces from it is impractical. The standard approach is either to fully rely on the SDK—which offers nearly every conceivable module for various development tasks—or to create a hybrid architecture. In this hybrid approach, the core rendering logic is written outside the Qt SDK using “pure” C++ and then integrated into the Qt project as a static library or raw source files. From personal experience, interacting with the shape rendering API can be challenging since it is an integral part of Qt’s rendering pipeline. Therefore, the workflow would be similar to using the Skia library: you would call Qt’s drawing routines, bake the results into a bitmap, and render the bitmap in your renderer via texture mapping.

Pros:

• Comprehensive Turnkey Solution: One of the best existing turnkey C++ solutions for rich, cross-platform graphics application development, provided you are comfortable using Qt SDK as is.
• Easy Learning Curve: Qt is known for its user-friendly API, making it relatively easy to learn.
• Fast Development Pace: Qt’s extensive modules and features facilitate rapid development.

Cons:

• 2D Constrained Vector Shapes: Vector shapes are limited to 2D space. To render in 3D, you must render to a texture and then map it onto a 3D model.
• Complex Source Modification: Modifying the source code is possible but can be extremely challenging. If budget allows, you might consider hiring Qt developers to customize the source code to fit your specific needs.
• Memory Overhead: The SDK introduces noticeable memory overhead due to its size. While building the SDK from source and excluding unused modules can help, some libraries are essential, so it cannot be reduced to a bare minimum.
• Licensing Concerns: The licensing model might conflict with your company’s business model.

Getting “Close to the Metal”

It’s important to note that the solutions mentioned above (except for browser-based rendering) can be considered “close to the metal” (CTM) by definition. The code is compiled to machine byte code with no virtual machines involved. However, libraries like Qt and Skia present relatively complex architectures with deep call stacks and processing pipelines, introducing performance overhead that cannot be ignored. So, how can we reduce these inefficiencies and achieve even more efficient communication with the hardware?

NV_PATH Rendering

Low-level graphics hardware APIs like OpenGL, Direct3D, Vulkan, and Metal (the low-level graphics libraries market is indeed thriving) allow for relatively fast access to the GPU. It’s important to understand that these libraries do not issue function calls directly to your graphics card. Instead, they are implemented in C/C++ and installed on the system as .dll or .so libraries. To get the payload to the GPU, you issue a function call into the user-space library, which runs on the CPU. This call then proceeds to the kernel driver, which dispatches the hardware-specific command to the GPU. While this process is extremely fast, you would need to write your own GPU driver implementations to achieve even lower latency.

I’ve mentioned NVIDIA’s path rendering extension (NVPR) already several times in this article. Skia supports NVPR as one of its GPU backends. If you have the capability to use “raw” OpenGL in your application and if deployment is planned for environments like AWS or Google Cloud backed by NVIDIA graphics cards, it is worth considering NVPR. Implementing a high-level interface to leverage NVPR can provide state-of-the-art, high-performance vector shape rendering directly on the GPU. From my experience using NVPR for both vector shapes and text rendering, it’s unfortunate that this extension hasn’t been promoted to the core specification. This means it will likely remain vendor-specific, and as a result, you cannot run OpenGL applications using this extension on anything other than NVIDIA cards. However, as of today, few people use Intel or AMD GPUs for server-side rendering in the cloud. NVPR not only offers extremely accurate vector rasterization (as discussed in the Skia section above), outperforming industry-standard alternatives like Qt, Skia, and Cairo, but it also achieves high performance. This is largely due to optimizations in the OpenGL driver that enable faster batching and pipelining of path commands with minimal hardware state changes at runtime.

NV_PATH Rendering (NVPR) offers a comprehensive set of features out of the box, including:

• Text Rendering
• SVG and PostScript Path Formats
• Quadratic and Cubic Curves
• Advanced Shape Strokes: Different join, line, and cap styles
• Path Interpolations
• Accurate Intersection Tests
• Gradient and Bitmap Fills: Bitmap fills are achieved via pixel shaders
• Seamless 3D Integration

Adobe Illustrator uses NVPR for vector shape rasterization, indicating that it is a robust solution for professional vector graphics.

Disadvantages:

Based on my experience with this extension, the main disadvantages are:

• Vendor Specific: NVPR is unsupported on non-NVIDIA platforms, limiting its usability.
• Fixed Pipeline API: The API is based on fixed pipeline functionality, though it includes some Direct State Access (DSA) methods, which simplify API usage and OpenGL state management. NVIDIA provides optimizations for fixed pipeline calls with minimal overhead compared to the programmable pipeline. However, this can lead to inconsistencies in execution flow if the rendering core is designed with modern OpenGL practices (e.g., using a programmable pipeline with matrix uploads via uniforms and GPU buffers, while NVPR uses immediate mode functions like glPushMatrix and glLoadMatrix).
• Fragment Shader Access Only: NVPR only provides access to the fragment shader stage, with no access to geometry buffers generated by NVPR. While this might not be a significant issue, it could be a limitation in some cases. Fragment shader attributes like UV coordinates are accessible only via built-in GLSL variables (similar to pre-GLSL 3.0).
• Anti-Aliasing Limitations: Multi-Sample Anti-Aliasing (MSAA) or other screen-space anti-aliasing methods must be used, as NVPR only smooths curvatures. Straight lines are aliased by default due to the stencil-then-cover path rendering technique.
• GPU Debugging Challenges: If you are accustomed to GPU debuggers like RenderDoc or NVIDIA Nsight, you might find debugging NVPR functions challenging. Last I checked, Nsight did not support NVPR function calls for debugging (as of the latest Nsight Graphics 2021.1.0 release). This lack of support for vendor-specific extensions is surprising, given that NVPR is an NVIDIA extension.

Direct2D

I am skipping Microsoft’s rendering library in this article, as I have never used it. Additionally, it is Windows OS-specific, which makes it less appealing for server-side rendering, which is predominantly based on Linux operating systems.

“Roll Your Own” Solution

Creating a GPU-accelerated vector graphics rendering library from scratch is a viable option. This approach requires at least a minimal understanding of computational geometry and linear algebra. For those proficient in these areas, it may take several months to develop a robust solution.

Pros:

• Cross-Platform: You can design your library to be compatible with multiple operating systems.
• In-House Development: Complete control over the implementation details and maintenance, tailored to your specific needs.
• Seamless Integration with Modern APIs: It can fit into modern low-level APIs like Vulkan without issues.

Cons:

• Development and Maintenance Time: Building and maintaining a custom solution is time-consuming.
• Complex Feature Implementation: Adding advanced features will require extensive research and multiple development iterations.
• Need for Expert Knowledge: Continuous involvement of experts in computational geometry and graphics programming is necessary.
• Performance Limitations: It is unlikely to achieve the same level of performance as specialized solutions like NVPR.

If you choose to pursue this route, I highly recommend reading “Resolution Independent Curve Rendering using Programmable Graphics Hardware” by Loop & Blinn. This paper serves as a foundation for many GPU-accelerated implementations today.

NVPR is based on the paper “Resolution Independent Curve Rendering using Programmable Graphics Hardware” by Loop and Blinn, with further enhancements like real-time triangulation made possible by stencil-and-cover techniques. While other techniques such as scanline rasterization are available, they may offer slower performance. For those willing to invest in research, developing a custom technique could be a viable option. For instance, Eric Lengyel, a noted computer scientist and mathematician, has created and patented some of the best GPU-accelerated vector text rendering solutions available. Investing in such state-of-the-art solutions can save both time and money, providing immediate access to cutting-edge technology.

Headless Game Engines and Video Editing Software

This section addresses unconventional solutions, such as running game engines or video editing software on the cloud. Although it may sound unusual, cloud-based game streaming and media companies often rely on such setups due to a lack of proprietary rendering solutions. While I do not recommend running Adobe After Effects on a server, game engines like Epic’s Unreal Engine 4 (UE4) can be considered for headless rendering servers.

UE4 can be configured as a headless rendering server with minimal effort but does not support vector shapes rendering by default. You would need to purchase or develop a plugin to meet your needs. At the time of writing, UE4 can run in headless mode on Linux with Vulkan as the rendering backend. UE4’s source code (C++) is fully exposed, allowing for extensive modifications. However, integrating low-level path rendering functionality like NVPR can be challenging, particularly with UE4’s shift from OpenGL to Vulkan, which complicates integration.

OpenGL-to-Vulkan interoperability extensions might facilitate rendering NVPR in an OpenGL context and then copying it to a Vulkan surface. This integration involves significant hacking into UE4’s Rendering Hardware Interface (RHI), which is complex and extensive.

In terms of resource consumption, UE4 can be a major bottleneck if you need to run multiple instances on the same server. A minimal UE4 application consumes around 1GB of RAM and utilizes nearly 100% of the GPU. More demanding applications can easily exceed 12GB of RAM. This high resource usage will increase cloud provider costs significantly, requiring more CPU and GPU instances to scale effectively. Additionally, consider the licensing model: some engines are free to use but may require royalties upon reaching certain revenue thresholds, while others are subscription-based or closed source. Open source options are generally preferable from an engineering perspective, as they allow for codebase maintenance and avoid bugs introduced in proprietary engine updates.

OpenVG

I omitted OpenVG intentionally. The Khronos Group’s OpenVG standard for GPU-accelerated scalable vector graphics, designed over a decade ago, did not gain widespread adoption across major hardware platforms. Intended as the vector graphics counterpart to OpenGL, it failed to become mainstream. Reasons for this may include limited demand for GPU 2D shape rendering in the gaming industry. The API is considered outdated (resembling OpenGL 1.1’s immediate mode) and has limited commercial and open-source implementations, mostly targeting embedded devices. My experience with these implementations is limited, so their robustness and performance remain uncertain.