API Reference#

All renderer types live in the nvblox::renderer namespace. The primary header is nvblox/renderer/renderer.h.

Note

This page is auto-generated from source-code doc comments via Doxygen + Breathe.

Configuration #

struct RendererConfig#

Configuration for renderer initialization. Pass to NvbloxRenderer::init() to configure the renderer.

Public Members

uint32_t width = 1920#: Render target width in pixels.

uint32_t height = 1080#: Render target height in pixels.

std::string title = "nvblox renderer"#: Window title (ignored in headless mode).

bool headless = false#: If true, create headless renderer (no window).

enum class nvblox::renderer::RenderMode#

Render mode for the viewer. Determines which visualizer is used during render() and initVisualizer().

Values:

enumerator kImage#: Render depth and/or color images as 2D quads.

enumerator kPointCloud#: Render 3D colored point cloud.

enumerator kMesh#: Render 3D colored mesh.

NvbloxRenderer #

class NvbloxRenderer#

Main Vulkan renderer class for visualizing nvblox data.

Manages Vulkan context, render targets, and visualizers. Supports both windowed and headless rendering modes.

Public Functions

bool init(const RendererConfig &config)#

Initialize the renderer with a configuration.

Parameters:: config – Renderer configuration (window or headless mode).
Returns:: True if initialization succeeded.

bool initWithWindow( uint32_t width, uint32_t height, const std::string &title )#

Initialize the renderer with a window.

Parameters:

width – Window width.
height – Window height.
title – Window title.

Returns:

True if initialization succeeded.

bool initHeadless(uint32_t width, uint32_t height)#

Initialize headless renderer (no window). Useful for offscreen rendering, testing, or server-side use.

Parameters:

width – Render target width.
height – Render target height.

Returns:

True if initialization succeeded.

void destroy()#: Destroy renderer resources.

bool initVisualizer(RenderMode mode)#

Initialize a visualizer for the given render mode.

Parameters:: mode – Which visualizer to initialize.
Returns:: True if initialization succeeded.

bool updateDepth( const DepthImage &depth_image, const CudaStream &stream )#

Update depth image for RGBD visualization.

Note

Data is copied asynchronously. Call stream.synchronize() before render() to ensure data transfer is complete.

Parameters:

depth_image – Device-resident depth image.
stream – CUDA stream.

Returns:

True if update was initiated, false if visualizer not initialized.

bool updateColor( const ColorImage &color_image, const CudaStream &stream )#

Update color image for RGBD visualization.

Note

Data is copied asynchronously. Call stream.synchronize() before render() to ensure data transfer is complete.

Parameters:

color_image – Device-resident color image.
stream – CUDA stream.

Returns:

True if update was initiated, false if visualizer not initialized.

bool updatePointCloud( const DepthImage &depth_image, const ColorImage &color_image, const Camera &depth_cam, const Camera &color_cam, const CudaStream &stream )#

Update point cloud from RGBD images. Converts depth + color images into a colored point cloud on the GPU using the depthToColoredPointCloud kernel.

Note

Internally synchronizes the stream to read back the point count.

Parameters:

depth_image – Device-resident depth image.
color_image – Device-resident color image.
depth_cam – Camera intrinsics for depth sensor.
color_cam – Camera intrinsics for color sensor.
stream – CUDA stream.

Returns:

True if update was initiated, false if visualizer not initialized.

bool updateMesh(const ColorMesh &mesh, const CudaStream &stream)#

Update mesh from nvblox ColorMesh.

Note

Data is copied asynchronously. Call stream.synchronize() before render() to ensure data transfer is complete.

Note

If mesh.triangles.size() is not divisible by 3, a warning is logged and the remaining indices are ignored.

Parameters:

mesh – Device-resident colored mesh with vertices, colors, and triangles. The triangles vector contains flat vertex indices where every 3 consecutive indices form one triangle. Triangle count is mesh.triangles.size() / 3.
stream – CUDA stream.

Returns:

True if update was initiated, false if visualizer not initialized.

bool updateMeshTexture( const ColorImage &atlas_image, const CudaStream &stream )#

Update the texture atlas for textured mesh rendering.

Parameters:

atlas_image – Color image containing the texture atlas (device memory).
stream – CUDA stream.

Returns:

True if upload succeeded.

inline void setDepthRange(float min_depth, float max_depth)#

Set depth range for point cloud conversion. Points outside this range are discarded by updatePointCloud().

Parameters:

min_depth – Minimum valid depth in meters (default: 0.1).
max_depth – Maximum valid depth in meters (default: 10.0).

bool render()#

Render a frame.

Note

Caller must ensure all CUDA operations that update visualizer data have completed (via cudaStreamSynchronize) before calling this method.

Returns:: True if rendering succeeded.

void pollEvents()#: Poll window events.

Note

In headless mode, this is a no-op since there is no window.

bool shouldClose() const#: Check if window should close.

Note

In headless mode, this always returns false since there is no window close event. For headless rendering loops, use your own termination condition.

inline bool isInitialized() const#: Check if renderer is initialized.

inline bool isHeadless() const#: Check if renderer is in headless mode.

inline VkContext *context()#: Get the Vulkan context.

inline ImageVisualizer *imageVisualizer()#: Get the image visualizer.

inline PointCloudVisualizer *pointCloudVisualizer()#: Get the point cloud visualizer.

inline MeshVisualizer *meshVisualizer()#: Get the mesh visualizer.

inline ViewCamera *viewCamera()#: Get the view camera.

inline void setCameraControlsEnabled(bool enabled)#: Enable/disable camera controls.

Note

In headless mode, camera controls have no effect since there is no input. Use ViewCamera methods directly to control the camera.

inline bool cameraControlsEnabled() const#: Check if camera controls are enabled.

inline void setRenderMode(RenderMode mode)#: Set the active render mode.

inline RenderMode renderMode() const#: Get the active render mode.

inline void setClearColor(float r, float g, float b, float a = 1.0f)#: Set the background clear color (RGBA, linear, range [0, 1]).

void resizeWindow(uint32_t width, uint32_t height)#

Resize the window (or render target in headless mode).

Parameters:

width – New width.
height – New height.

void setKeyCallback(VkWindow::KeyCallback callback)#: Set callback for key events (for app-specific handling).

Note

In headless mode, this has no effect since there is no window to receive key events.

In windowed mode, render() handles special cases automatically:

Minimized window: Returns true immediately (no draw, no error).
Window resize: The swapchain is recreated automatically. One frame is skipped after the resize. The ViewCamera aspect ratio updates automatically from the new framebuffer size.

The background clear color is dark gray (0.1, 0.1, 0.1).

Visualizers #

All visualizers share a common bool hasData() const method that returns true if the visualizer has data to render.

Note

Visualizers are not thread-safe. All calls (init, destroy, render, update) must come from a single thread or be externally synchronized.

ImageVisualizer #

struct ImageVisualizerConfig#

Configuration for ImageVisualizer initialization. Provides a unified way to specify image dimensions with sensible defaults.

Public Functions

inline ImageVisualizerConfig(uint32_t width, uint32_t height)#

Create a config with same dimensions for both depth and color.

Parameters:

width – Width for both depth and color images.
height – Height for both depth and color images.

inline ImageVisualizerConfig( uint32_t depth_w, uint32_t depth_h, uint32_t color_w, uint32_t color_h )#: Create a config with independent depth and color dimensions.

Public Members

uint32_t depth_width = 1#: Width of depth images (default: 1 for lazy init).

uint32_t depth_height = 1#: Height of depth images (default: 1 for lazy init).

uint32_t color_width = 1#: Width of color images (default: 1 for lazy init).

uint32_t color_height = 1#: Height of color images (default: 1 for lazy init).

class ImageVisualizer : public TexturedVisualizer<ImageVisualizer, 2>#

Visualizer for depth and color images. Renders images as textured quads with optional depth colormapping.

Note

NOT thread-safe. See VisualizerInterface for thread safety requirements.

Public Types

enum class DepthColormap#

Depth colormap options.

Values:

enumerator kGrayscale#

enumerator kJet#

enumerator kTurbo#

enum class Layout#

Display layout options.

Values:

enumerator kSideBySide#

enumerator kColorOnly#

enumerator kDepthOnly#

enumerator kOverlay#

Public Functions

bool init(VkContext *ctx) override#: Initialize with placeholder 1x1 textures. Call updateDepthImage/updateColorImage to set actual dimensions and data. Use init(ctx, config) if dimensions are known upfront.

void render( VkCommandBuffer cmd, uint32_t x, uint32_t y, uint32_t width, uint32_t height )#: Render into an explicit viewport rect (no camera needed for 2D quads).

bool init(VkContext *ctx, const ImageVisualizerConfig &config)#

Initialize the visualizer with a configuration struct.

Parameters:

ctx – Vulkan context.
config – Configuration specifying image dimensions.

Returns:

True if initialization succeeded.

inline void updateDepth( const float *depth_ptr, const CudaStream &stream )#

Update depth image from CUDA memory.

Parameters:

depth_ptr – CUDA device pointer to float depth data (contiguous).
stream – CUDA stream.

inline void updateColor( const void *color_ptr, const CudaStream &stream )#

Update color image from CUDA memory.

Parameters:

color_ptr – CUDA device pointer to RGBA8 color data (contiguous).
stream – CUDA stream.

inline void setDepthColormap(DepthColormap colormap)#: Set depth colormap.

inline void setLayout(Layout layout)#: Set display layout.

inline void setDepthRange(float min_depth, float max_depth)#: Set depth range for colormap normalization.

inline bool resizeDepthTexture(uint32_t width, uint32_t height)#

Resize depth texture if dimensions differ.

Parameters:

width – New width.
height – New height.

Returns:

True if resize succeeded or not needed.

inline bool resizeColorTexture(uint32_t width, uint32_t height)#

Resize color texture if dimensions differ.

Parameters:

width – New width.
height – New height.

Returns:

True if resize succeeded or not needed.

inline uint32_t depthWidth() const#: Get depth texture width.

inline uint32_t depthHeight() const#: Get depth texture height.

inline uint32_t colorWidth() const#: Get color texture width.

inline uint32_t colorHeight() const#: Get color texture height.

Public Static Attributes

static constexpr size_t kDepthTexture = 0#: Texture index for the depth image.

static constexpr size_t kColorTexture = 1#: Texture index for the color image.

Rendering behavior:

Depth pixels that are ≤ 0 or greater than max_depth are rendered as gray (0.2, 0.2, 0.2). Depth below min_depth (but positive) is clamped to the low end of the colormap rather than rendered as gray.
In kOverlay layout, valid depth pixels are blended on top of the color image at alpha 0.5; pixels where the depth is invalid (≤ 0 or > max_depth) show the color image unmodified.
In kSideBySide layout, the viewport is split at the horizontal midpoint.

Note

ImageVisualizer::setDepthRange() (defaults 0.1 / 5.0) sets the colormap normalization range for the depth texture display. This is a different setting from NvbloxRenderer::setDepthRange() (defaults 0.1 / 10.0), which controls depth filtering inside updatePointCloud(). The two ranges are independent and can be configured separately.

PointCloudVisualizer #

struct PointCloudPoint#: Point data structure (matches nvblox Pointcloud format).

class PointCloudVisualizer : public BufferedVisualizer<PointCloudVisualizer, PointCloudPoint>#

Visualizer for colored point clouds. Renders points with per-vertex position and color.

Note

NOT thread-safe. See VisualizerInterface for thread safety requirements.

Public Types

using Point = PointCloudPoint #: Point type alias for backward compatibility.

Public Functions

void render( VkCommandBuffer cmd, const float *view_proj, float point_size, uint32_t x, uint32_t y, uint32_t width, uint32_t height )#: Render with an explicit view-projection matrix and viewport rect.

void updatePoints( const Point *points_ptr, size_t num_points, const CudaStream &stream )#

Update point cloud from CUDA memory.

Parameters:

points_ptr – CUDA device pointer to Point array.
num_points – Number of points.
stream – CUDA stream.

inline void setPointSize(float size)#: Set point size.

inline float pointSize() const#: Get point size.

inline size_t numPoints() const#: Get number of points.

Rendering behavior:

Points are rendered as circles (not squares).
Vertex colors undergo sRGB-to-linear conversion.

MeshVisualizer #

class MeshVisualizer : public BufferedVisualizer<MeshVisualizer, MeshVertex>#

Visualizer for colored triangle meshes. Renders indexed triangles with per-vertex position and color.

Note

NOT thread-safe. See VisualizerInterface for thread safety requirements.

Public Types

using Vertex = MeshVertex #: Vertex type alias (uses MeshVertex from mesh_to_vertex.h).

Public Functions

void render( VkCommandBuffer cmd, const float *view_proj, uint32_t x, uint32_t y, uint32_t width, uint32_t height )#: Render with an explicit view-projection matrix and viewport rect. Used by renderViews() for XR multi-view rendering.

void updateMesh( const float *positions, const uint8_t *colors, const int *triangles, size_t num_vertices, size_t num_triangles, const CudaStream &stream, const float *uvs = nullptr )#

Update mesh from separate vertex position, color, UV, and triangle arrays.

Note

Triangle indices are assumed to be valid (in range [0, num_vertices)). Passing out-of-bounds indices results in undefined rendering behavior. Index validation is not performed at runtime for performance reasons (indices reside on GPU memory). The caller is responsible for ensuring index validity.

Parameters:

positions – CUDA device pointer to float3 positions (x, y, z per vertex).
colors – CUDA device pointer to uint8 colors (r, g, b per vertex - 3 bytes).
uvs – CUDA device pointer to float2 UVs (u, v per vertex), or nullptr if no UVs (all vertices use vertex color).
triangles – CUDA device pointer to int triangle indices (3 ints per triangle).
num_vertices – Number of vertices.
num_triangles – Number of triangles.
stream – CUDA stream.

bool updateTexture( const void *src, uint32_t width, uint32_t height, const CudaStream &stream )#

Update the texture atlas for projective texture mapping.

Parameters:

src – CUDA device pointer to texture data. Accepts both RGB8 (3 bytes per pixel) and RGBA8 (4 bytes per pixel) — internally stored as RGBA via SharedTexture format conversion.
width – Texture width in pixels.
height – Texture height in pixels.
stream – CUDA stream.

Returns:

True if upload succeeded.

inline bool hasTexture() const#: Check if a texture atlas is currently bound.

inline size_t numVertices() const#: Get number of vertices.

inline size_t numTriangles() const#: Get number of triangles.

inline void setWireframe(bool enabled)#: Set wireframe mode.

inline bool wireframe() const#: Get wireframe mode.

inline void toggleWireframe()#: Toggle wireframe mode.

Rendering behavior:

Backface culling is disabled. Both sides of every triangle are rendered.
Vertex colors undergo sRGB-to-linear conversion.
When a texture atlas is uploaded and UVs are non-negative, the texture is sampled.

Low-Level Data Types #

struct MeshVertex#: Mesh vertex structure for rendering (position + color + UV).

void nvblox::renderer::interleaveMeshVertexData( const float *positions, const uint8_t *colors, MeshVertex *output, size_t num_vertices, const CudaStream &stream, const float *uvs = nullptr )#

Interleave separate position, color, and UV arrays into vertex format.

nvblox stores mesh vertex positions and colors in separate arrays (struct-of-arrays), but Vulkan expects a single interleaved vertex buffer (array-of-structs). This kernel converts between the two layouts on the GPU, writing directly into the CUDA/Vulkan shared buffer so no additional copy is required before rendering.

Parameters:

positions – CUDA device pointer to float positions (x, y, z per vertex).
colors – CUDA device pointer to uint8 colors (r, g, b per vertex).
uvs – CUDA device pointer to float UVs (u, v per vertex), or nullptr if no UVs (all UVs set to -1,-1).
output – CUDA device pointer to output MeshVertex array.
num_vertices – Number of vertices.
stream – CUDA stream.

ViewCamera #

class ViewCamera#

3D view camera with arcball rotation (quaternion-based, no gimbal lock). Provides view and projection matrices for rendering 3D content.

Public Functions

void setTarget(float x, float y, float z)#: Set camera target (look-at point).

void setTarget(const Eigen::Vector3f &target)#: Set camera target (look-at point).

void setDistance(float distance)#: Set camera distance from target.

void setOrbitAngles(float azimuth, float elevation)#

Set camera orbit angles (for initial setup, converts to quaternion).

Parameters:

azimuth – Horizontal angle in radians.
elevation – Vertical angle in radians.

void rotate(float delta_x, float delta_y)#

Rotate camera using arcball rotation (quaternion-based, no gimbal lock).

Parameters:

delta_x – Horizontal rotation delta (in radians).
delta_y – Vertical rotation delta (in radians).

void orbit(float delta_azimuth, float delta_elevation)#: Legacy orbit function (maps to rotate).

void zoom(float delta)#: Zoom camera (change distance).

void pan(float delta_x, float delta_y)#: Pan camera (move target in screen space).

void setRotation(const Eigen::Quaternionf &rotation)#: Set rotation directly (quaternion-based, no gimbal lock).

void reset()#: Reset camera to default orientation.

void setPerspective(float fov_y, float aspect, float near, float far)#

Set perspective projection parameters.

Parameters:

fov_y – Field of view in radians (must be positive).
aspect – Aspect ratio (must be positive).
near – Near clip plane distance (must be positive).
far – Far clip plane distance (must be > near).

void setAspect(float aspect)#

Set aspect ratio.

Parameters:: aspect – Aspect ratio (must be positive).

inline const Eigen::Matrix4f &viewMatrix() const#: Get view matrix.

inline const Eigen::Matrix4f &projMatrix() const#: Get projection matrix.

inline const Eigen::Matrix4f &viewProjMatrix() const#: Get view-projection matrix.

inline const float *viewProjMatrixPtr() const#: Get view-projection matrix as float pointer (for Vulkan push constants). Eigen stores matrices in column-major order, matching Vulkan/OpenGL.

inline const Eigen::Vector3f &position() const#: Get camera position.

inline float distance() const#: Get distance from target.

inline const Eigen::Vector3f &target() const#: Get the look-at point.

inline const Eigen::Quaternionf &rotation() const#: Get the orientation quaternion.

Default projection parameters: FOV 60°, aspect 16:9, near 0.01, far 100.0. Default distance from target: 3.0 m.

Coordinate Conventions #

The renderer uses a right-handed, Y-up coordinate system for 3D visualization. The view matrix follows the OpenGL lookAt convention (forward = -Z in eye space, up = +Y). Matrices are stored column-major (Eigen default).

Point Clouds #

updatePointCloud() unprojects depth pixels using nvblox camera intrinsics (CV camera frame: X-right, Y-down, Z-forward) and writes them into the display frame by negating Y (CV Y-down → Y-up) and negating X as a mirror flip for an intuitive selfie-style view. The Z axis is passed through unchanged. The resulting point cloud is camera-centered – no world transform is applied.

Note

The X negation is a mirror, not a rigid coordinate-system rotation, so the point cloud is not a metrically-correct representation of the scene in any canonical world frame. If you need world-frame points, compute them yourself and use PointCloudVisualizer::updatePoints() instead.

When using the low-level PointCloudVisualizer::updatePoints(), coordinates are passed through as-is and the caller is responsible for axis conventions.

Meshes #

updateMesh() passes vertex positions through without any axis flip or transform. Meshes are rendered in whatever coordinate frame the input data uses (typically the nvblox world frame for ColorMesh).

Note

Point clouds from updatePointCloud() are in a camera-centered frame, while meshes are in the nvblox world frame. To display both in the same scene, use PointCloudVisualizer::updatePoints() with world-frame points instead of updatePointCloud().

Initialization Model #

NvbloxRenderer uses a two-phase lifecycle: a trivial default constructor followed by an explicit init() (or initWithWindow() / initHeadless()) call. There is no constructor that takes a RendererConfig directly.

State before `init()`#

A default-constructed renderer is not yet ready to render. Until init() succeeds:

Query methods (isInitialized(), isHeadless(), renderMode(), cameraControlsEnabled()) and the configuration setters are safe to call.
init*(), initVisualizer(), and render() return false cleanly.
The update methods (updateDepth, updateColor, updatePointCloud, updateMesh, updateMeshTexture) log a warning and return false.
pollEvents() is a no-op and shouldClose() returns true.
destroy() is a safe no-op.
The visualizer, camera, and context accessors (imageVisualizer(), pointCloudVisualizer(), meshVisualizer(), viewCamera(), context()) return nullptr. Any chained call through those pointers will crash.

Even after init() succeeds, the per-mode accessors remain nullptr until initVisualizer(mode) is called for that mode. viewCamera() is created lazily on the first initVisualizer(kPointCloud) or initVisualizer(kMesh) call, and stays nullptr if you only use kImage.

Why initialization is split out #

Init can fail. Vulkan setup may fail because no driver is installed, no GPU is compatible, or no display server is available. The renderer reports such failures via bool return so callers can fall back to a different configuration.
Re-initialization. After destroy(), the same renderer can be init()-ed again with a different configuration — for example to switch between windowed and headless mode, or change the resolution. Note that switching between render modes (e.g. kPointCloud → kMesh) does not require destroy + re-init: call initVisualizer() once for each mode you want, then flip between them at runtime with setRenderMode().
Use as a class member. Owning classes can declare NvbloxRenderer renderer_; and initialize it later, once their configuration is known (from CLI flags, config files, etc.).

Error Handling #

All init and render methods return bool:

Initialization failures return false and log details via glog.
Update methods (updateDepth, updateColor, etc.) return false if the corresponding visualizer has not been initialized.
``render()`` returns false only on Vulkan errors. If the active mode’s visualizer is missing, it logs a warning (rate-limited to once per mode) and returns true with nothing drawn.
Double initialization of the renderer (without destroy()) returns false. Re-initializing a visualizer via initVisualizer() with the same mode destroys and recreates it (with a warning).

Thread Safety #

NvbloxRenderer and all visualizers are not thread-safe. All calls must originate from the same thread.

In windowed mode that thread must specifically be the program’s main thread. GLFW only officially supports window creation, glfwPollEvents(), and window destruction on the main thread; calling them from a worker thread may appear to work but can deadlock or misbehave under some X11 / Wayland compositors.

In headless mode there is no GLFW and no window, so any single thread works.

The synchronization contract between CUDA and Vulkan is:

Call the update methods (updateDepth, updateColor, updatePointCloud, updateMesh, updateMeshTexture) with a CUDA stream.
Call stream.synchronize() to ensure all CUDA writes are complete.
Call render() to submit Vulkan commands.

This CPU-side synchronization is used for simplicity and portability.

Limits #

Limit	Value
Max points / vertices / triangles	50,000,000 each
Max buffer size	1 GB
Texture dimensions	1 to 16,384 per side

Buffers grow automatically as needed. If a data update exceeds the maximum count or buffer size, the renderer logs an error and silently drops the update (the previous data remains unchanged).