Stage Manager

The Stage Manager handles all interactions with the OpenUSD scene. It owns the USD Stage object, controls both fixed and free camera behavior, and generates the necessary metadata for NeRF training. Key responsibilities include:

• Loading and managing the USD stage through loadUsdStage(), supporting both static and dynamic assets.
• Camera frame generation via generateCameraFrames(), which supports keyframe-based hemispherical sampling or manually configured free camera states.
• Exporting data using exportDataJson() to write camera intrinsics and extrinsics to structured JSON, alongside the rendered frames.

1
void StageManager::exportDataJson() const
2
{
3
  QJsonObject json;
4
  QJsonArray frames;
3 collapsed lines
5

6
  for (const auto& frameMeta : m_allFrameMeta) {
7
    frames.append(frameMeta->toJson(m_outputDataJsonPath));
8
  }
9

10
  float aperture, focal;
11
  /* UsdGeomCamera */ m_geomCamera.GetHorizontalApertureAttr().Get(&aperture);
12
  m_geomCamera.GetFocalLengthAttr().Get(&focal);
13

14
  json["frames"] = frames;
15
  // compute FOV x using half-angle tangent trig rule
16
  json["camera_angle_x"] = 2.f * atanf(aperture / (2.f * focal));
17

18
  QFile file(m_outputDataJsonPath);
19
  if (file.open(QFile::WriteOnly)) {
20
    file.write(QJsonDocument(json).toJson());
21
  }
22
}

• State reset and reinitialization through reset(), ensuring the stage remains clean between different asset loads or capture runs.

Internally, the Stage Manager stores:

• A UsdStage reference for scene graph traversal
• A UsdGeomCamera for viewport alignment
• A free camera controller
• A pose generation module for determining view samples

Render Engine

The Render Engine integrates with Hydra to provide OpenGL-based rendering of the current scene state. It directly queries framebuffers and AOVs to retrieve rendered color data and facilitates continuous rendering for multi-view frame capture. Core responsibilities include:

• Render dispatch and viewport clearing via render() and clearRender(), respectively.
• Triggering recording cycles through record(), which iterates over camera poses and requests the corresponding render output.
• Frame resizing with resize() to dynamically adjust to viewport changes.
• Scene change tracking using isDirty() to avoid redundant re-renders.

It also holds references to:

• The Hydra scene delegate
• Current rendering parameters
• A shared isDirty state flag for UI-refresh synchronization

1
void RenderEngine::record(StageManager* manager)
2
{
15 collapsed lines
3
  HgiTextureHandle textureHandle = m_imagingEngine.GetAovTexture(HdAovTokens->color);
4

5
  HioImage::StorageSpec storage;
6
  storage.flipped = true;
7

8
  size_t size = 0;
9
  HdStTextureUtils::AlignedBuffer<uint8_t> mappedColorTextureBuffer;
10

11
  // set buffer property values
12
  storage.width = textureHandle->GetDescriptor().dimensions[0];
13
  storage.height = textureHandle->GetDescriptor().dimensions[1];
14
  storage.format = HdxGetHioFormat(textureHandle->GetDescriptor().format);
15

16
  // where `Hgi` stands for "Hydra Graphics Interface"
17
  mappedColorTextureBuffer = HdStTextureUtils::HgiTextureReadback(m_imagingEngine.GetHgi(), textureHandle, &size);
18
  storage.data = mappedColorTextureBuffer.get();
19

20
  int frame = manager->getCurrentFrame();
21
  QString filename = manager->getOutputImagePath(frame);
22

23
  // where `Hio` stands for "Hydra Input/Output"
24
  const HioImageSharedPtr image = HioImage::OpenForWriting(filename.toStdString());
25
  const bool writeSuccess = image && image->Write(storage);
26
}

Together, these components create a tightly coupled yet extensible rendering loop that supports user-driven data collection for downstream neural rendering. The entire system is built with modular C++ and configured through CMake, enabling cross-platform compilation and integration into broader pipelines.

Input and Sampling Strategy

At the core of NeRF is a learnable function $f_{\theta }$ that represents the scene as a continuous volumetric field. This function defines how any point in 3D space appears when viewed from a particular direction.

Here, $f_{\theta }$ maps a 3D point $𝐱$ along a camera ray and the corresponding viewing direction $𝐝$ to a volume density $\sigma$, which encodes scene geometry, and an RGB color $c$, which encodes appearance.

Rays are generated per-image by the projection of pixels through the pinhole camera model. For each pixel, a ray origin and direction are computed using the camera intrinsics and extrinsics. Along each ray, multiple sample points are drawn between the near and far depth bounds.

At a resolution of 100×100 pixels with 64 samples per ray, training on 106 images results in $100\times 100\times 64\times 106=\mathrm{67,840,000}$ 3D point samples per batch.

1
def get_rays(
2
  height: int, width: int, focal_length: float, c2w: torch.Tensor
3
) -> Tuple[torch.Tensor, torch.Tensor]:
4

5
  # Create two tensors that correspond with x & y values of pixels
6
  # i is a (width, height) grid where each row has same x-value
11 collapsed lines
7
  # j is a (width, height) grid where each column as same y-value
8
  i, j = torch.meshgrid(
9
    torch.arange(width, dtype=torch.float32).to(c2w),
10
    torch.arange(height, dtype=torch.float32).to(c2w),
11
    indexing="ij",
12
  )
13

14
  # Transpose to swap dimensions. Industry-standard, as images are inherently row-major (height, width).
15
  i, j = i.transpose(-1, -2), j.transpose(-1, -2)
16

17
  # Convert ray directions to camera-space
18
  # For pinhole camera model, map to [(-1, -1), (1, 1)] and then scale by focal length
19
  directions = torch.stack(
20
    [
21
    (i - width * 0.5) / focal_length, # x: (i - width/2) / focal
22
    -(j - height * 0.5) / focal_length, # y: -(j - height/2) / focal
23
    -torch.ones_like(i), # z: -1 (-1 is camera's forward)
24
    ],
25
    dim=-1,
26
  )
27

28
  # Use "camera-to-world" matrix to convert ray directions to world coords
29
  rays_d = torch.sum(directions[..., None, :] * c2w[:3, :3], dim=-1)
30

31
  # Ray origin is uniform, extracted from the last column of "camera-to-world" matrix
32
  rays_o = c2w[:3, -1].expand(rays_d.shape)
33

34
  return rays_o, rays_d

Positional Encoding

A standard multilayer perceptron (MLP) is biased toward learning low-frequency, smooth functions. However, Radiance Fields should be able to represent sharp geometric features. To address this, NeRF applies a positional encoding to the input coordinates before passing them to the network.

This encoding is applied to each scalar component of these vectors. So let $p$ denote a scalar input component, where:

The encoding uses a fixed set of frequency bands $\{{\omega }_k\}$, chosen as powers of two in NeRF:

where $L$ is an empirically-chosen parameter. Larger values of L increase frequency, so typically in NeRF $L=10$ for spatial positions and $L=4$ for viewing directions, as position (which corresponds to scene geometry) requires more higher-frequency variation than direction.

Then, for each scalar input $p$, the positional encoding will be computed as:

So for position vector $𝐱$, the encoding will be:

And similar for viewing direction vector $𝐝$.

These encoded vectors form the actual inputs to the MLP, effectively representing each input component as sinusoidal basis functions rather than raw $\{x,y,z\}$ and $\{d_x,d_y,d_z\}$ values.

MLP Model Architecture

In practice, NeRF parameterizes $f_{\theta }$ using a deep multilayer perceptron (MLP). The MLP acts as a continuous function approximator, allowing the scene to be queried at arbitrary spatial locations and view directions without discretizing space.

The MLP architecture is designed to reflect the structure of the mapping defined by $f_{\theta }$:

• 8 hidden (intermediate) layers, each with 256 channels and ReLU (Rectified Linear Unit) activations, provide sufficient expressive capacity to model complex geometry and appearance.
• A residual (skip) connection after the 4th layer helps preserve low-frequency spatial information while still enabling deeper layers.
• Separate output heads are used for RGB color and volume density ($\sigma$).
  • Density depends only on spatial position $𝐱$
  • Color depends on both position $𝐱$ and viewing direction $𝐝$.
  • Accordingly, density is predicted from the base network, while color is predicted from a branch that incorporates view-direction information.

3 collapsed lines
1
class MLP(nn.Module):
2
  """
3
  Multilayer Perceptron module.
4
  """
5

6
  def __init__(
7
    self,
8
    d_input: int = 3, # `d_` prefix commonly used for dimensions
9
    n_layers: int = 8,
10
    d_filter: int = 256, # dimension of hidden layers
11
    skip: Tuple[int] = (4,),
12
    d_viewdirs: Optional[int] = None, # do not set for view-independency, i.e. predicted color will depend only on position
13
  ):

Volumetric Rendering

NeRF renders images by querying the scene function $f_{\theta }$ along camera rays and compositing the resulting predictions using volumetric rendering. This process proceeds in two stages: a coarse pass to locate relevant regions along each ray, followed by a fine pass to refine the rendering.

Final Volumetric Compositing

The raw outputs of the MLP—density and color predictions at sampled points—are converted into final renderings using the volume rendering equation. For each ray, this produces:

• an RGB map, obtained by accumulating color contributions along the ray,
• a depth map, computed as the expected distance along the ray,
• an opacity map, representing the accumulated opacity (one minus transmittance),
• and sample weights, which quantify each sample’s contribution.

The final rendered outputs are computed using the predictions from the fine pass, while the coarse pass serves to guide sampling during training.

Python CLI and Workflow

The CLI, implemented using Click and Questionary PyPI packages, walks users through each stage of the pipeline.