Neural Reconstruction (NuRec / NRE)

Source And Attribution

Adapted from the NVIDIA Omniverse NuRec agent skills at https://github.com/NVIDIA/nurec-skills (skills/nre, skills/physical-ai-datasets, skills/ncore) and the NVIDIA NCore data library at https://github.com/NVIDIA/ncore.

The capability routing table, the easy mix-ups, the safe secret-verification pattern, and several troubleshooting rows below are adapted from the NVIDIA router skill https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-neural-reconstruction (Apache-2.0), pinned at commit 0122ea0 (2026-08-01). That skill is a router: it never runs anything, it decides which upstream sibling skill answers a question. This skill is the opposite — it is the workbench implementation — so the router's picker table is re-pointed at real npa workbench nurec verbs, and each row upstream owns is marked as such rather than reproduced.

Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. Upstream licenses are Apache-2.0 and CC-BY-4.0. Trademarks (NVIDIA, Omniverse, NuRec, NRE, Isaac Sim, Cosmos) belong to NVIDIA. See skills/NOTICE-NVIDIA-SKILLS.

The NCore conversion image packages pinned Apache-2.0 NVIDIA source and its MIT COLMAP reader; it contains no dataset, weights or proprietary NRE payload. NRE remains a separately licensed NGC runtime. The new COLMAP ingestion path is not yet live validated and has no accepted public image release. See skills/NOTICE-NVIDIA-NCORE-COLMAP and docs/workbench/guides/nurec-colmap-reconstruct.md. Existing preconverted-NCore results below do not validate the new conversion path.

When To Use

Load this skill when the user wants to:

• turn a real sensor capture (photographs, multi-camera clips) into a renderable 3D Gaussian scene and render novel views from it;
• run, modify, or debug npa workbench nurec or npa/src/npa/workbench/nurec/examples/nurec-reconstruct.yaml;
• work out why an NCore sequence fails to load in NRE;
• choose the GPU for a reconstruction or rendering job;
• make a reconstruction run show up in the NPA agent's Rerun panel.

Do NOT use this skill for: Cosmos video augmentation (use skills/workflows/physical-ai-data-factory/SKILL.md), per-object asset extraction from sparse views (upstream asset-harvester, not implemented here), or generic Rerun visualization of a Sim2Real run (skills/workbench/sim2real-engine/SKILL.md).

GPU Routing (hard constraint)

Gaussian reconstruction and rasterization are RT-core work. Route these jobs at:

• RTX PRO 6000 Blackwell — RTXPRO-6000-BLACKWELL-SERVER-EDITION:1 on the RT-core Kubernetes context (what the shipped workflow uses), or
• L40S — L40S:1.

Never route the reconstruct/render path at H100 / H200 / A100 / B200: they have no RT cores. npa workbench nurec check fails with an explicit error when the visible GPU is one of those (has_rt_cores). NRE additionally requires driver R580+ on Blackwell and >= 24 GB VRAM (48 GB+ recommended). See skills/atomic/gpu-selection/SKILL.md.

Container Access (read this before debugging a pull failure)

NRE ships only as a closed-source NGC container, so the container is the runtime — there is no source/pip path.

Repository	With a standard NGC_API_KEY
nvcr.io/nvidia/nre/nre	402 Payment Required — needs an extra entitlement
nvcr.io/nvidia/nre/nre-ga	pullable — the General Availability channel
nvcr.io/nvidia/nre/nre-tools-ga	pullable (auxiliary seg/depth data only)

Use the -ga repositories. npa workbench nurec check reports ngc_image: entitlement-required rather than failing opaquely when a non-GA reference is configured.

Image startup preflight overrides the vendor entrypoint with /bin/bash, as SkyPilot does, and installs missing SSH, rsync, and service helpers before checking worker capabilities. NRE's /app/run entrypoint cannot forward an arbitrary shell probe. Prepared results are never cached as image attestations; they depend on package access from the selected cluster. First-party image attestations and strict runtime byte probes keep their existing requirements.

The live regression is test_vendor_image_runtime_bootstrap_live in npa/tests/e2e/test_image_bootstrap_terminal_probe_live.py. Select the exact preflighted image/digest, NPA_E2E_KUBECONTEXT, KUBECONFIG, and any existing pull Secret names through NPA_E2E_IMAGE_PULL_SECRETS (comma separated). It requires compatible worker capabilities and verified deletion of its probe pod.

NPA_INTEGRATION_E2E=1 npa/.venv/bin/python -m pytest \
  npa/tests/e2e/test_image_bootstrap_terminal_probe_live.py \
  -k vendor_image_runtime_bootstrap -q

Real Entrypoints

Every stage is a real command; nothing here is a manifest stub.

npa workbench nurec check       # NGC pullability + HF download rights + RT-core GPU
npa workbench nurec convert-colmap # CPU: official Apache-2.0 COLMAP -> NCore V4
npa workbench nurec fetch       # real NCore V4 shards + derived rig pose edge
npa workbench nurec reconstruct # NRE 3DGUT training -> renderable USDZ + metrics
npa workbench nurec render      # `nre render` novel views (rig offset, not training views)
npa workbench nurec visualize   # reports/sim2real.rrd for the agent's Rerun panel
npa workbench nurec finalize    # reports/final.json aggregate
npa workbench nurec status      # what a run prefix holds, stage by stage

Concern	Implementation
Pure logic + argv builders	npa/src/npa/workbench/nurec/nurec.py
COLMAP ingestion + complete V4 verification	npa/src/npa/workbench/nurec/colmap.py
NCore rig-pose derivation	npa/src/npa/workbench/nurec/ncore_rig.py
CLI	npa/src/npa/cli/nurec/__init__.py
SDK	npa.sdk.workbench.nurec (check, convert_colmap, fetch, reconstruct, render, visualize, finalize, status); the framework-free API is re-exported from npa.workbench.nurec
SkyPilot workflow	npa/src/npa/workbench/nurec/examples/nurec-reconstruct.yaml
Main declarative workflow	workflows/main/nurec-reconstruct.yaml
COLMAP source workflow (not yet live validated)	workflows/testing/nurec-colmap-reconstruct.yaml
Rerun recording	npa.workflows.data_factory_viz.build_run_rrd

Input Data

Default: nvidia/PhysicalAI-NuRec-PPISP — ungated, CC-BY-4.0, real photographic captures of four outdoor object-centric scenes shipped already in NCore V4, which is what NRE consumes. Scene struktur28, variant auto, is the small default (59 images across two cameras, ~1.1 GB archive).

The nvidia/PhysicalAI-Autonomous-Vehicles* family (raw clips, -NCore, and the pre-built -NuRec USDZ scenes) is gated: the account that owns HF_TOKEN must accept the NVIDIA AV Dataset License on each dataset page before any token can pull a byte. npa workbench nurec check reports hf_dataset: gated for those until it is accepted — it probes real download authorization, not just metadata visibility, because a gated repo still answers 200 for /api/datasets/<id>.

COLMAP source ingestion (not yet live validated)

Use workflows/testing/nurec-colmap-reconstruct.yaml for original COLMAP images, camera calibration/poses and sparse points. Its five states are conversion -> existing NRE reconstruction -> render -> visualize -> finalize; the workflow owns the graph, with no separate Python orchestrator. The exact CLI/toolRef contract is documented in docs/workbench/guides/nurec-colmap-reconstruct.md.

• Select the full struktur28 directory in the PPISP ZIP pinned at 2521064a3af6ab1c1caa2ba1b01ddde7eecded69: 518 images, 3 cameras and 163,453 sparse source points. The separate struktur28_auto directory is not the selected capture. Never replace the full source with that smaller variant.
• --input-path takes an S3 ZIP or prefix; --output-path is the exact portable sequence prefix. Feed the same trailing-slash prefix to NRE's --ncore-uri. Use --dataset-root struktur28 --rig-mode derive and npa_rig poses.
• Choose a fresh destination prefix for every publication attempt. The permanent provider-conditional claim prevents overlapping writers and replacement; after interruption, retry into a new prefix rather than deleting or expiring a claim.
• The CPU image runs NVIDIA/ncore 59c698d206da92b406a4f72619fce3b3a2c64bfd, with its pinned MIT trueprice/pycolmap reader. Scope the immutable development image override to workbench.nurec.convert_colmap; no accepted/public availability is implied.
• Preserve virtual per-camera 1 FPS timestamps as photographic ordering, not synchronized capture time. Sparse SfM points are not physical LiDAR. Record upstream's near-origin point filtering and any derived rig changes.
• Reopen every image/calibration/pose/point, compare full source counts and finite geometry, and retain hashes, source notices and CC-BY-4.0 attribution with runtime artifacts. No datasets or weights may be baked.
• Compare the complete camera model and distortion coefficients, including each downsampled camera, against source calibration. Matching counts and intrinsics alone miss equal-intrinsics cameras with different distortion. Keep the drift-checked upstream compatibility patches and post-patch source inventory.
• Verify converted member hashes and the publication claim before the NuRec reader opens a sequence. Return only the fresh download generation, never an overlay into an old cache: even two valid inventories can select the wrong capture after merging. Reject an explicit source without sequence metadata instead of falling back to a previously fetched dataset.
• NRE remains proprietary and separately gated. Keep its full native recipe (max_epochs: "0") on the explicit RTX PRO 6000 profile; never remap to B200.
• Live matrix: real executing gpu case, with complete hash-verified source seeding and S3 conversion-member verification. Supply the development digest through NPA_E2E_IMAGE_OVERRIDE_NCORE. Report actual numeric GPU results before changing the pending validation status; older NuRec results do not qualify.

The rig -> world Pose Edge (the thing that breaks first)

NRE's NCore data source requires a ("rig", "world") pose-graph edge:

# nre/datasets/ncore.py, nre-ga 26.04
# TODO: frame-pose only data might fail here as there are no rig poses ...
rig_world_edge = unpack_optional(
    sequence_loader.pose_graph.get_edge("rig", "world"),
    msg="Rig-to-world poses are currently required to determine scene extend")

Object-centric captures have no vehicle rig, and NVIDIA's own COLMAP -> NCore converter stores per-camera <camera> -> world poses with no rig node (tools/data_converter/colmap/converter.py: reference_frame = "world"). So every COLMAP-derived NCore sequence — including PPISP's own export — fails to load with:

ValueError
Rig-to-world poses are currently required to determine scene extend

dataset.frame_generic_data_pose_overwrite=true does not help either; it needs a T_sensor_worlds generic-data field these sequences do not carry.

npa workbench nurec fetch fixes this by default. For a single-camera capture the rig is the camera, so rig -> world is exactly that camera's pose trajectory. ncore_rig.derive_rig_poses writes one small extra component store plus a new sequence meta-file that symlinks the original shards, so the source data is never modified:

• the derived poses live in their own component instance npa_rig, not default — open_component_readers asserts instance names are unique across a sequence's stores, and re-using default raises Component instance default encountered multiple times;
• selecting a poses group replaces the pose set rather than merging, so the derived component carries a complete copy of the original edges plus the rig edge;
• reconstruct then passes dataset.poses_component_group=npa_rig.

Pass --no-derive-rig for AV-style sequences that already ship a rig edge (the derivation short-circuits with already_present: true anyway), and --reference-camera <id> to pin which camera becomes the rig.

Recipe Selection

The container resolves --config-name against its own configs/ tree. Pick by capture shape:

Capture	Recipe
Object-centric / static / camera-only (PPISP, COLMAP-style)	configs/experimental/3dgut/3dgut_colmap.yaml (the default)
Waymo Open Dataset	configs/apps/AV/Waymo/3dgut_dynamic*.yaml
PhysicalAI Autonomous Vehicles (Hyperion-8.1)	configs/apps/prod/Hyperion-8.1/car2sim_6cam.yaml
PandaSet / NV / Tesla / Alpasim	configs/apps/AV/{PandaSet,NV,Tesla}/..., configs/apps/Alpasim/...

The default recipe already composes options/artifact: default (which is what sets checkpoint.artifact.enabled, i.e. the renderable USDZ), MCMC densification, SfM-point-cloud initialization, and disables difix/mesh/ground. For a derived-rig capture with multiple selected cameras, NPA replaces only the background initializer through the native model/gaussians/initialization@model.layers.background.initialization=accumulated_point_cloud config group. The native SfM initializer asserts one camera on the entire data source; restricting an initialization-only camera list cannot fix it. Keep all discovered cameras, export all verified NCore world XYZ and uint8 RGB points to the reconstruction output's initialization/ncore-sfm.ply, and let NRE apply its world-to-NRE transform. Match the complete point count and disable optional random near/far seed points to preserve source-point-only initialization. Never mutate the conversion generation or change the native training budget. Single-camera selections retain native SfM; custom recipes and explicit initialization overrides remain the caller's responsibility. The dry run shows the planned native arguments without writing initialization artifacts.

Retain initialization/ncore-sfm.json with the source/conversion and PLY hashes, selected cameras, count, and recipe/image identity. Native parsed.yaml records effective configuration, while USDZ data_info.json is input sequence metadata. Available frames and exported ground truth do not establish split membership or actual sampled training frames. Require separate native split/sampler evidence before claiming that every source image participated in training. The COLMAP multi-camera path still requires a real full RTX run after integration; offline PLY tests do not validate native reconstruction quality.

Enumerate what a given release actually ships with:

find /app/run.runfiles/_main/configs -name '*.yaml' | sort
/app/run --help          # sub-command inventory
/app/run render --help   # authoritative flag surface

Novel Views vs Training Views

nre render defaults to --replicate-training-views, which re-renders views the model was trained on. That is not a novel view. The tool therefore emits --no-replicate-training-views plus a rig offset by default:

• --rig-translation-offset and --rig-rotation-offset are FLOAT... (three values) upstream; the CLI accepts one "x,y,z" string and expands it;
• a zero offset with no custom trajectory is rejected rather than silently producing training views;
• --renderer default (the artifact's own trained renderer) is the default. nrend is faster but needs the nrend model dictionary embedded in the USDZ, which the object-centric recipe disables.

Artifact Layout

One S3 run prefix per run, so the agent's artifact browser picks it up:

s3://<bucket>/<prefix>/neural-reconstruction/<run_id>/
  ncore/manifest.json                    # dataset, scene, sensors, rig derivation
  input/camera_images/<camera>/*.jpg     # real capture frames (export-ncore-benchmark-gt)
  reconstruction/last.usdz               # renderable Gaussian scene
  reconstruction/parsed.yaml
  reconstruction/metrics.yaml            # test/psnr, test/ssim, test/lpips
  reconstruction/val/...                 # NRE validation renders + videos
  novel_views/<camera>/*.png             # rig-offset novel views
  novel_views/<camera>.mp4
  reports/final.json
  reports/sim2real.rrd                   # preferred artifact for the Rerun panel

<run_id> must be a single safe segment embedding the submit timestamp, e.g. neural-reconstruction-struktur28-20260731t050118z, so the run picker dates the run by when it started (npa.workflows.artifacts._run_started_at).

.usdz classifies as download, which is correct: no browser renders USDZ and the agent has no USDZ viewer. Viewability comes from the .rrd, .png, .mp4 and .json.

For a COLMAP-derived run, visualization fails closed unless the source attribution, conversion report, and rig-derivation sidecar are all readable. Detection uses both the pre-download S3 inventory and local files, so a partial subtree download cannot produce an RRD backed only by novel-view media. This strict trio applies only after a COLMAP lineage marker is present; optional data-factory subtrees and legacy preconverted-NCore runs remain best effort.

Which Capability Answers This?

Adapted from the NVIDIA router skill's picker table, re-pointed at what this repo actually implements. "Upstream" means the workbench has no verb for it: read the named sibling skill at https://github.com/NVIDIA/nurec-skills and run it yourself; do not invent a workbench command for it.

I want to...	Where
Check NGC/HF access and that the GPU has RT cores, before pulling 14 GB	npa workbench nurec check
Download a published NVIDIA NuRec/PhysicalAI capture in NCore V4	npa workbench nurec fetch
Train a reconstruction from an NCore clip and get a USDZ	npa workbench nurec reconstruct
Render novel views along a shifted rig trajectory	npa workbench nurec render
Get a Rerun recording the NPA agent will display	npa workbench nurec visualize
Run all of the above on a GPU as one pipeline	workflows/main/nurec-reconstruct.yaml
Measure PSNR / SSIM / LPIPS	Already emitted -- reconstruction/metrics.yaml, and gaussians/summary in the .rrd
Convert a COLMAP reconstruction to NCore V4	npa workbench nurec convert-colmap on CPU; not yet live validated
Convert a non-COLMAP recording (drone, RGB-D, ROS 2 bag, ScanNet++)	Upstream ncore; these input-specific converters are not wired here
Serve frames to CARLA / Isaac Sim / a custom simulator	Upstream nre (serve-grpc) -- not wired, see Limitations
Render LiDAR sweeps from a USDZ	Upstream nre (render-grpc --lidar) -- not wired
Extract individual 3D objects (cars, pedestrians) from a clip	Upstream asset-harvester -- not wired
Clean up ghosting / floaters / flicker in rendered frames	Upstream nurec-fixer (DiffusionHarmonizer), or NRE's inline --enable-difix -- neither wired
Generate segmentation / depth / ego-mask auxiliary inputs	Upstream nre via the nre-tools image -- not wired, see Limitations
Package CAD or source meshes for simulation	Not NuRec at all -- that is SimReady, a different pipeline

Easy Mix-Ups

Adapted from the router skill's references/mix-ups.md; the last row is workbench-specific.

• NuRec vs NRE. NuRec is the product, NRE ("Neural Reconstruction Engine") is the engine that trains and renders. Used interchangeably in most docs.
• ncore then nre, never instead of. NCore V4 is the input format; NRE reads it. They run in order. If NRE says a clip "is not valid NCore V4", the conversion step is missing, not a training bug.
• 3DGUT vs 3DGRT. Two Gaussian-splatting flavours inside NRE. The Hydra recipe picks one; you should not normally set it by hand.
• PhysicalAI-Autonomous-Vehicles-NuRec vs Cosmos-Drive-Dreams. Both AV datasets on Hugging Face and easy to confuse. The former is real driving footage under the gated AV license; the latter is synthetic weather-augmented video under CC-BY-4.0.
• NRE's inline --enable-difix vs the standalone nurec-fixer. Upstream documents --enable-difix as a built-in cleanup pass during rendering, while nurec-fixer wraps the public DiffusionHarmonizer release for frames already rendered. Neither is wired into a workbench verb, and the flag has not been exercised against nre-ga 26.04 here -- treat it as upstream-documented, not as a verified workbench feature.
• A missing rig -> world edge is not a corrupt download. The most common first failure is a pose-graph gap this workflow derives for you. See The rig -> world Pose Edge.

Troubleshooting

Rows marked (upstream) are adapted from the router skill's troubleshooting table; the rest were hit for real while landing this capability.

Symptom	Cause	Fix
402 Payment Required pulling nvcr.io/nvidia/nre/nre	That repo needs an extra entitlement (upstream: denied: requested access ...)	Use the -ga channel, nvcr.io/nvidia/nre/nre-ga:26.04. nurec check reports entitlement-required
401/403 on a gated nvidia/PhysicalAI-Autonomous-Vehicles* override (upstream)	Gated dataset access is absent, or HF_TOKEN lacks read	Accept/request access on Hugging Face as the token owner, then re-run nurec check; the default PhysicalAI-NuRec-PPISP remains anonymous
NRE will not load a clip: "not valid NCore V4" (upstream)	The recording was never converted	For COLMAP, use nurec convert-colmap; other source formats still need upstream conversion
KeyError: ('rig', 'world') / no scene extent	The clip has camera poses but no rig edge -- NVIDIA's own COLMAP converter omits it	Automatic: reconstruct derives it. See the pose-edge section
Requested lidars not present in the data: dummy_lidar	The recipe ships placeholder sensor ids	Automatic: the sequence's real ids are adopted
Requested cameras not present: camera_front_wide_120fov	Same, for AV camera names	Automatic: same adoption path
Only one camera sensor is currently supported	SfM point-cloud init is single-camera	Automatic: trains on the recorded reference camera and warns which cameras were dropped
Cluster never finishes provisioning, sudo: command not found	The image ships no sudo; SkyPilot's K8s bootstrap calls it unconditionally	Automatic: pod_config initContainer installs a shim
OOM / bus error early in training	/dev/shm defaults to 64 MB	Automatic: 64 Gi emptyDir{medium: Memory}
USDZ looks like an early preview	<run>/artifacts/<step>.usdz, first-alphabetical picks step 1000	Automatic: latest_usdz() picks the newest by step
Renders look identical to the input frames	nre render defaults to --replicate-training-views	Automatic: the negation plus a non-zero rig offset is always emitted; a zero offset is rejected
Output files owned by root after a local docker run (upstream)	-u $(id -u):$(id -g) was omitted	sudo chown -R "$(id -u):$(id -g)" <dir>, and pass -u next time. Not an issue in-pod, which runs as root by design
Ghosting / floaters / flicker in rendered frames (upstream)	No cleanup pass	Upstream nurec-fixer, or NRE's inline --enable-difix. Neither is wired here
A stage runs but publishes nothing to S3	The stage ran in its own pod and wrote only to /tmp	Pass the handoff URIs. Every declarative stage is a separate pod

Verifying Secrets Safely

From the router skill's references/secrets-handling.md. Never interpolate a token into an ad-hoc shell check. In particular this common line prints the token:

echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"   # WRONG: emits yes<token>