USD Asset Pipeline

Purpose

Discover assets, measure bounds and shaders, swap placeholders, correct offsets, and validate headless shader compatibility before rendering.

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Targets Isaac Sim 6 / Kit 110 unless a section states otherwise.
• Does not replace official NVIDIA documentation for unsupported edge cases.

Troubleshooting

Error / symptom	Cause	Solution
Extension or import not found	Wrong $ISAAC_SIM_DIR or stale build	Point env vars at _build/linux-x86_64/release or rebuild
Black or empty frames	Missing lights or non-RTX render mode	Add dome/key light; confirm RTX / PathTracing settings
Hang on stage load or first render	MDL compile or oversized stage	Follow isolation steps in isaac-sim-troubleshooting

Available Scripts

Script	Purpose	Arguments
scripts/articulation_builder.py	Helpers for building USD articulation trees with physics joints	see script --help
scripts/measure_asset.py	Measure a USD asset: bounding box, shader type, and prim count	see script --help
scripts/place_assets.py	Place USD assets at block positions with bbox-center offset correction	see script --help

Running scripts

From agent runtimes that expose skill execution helpers, invoke helpers with run_script():

run_script("scripts/articulation_builder.py", args=["--help"])

From a built Isaac Sim tree, run the same file with ./python.sh (Linux) or python.bat (Windows) from _build/*/release, or execute shell helpers directly when they do not require the simulator.

When to use

• Catalog USD assets from a folder tree (sizes, shaders, prim counts).
• Replace placeholder geometry (cubes/spheres/bboxes) with real assets.
• Build set-dressed scenes from modular libraries.
• Validate headless rendering compatibility (MDL vs UsdPreviewSurface).
• Cube-prototype to real-asset swap workflows.

Core Concepts

The Placeholder-to-Asset Pipeline

Real-world USD scene building follows this pattern:

1. Prototype with cubes — Layout spatial zones using colored UsdGeom.Cube meshes
2. Catalog assets — Measure every candidate USD asset (bbox, shaders, prim count)
3. Map blocks to assets — Match placeholder types to appropriately-sized real assets
4. Place with offset correction — Reference assets at block positions, correcting for asset bbox center offset
5. Validate renders — Vision model + domain expert scoring
6. Iterate — Fix overshoot, corridor intrusion, scale mismatches

Why Bbox Offset Correction Matters

Most USD assets are NOT centered at origin. A rack asset might have its bbox center at (46.7, 104.6, 4.5) — if you place it at the target position (112.0, 20.0, 0.0) without correction, it lands 46.7m east and 104.6m north of where you want it.

Formula:

translate_x = target_x - asset_bbox_center_x
translate_y = target_y - asset_bbox_center_y
translate_z = -asset_bbox_min_z  (puts asset base on ground plane)

Phase 1: Asset Discovery & Measurement

Script Pattern (Kit Python — No Renderer Needed)

measure_asset(path) — open a USD file, compute bbox, detect UsdPreviewSurface shader, count prims. Returns dict with width/depth/height/center/mpu/prims/dual_shader/shader_tag. catalog_assets(root_dir, extensions) — recursively find and measure all USD assets.

See scripts/measure_asset.py.

Key Learnings

• Use Usd.Stage.Open() not Sdf.Layer.FindOrOpen() — Sdf fails silently on binary .usd crate files, returns default mpu=1.0
• Always check mpu — Some assets use cm (mpu=0.01), some use meters (mpu=1.0). Scale measurements accordingly.
• Invalid bbox (min > 1e30) means the asset didn't compose — usually missing references or payloads
• Kit Python (kit/python/bin/python3) is fastest for measurement — no renderer startup needed
• For bbox in Kit runtime (SimulationApp), define a temp prim with reference, update a few frames, then compute bbox — more reliable for complex compositions

Phase 2: Shader Compatibility Check

The MDL Problem on Headless arm64

Shader Type	Headless SimulationApp	Isaac Sim GUI	OVRTX
UsdPreviewSurface only	renders	renders	renders
MDL + UsdPreviewSurface (dual)	falls back to Preview	uses MDL	uses MDL
MDL only (sourceAsset)	black	renders	renders
No materials	grey/invisible	grey	grey

Rule: For headless rendering pipelines, ONLY use assets with UsdPreviewSurface fallback (dual-shader) or native UsdPreviewSurface.

Xvfb Discovery

Running Isaac Sim under DISPLAY=:99 (Xvfb virtual framebuffer) instead of the locked real display :0 produces non-black renders for some MDL assets. Not fully reliable but worth trying:

# Start Xvfb
Xvfb :99 -screen 0 1920x1080x24 &>/dev/null &

# Run with virtual display
DISPLAY=:99 isaac-sim.sh --exec script.py

Identifying Dual-Shader Assets

Look for these patterns in USD:

• info:id = "UsdPreviewSurface" on any Shader prim → headless-safe
• info:mdl:sourceAsset without UsdPreviewSurface sibling → MDL-only, headless-unsafe
• Lightspeed-processed assets typically = MDL-only
• "Collected" Dematic assets often = dual-shader

Phase 3: Placeholder-to-Asset Mapping

Strategy

1. Group placeholder cubes by name prefix (e.g., CvL001→CvL, BRk045→BRk)
2. For each prefix, find the best-fit asset by:
   • Similar function (racks→rack assets, conveyors→conveyor assets)
   • Compatible size (asset shouldn't massively overshoot the placeholder zone)
   • Dual-shader compatibility (headless rendering requirement)
3. Document the mapping table before building

Mapping Table Format

| Block Prefix | Count | Placeholder Size | Asset | Asset Size | Shader | Notes |
|---|---|---|---|---|---|---|
| BRk | 352 | 3.5×1.2×5.0m | ASRS_Racks_Center | 2.36×2.82×8.29m | dual | Per-block, no scaling |
| CvL | 190 | 2.0×4.0×1.5m | Conveyor_09 | 0.94×6.72×1.66m | dual | Roller conveyor |

Size Philosophy

Use natural asset sizes, NOT scaled-to-cube. Scaling assets to match cube dimensions destroys visual density and realism. Place at the block's XY position with the asset's natural dimensions.

Exception: If an asset is dramatically larger than its zone (e.g., 83m assembly in a 20m zone), use smaller modular pieces instead.

Phase 4: Placement Script Pattern

SimulationApp Runtime Placement

get_asset_bbox(stage, asset_path, app) — reference asset temporarily to get accurate bbox center. collect_blocks(stage) — group visible Cube prims by name prefix with world positions. place_assets(stage, blocks, asset_map, asset_bboxes, module_name) — place assets at block positions with bbox-center offset correction; hides original cubes.

See scripts/place_assets.py.

Hierarchy Convention

/World/
  Module1/          # Racks
    VNA/
      VNA001        # Individual asset reference
      VNA002
    BRk/
      BRk001
  Module2/          # Conveyors, sorters
    CvL/
      CvL001
    Sort/
      Sort001
  Module3/          # Safety, humans

Phase 5: Animation & Articulation

Animation Baking

Use bake_waypoints() from spatial-reasoning skill to animate robots, humans, or objects along paths.

bake_waypoints(xform_op, waypoints, speed_mps, fps, mpu) — keyframe an XformOp along a list of (x,y,z) waypoints. Defined in the spatial-reasoning skill at ../spatial-reasoning/scripts/spatial.py.

Articulation Builder

For robot USD:

build_forklift_articulation(output_path) — create chassis + fixed-joint mast + prismatic lift joint; includes _create_box, _create_fixed_joint, _create_prismatic_joint helpers.

See scripts/articulation_builder.py.

UV Mapping & Materials

For textures:

• Use sphereUV mapping for global assets
• Use linear for flat planes (floors, walls)
• Always use UsdPreviewSurface as fallback
• Never use MDL-only materials for headless

Phase 6: Validation

Color Key for Placeholder Flow

Color	Value	Represents
Red	(1.0, 0.2, 0.2)	Failed validation
Yellow	(1.0, 1.0, 0.2)	Warning (near overlap)
Green	(0.2, 1.0, 0.2)	Valid, ready to replace with asset

Format Validation Checklist

• All assets have .usd, .usda, or .usdc extension
• All paths use forward slashes /
• No .. relative paths
• mpu = 1.0 for all assets (script validates scale)
• No ./ or ../ syntax in references
• Topology: artifacts must not be nested under Xform if empty

Asset Recommendation (Based on Scan)

First, scan all assets with catalog_assets(), then:

• Filter: dual_shader is True
• Sort: prims < 5000
• Assign: Match bbox dimensions within 20% tolerance of placeholder cube
• Reject: All blockpallet_a*, palletstack_a* — these are not real assets

Hard-Won Lessons

1. Never scale assets to match cube dimensions — destroys visual density. Use natural sizes.
2. Large assemblies (>20m) rarely fit block clusters — use smaller modular pieces instead.
3. Always correct for bbox center offset — most assets aren't origin-centered.
4. Lightspeed-processed assets = MDL-only = BLACK on headless arm64. Only use "Collected" dual-shader variants.
5. Tote_01, Pallet_Pile, AMR_Table = MDL-only — they'll place but render black. Use Scissor_Lift or Electrical_Panel as functional placeholders.
6. Kill ALL kit processes before new Isaac Sim launch — zombie processes cause 90-170s cold starts.
7. Clean /dev/shm/carb-* between restarts to prevent SIGKILL.
8. SimulationApp headless requires explicit DomeLight + DistantLight — GUI adds viewport lights automatically, headless does NOT.
9. Xvfb (DISPLAY=:99) can improve MDL rendering over locked desktop (:0), but not a universal fix.