Isaac Sim Remote

Purpose

Drive an already-running Isaac Sim over the code-editor TCP socket so agents can iterate on stages, prims, physics, and screenshots without restarting the app.

Security

This skill is a local developer control plane, not a remote-execution service.

• Bind: clients default to 127.0.0.1:8226. Do not publish that port off-host.
• Trust boundary: equal to the OS user who launched Isaac Sim on this machine.
• Design (intentional): isaacsim_send.py delivers Python source to the local code-editor server; execute_command.py dispatches any name already present in omni.kit.commands.get_commands() after an identifier check and audit log. An allowlist would defeat live Kit iteration — treat registry-wide dispatch as the product, not a hole.
• Mitigations: loopback-only default; --dry-run prints payloads without send; CLI --arg values use ast.literal_eval (literals only).

Prerequisites

• Built Isaac Sim ($ISAAC_SIM_DIR or _build/linux-x86_64/release).
• NVIDIA GPU with a current driver (nvidia-smi).
• Shell env contract from isaac-sim-orchestrator: $ISAAC_SIM_DIR, $ISAAC_LAB_DIR, $WORKSPACE_DIR.

Limitations

• Requires Isaac Sim launched with the Python code-editor server enabled.
• Remote scripts cannot import arbitrary repo modules unless paths are injected.

Troubleshooting

Error / symptom	Cause	Solution
Connection refused on 8226	Python server disabled	Launch Isaac Sim with code-editor server enabled
NameError in remote cell	Module not on sys.path	Inline helpers or inject repo paths before import
Stale stage state	Prior remote edits	Reset prims or reopen stage before repro steps

Related: debug-with-local-kit (when behavior depends on a Kit-from-source build), profile-isaac-sim (to attach Tracy to the running process), isaac-sim-validator (final QA gate on any rendered output).

Upstream isaac-sim-ui (menu/widget OmniUIQuery automation) and isaac-sim-recording (cursor tracking, tutorial video capture) are not imported. The inline UI patterns here (Play-button click via OmniUIQuery, full-app vs viewport screenshots) cover the common cases.

Every helper in scripts/ is invoked through scripts/isaacsim_send.py. From agent runtimes that expose skill execution helpers, call e.g. run_script("scripts/app_screenshot.py", args=["--help"]); from a built tree run the file with ./python.sh (Linux) / python.bat (Windows) under _build/*/release. Command recipes live in references/workflows.md.

Available Scripts

Script	Purpose	Arguments
scripts/app_screenshot.py	App screenshot	see script --help
scripts/camera_control.py	Camera control	see script --help
scripts/capture_annotator.py	Capture annotator	see script --help
scripts/console_log.py	Console log	see script --help
scripts/execute_command.py	Dispatch a registered omni.kit.commands entry (loopback trust; see Security)	see script --help
scripts/health_check.py	Health check	see script --help
scripts/isaacsim_send.py	Localhost python_server IPC client (--dry-run supported)	CLI flags via argparse (see script --help)
scripts/open_stage.py	Open stage	see script --help
scripts/prim_properties.py	Prim properties	see script --help
scripts/prim_transform.py	Prim transform	see script --help
scripts/select_prim.py	Select prim	see script --help
scripts/set_asset_root.py	Set asset root	see script --help
scripts/simulation_control.py	Simulation control	see script --help
scripts/stage_info.py	Stage info	see script --help
scripts/verify_asset.py	Verify asset	see script --help
scripts/viewport_screenshot.py	Viewport screenshot	see script --help
scripts/viewport_video.py	Viewport video	see script --help

Launching Isaac Sim

cd _build/linux-x86_64/release
bash isaac-sim.sh --no-window --no-ros-env \
    --enable isaacsim.code_editor.python_server

Wait for app ready. TCP server: 127.0.0.1:8226. Use --enable isaacsim.code_editor.python_server (not carb /enabled=true). Full launch variants and health/asset/send/screenshot/control workflows: references/workflows.md. Pitfalls: references/pitfalls.md. APIs: references/api-reference.md.

Important Notes

• Lighting in headless mode: No lights = black image. Always add a DomeLight with intensity 1000–5000.
• Renderer warm-up: After new stage, call await app_utils.update_app_async(steps=120) before rendering.
• Connection errors: Ensure Isaac Sim shows app ready and python_server is --enabled.
• Server hangs: Crashed examples can wedge the server. Restart Isaac Sim. Use health_check.py to verify.
• Assets not found: Configure S3 fallback with set_asset_root.py.