feat(planner): integrate NVIDIA cuRobo V2 by yuecideng · Pull Request #377 · DexForce/EmbodiChain

yuecideng · 2026-07-11T16:17:03Z

Description

This draft PR adds NVIDIA cuRobo V2 as an optional CUDA collision-aware motion-planning backend for EmbodiChain.

It adds:

Lazy cuRobo V2 planner, profile, world configuration, and named joint mapping support.
MotionGenerator and supported atomic-action routing for end-effector and joint motions.
Batched per-environment collision worlds and dynamic obstacle updates.
Panda DexSim demo, optional offscreen MP4 recorder, documentation, unit tests, CUDA integration tests, and atomic end-to-end coverage.

Motivation: expose GPU-accelerated collision-aware planning through the existing robot, planner, motion-generator, and atomic-action abstractions.

Dependencies: NVIDIA cuRobo V2 is optional and must be installed in a CUDA/PyTorch-compatible environment. This PR adds no mandatory EmbodiChain dependency.

Type of change

Bug fix (non-breaking change which fixes an issue)
Enhancement (non-breaking change which improves an existing functionality)
New feature (non-breaking change which adds functionality)
Breaking change (existing functionality will not work without user modification)
Documentation update

Screenshots

The headless Panda demo can generate an MP4 recording when requested.

Draft note

The initial offscreen recorder samples wall time. Because simulation playback is very fast, a recording can appear nearly static even though trajectory generation and simulation execution are verified. Simulation-time recording is a follow-up before the PR is marked ready for review.

Checklist

I have run the black . command to format the code base.
I have made corresponding changes to the documentation.
I have added tests that prove the feature works.
Optional cuRobo V2 dependency requirements are documented.

Add BasePlanner.preinterpolate_targets / preserve_plan_samples / default_plan_options / with_motion_context so MotionGenerator is capability-driven rather than special-casing NeuralPlanner. Neural and TOPPRA migrate to the new hooks; behavior is preserved. Co-Authored-By: Claude <noreply@anthropic.com>

Add CuroboRobotProfileCfg/CuroboWorldCfg/CuroboPlannerCfg/CuroboPlanOptions and pure helpers (_reorder_by_names, _matrix_to_position_quaternion, _validate_dynamic_obstacles, _require_curobo). The package exports the configs without importing curobo; construction triggers the lazy V2 import. Co-Authored-By: Claude <noreply@anthropic.com>

Implement CuroboPlanner.plan (pose + cspace), backend cache keyed by (control_part, batch_size, multi_env), segment chaining without resampling, failure/over-budget hold, dynamic obstacle updates, and close(). Add fake-binding unit tests (require CUDA) and an optional real V2 integration test skipped without cuRobo/CUDA. Includes the shared cuboid world asset. Co-Authored-By: Claude <noreply@anthropic.com>

ActionCfg validates motion_source/planner_type. TrajectoryBuilder dispatches by capability (preinterpolate_targets / preserve_plan_samples) with strict planner-type matching, result validation, and a new plan_joint_motion path. MoveJoints, PickUp, Place, Press allocate from returned phase lengths; Press returns via plan_joint_motion; PickUp skips the IK prefilter for cuRobo. Coordinated dual-arm primitives reject the cuRobo backend. Co-Authored-By: Claude <noreply@anthropic.com>

Harden batched V2 collision worlds and add the documented atomic-action demo with tests.

Co-authored-by: matafela <chenjian@dexforce.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Pull request overview

Copilot reviewed 35 out of 36 changed files in this pull request and generated 4 comments.

+        raise ImportError(
+            "cuRobo V2 is required for the 'curobo' planner but was not found. "
+            "Install it using NVIDIA's CUDA-matched extras, e.g. "
+            "`pip install .[cu12]` or `pip install .[cu13]` "
+            "(also `.[cu12-torch]` / `.[cu13-torch]`). "
+            f"See {_CUROBO_INSTALL_URL} for details."
+        ) from exc


+import torch
+
+from embodichain.lab.sim.planners import MotionGenOptions, PlanState
+from embodichain.lab.sim.planners.curobo_planner import CuroboPlanOptions


+returned full-DoF trajectory. It disables cuRobo CUDA graph capture by default because graph
+capture can conflict with DexSim GPU physics; pass `--cuda-graph` only after
+validating that the local simulator stream setup supports it. Headless runs


+    sim: SimulationManager | None = None
+    # try:
+    sim, robot, demo_block = _build_scene(args.headless)
+    if not args.headless:


Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Pull request overview

Copilot reviewed 36 out of 37 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

.github/workflows/main.yml:163

The workflow now installs curobo-cu12 for all pull-request test runs, which makes the optional cuRobo backend effectively mandatory for CI and can fail/slow down on runners that don't have a CUDA-capable environment. Consider keeping the base PR test job dependency-free and installing the cuRobo extra only right before the dedicated GPU test step (or in a separate GPU-only job).

      - name: Install test package
        run: |
          pip install -e ".[gensim, curobo-cu12]" \
            --extra-index-url http://pyp.open3dv.site:2345/simple/ \
            --trusted-host pyp.open3dv.site \

docs/source/overview/sim/planners/curobo_planner.md:96

This page references CuroboPlannerCfg.use_cuda_graph, but the cuRobo backend config/tests indicate there is no use_cuda_graph toggle (the subprocess worker always captures graphs). The docs should match the actual public API to avoid misleading users.

`CuroboPlannerCfg.use_cuda_graph` defaults to `False` for the same DexSim GPU
stream-safety reason. Enable it explicitly only after validating the local
simulation stack.

docs/source/overview/sim/planners/curobo_planner.md:215

The demo section mentions --cuda-graph, but the example CLI doesn't define that flag and the backend doesn't expose a use_cuda_graph toggle. Update the demo instructions to reflect the current behavior (graphs enabled in the isolated worker) or add the flag consistently across code/docs.

returned full-DoF trajectory. It disables cuRobo CUDA graph capture by default because graph
capture can conflict with DexSim GPU physics; pass `--cuda-graph` only after
validating that the local simulator stream setup supports it. Headless runs

Copilot

Pull request overview

Copilot reviewed 36 out of 37 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

examples/sim/planners/curobo_planner.py:369

The robot is None guard dereferences robot.uid, which will raise AttributeError and hide the real failure to add the robot.

    if robot is None:
        raise RuntimeError(f"Failed to add robot '{robot.uid}' to the cuRobo demo.")

examples/sim/planners/curobo_planner.py:472

main() no longer wraps simulation setup/execution in a try/finally, so exceptions (including failed planning) will skip sim.destroy() and flush_cleanup_queue(), potentially leaking the simulator process/resources in long-running sessions.

    sim: SimulationManager | None = None
    # try:
    sim, robot, demo_block, target_xpos, control_part = _build_scene(
        args.headless, args.robot
    )

examples/sim/planners/curobo_planner.py:552

The example always blocks on input() even in --headless mode, which prevents non-interactive runs (e.g., CI, remote execution) from completing.

    input("Press Enter to exit the cuRobo demo...")

.github/workflows/main.yml:164

The PR test workflow now unconditionally installs the curobo-cu12 extra. Since this extra is a git-based dependency (NVlabs/curobo) and cuRobo requires a CUDA-capable environment, this can make CPU-only PR CI fail during installation and also undermines the PR’s stated goal of keeping cuRobo optional for the codebase.

      - name: Install test package
        run: |
          pip install -e ".[gensim, curobo-cu12]" \
            --extra-index-url http://pyp.open3dv.site:2345/simple/ \
            --trusted-host pyp.open3dv.site \
            --extra-index-url https://download.blender.org/pypi/

        success = (
            result.success
            if isinstance(result.success, torch.Tensor)
            else torch.tensor(result.success, device=self.device)
        )


…/EmbodiChain into yueci/curobo-integraion

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…/EmbodiChain into yueci/curobo-integraion

Copilot

Pull request overview

Copilot reviewed 37 out of 38 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (6)

embodichain/lab/sim/planners/curobo/curobo_planner.py:436

The missing-cuRobo ImportError message suggests installing .[cu12] / .[cu13], but those extras are for the cuRobo repo, not for EmbodiChain. This is likely to confuse users because EmbodiChain exposes the dependency via curobo-cu12 / curobo-cu13 extras (see pyproject/docs/example).

        raise ImportError(
            "cuRobo V2 is required for the 'curobo' planner but was not found. "
            "Install it using NVIDIA's CUDA-matched extras, e.g. "
            "`pip install .[cu12]` or `pip install .[cu13]` "
            "(also `.[cu12-torch]` / `.[cu13-torch]`). "
            f"See {_CUROBO_INSTALL_URL} for details."
        ) from exc

docs/source/overview/sim/planners/curobo_planner.md:96

This doc references CuroboPlannerCfg.use_cuda_graph, but the implementation runs cuRobo in a subprocess worker with use_cuda_graph=True unconditionally and does not expose a use_cuda_graph config field. The docs should reflect the actual behavior/API to avoid misleading configuration guidance.

`CuroboPlannerCfg.use_cuda_graph` defaults to `False` for the same DexSim GPU
stream-safety reason. Enable it explicitly only after validating the local
simulation stack.

docs/source/overview/sim/planners/curobo_planner.md:215

The demo docs say CUDA graph capture is disabled by default and mention a --cuda-graph CLI flag, but the example CLI does not define that flag and the backend worker captures CUDA graphs by design. This section should be updated to match the current CLI and backend behavior.

auto-generated), prints the result status and trajectory shape, then replays the
returned full-DoF trajectory. It disables cuRobo CUDA graph capture by default because graph
capture can conflict with DexSim GPU physics; pass `--cuda-graph` only after
validating that the local simulator stream setup supports it. Headless runs

.github/workflows/main.yml:164

The PR description says cuRobo is optional and adds no mandatory dependency, but the PR CI workflow now installs the curobo-cu12 extra for all pull_request test runs. This makes the optional dependency effectively required in CI and may introduce install-time failures or large slowdowns on runners that don't support cuRobo.

      - name: Install test package
        run: |
          pip install -e ".[gensim, curobo-cu12]" \
            --extra-index-url http://pyp.open3dv.site:2345/simple/ \
            --trusted-host pyp.open3dv.site \
            --extra-index-url https://download.blender.org/pypi/

examples/sim/planners/curobo_planner.py:552

input() will block indefinitely in headless/non-interactive runs (including automated environments), preventing the demo from exiting and writing recordings/cleaning up. Gate the prompt on interactive mode (or at least not args.headless).

    input("Press Enter to exit the cuRobo demo...")

examples/sim/planners/curobo_planner.py:472

The demo has a commented-out try: and no finally cleanup. If anything raises after the sim is created, the window recorder and simulation may not be stopped/destroyed cleanly. Consider restoring a try/finally around the main body so cleanup always runs.

    sim: SimulationManager | None = None
    # try:
    sim, robot, demo_block, target_xpos, control_part = _build_scene(
        args.headless, args.robot
    )

…/EmbodiChain into yueci/curobo-integraion

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

yuecideng and others added 7 commits July 11, 2026 10:43

docs: add cuRobo V2 integration design

9629eb7

wip

62f9edc

feat(planner): finalize curobo v2 integration

19000f8

Harden batched V2 collision worlds and add the documented atomic-action demo with tests.

yuecideng added enhancement New feature or request solver Robot kinematics solver motion gen Things related to motion generation for robot atomic action atomic action related functionality dexsim Things related to dexsim docs Improvements or additions to documentation labels Jul 11, 2026

yuecideng requested a review from chase6305 July 12, 2026 03:50

yuecideng requested a review from matafela July 20, 2026 02:18

yuecideng and others added 2 commits July 23, 2026 16:47

Merge branch 'main' into yueci/curobo-integraion

3e01ebb

update curobo integration (#416)

d6c66ef

Co-authored-by: matafela <chenjian@dexforce.com>

yuecideng marked this pull request as ready for review July 23, 2026 08:58

Copilot AI review requested due to automatic review settings July 23, 2026 08:58

Copilot AI reviewed Jul 23, 2026

fix example

113c3c6

Copilot AI review requested due to automatic review settings July 23, 2026 10:25

Copilot started reviewing on behalf of matafela July 23, 2026 10:25 View session

fix ci

8fc00df

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 23, 2026 10:30

Copilot AI reviewed Jul 23, 2026

fix example

099e562

Copilot AI review requested due to automatic review settings July 23, 2026 14:49

Copilot started reviewing on behalf of matafela July 23, 2026 14:50 View session

Copilot started reviewing on behalf of matafela July 24, 2026 08:49 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread examples/sim/planners/curobo_planner.py Outdated

use sphere fit as default collision model

4ff778a

Copilot AI review requested due to automatic review settings July 24, 2026 11:08

Copilot started reviewing on behalf of matafela July 24, 2026 11:08 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread embodichain/lab/sim/atomic_actions/trajectory.py

Comment on lines 431 to 435

success = (

result.success

if isinstance(result.success, torch.Tensor)

else torch.tensor(result.success, device=self.device)

)

yuecideng added 2 commits July 24, 2026 15:18

wip

eaf0748

Merge branch 'yueci/curobo-integraion' of https://github.com/DexForce…

5626952

…/EmbodiChain into yueci/curobo-integraion

Copilot AI review requested due to automatic review settings July 24, 2026 15:19

Copilot AI reviewed Jul 24, 2026

matafela added 2 commits July 24, 2026 23:39

fix hand collision generation bug

b6731cb

Merge branch 'yueci/curobo-integraion' of https://github.com/DexForce…

4f4c5c3

…/EmbodiChain into yueci/curobo-integraion

Copilot AI review requested due to automatic review settings July 24, 2026 15:40

Copilot started reviewing on behalf of matafela July 24, 2026 15:40 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

yuecideng added 2 commits July 25, 2026 15:16

wip

82da3d7

Merge branch 'yueci/curobo-integraion' of https://github.com/DexForce…

e76d006

…/EmbodiChain into yueci/curobo-integraion

Copilot AI review requested due to automatic review settings July 25, 2026 07:16

Copilot AI reviewed Jul 25, 2026

wip

f0a984f

Copilot AI review requested due to automatic review settings July 25, 2026 15:38

Copilot AI reviewed Jul 25, 2026

wip

ea3c1fa

Copilot AI review requested due to automatic review settings July 26, 2026 15:24

Copilot AI reviewed Jul 26, 2026

wip

f763af7

Copilot AI review requested due to automatic review settings July 26, 2026 15:48

Copilot AI reviewed Jul 26, 2026

yuecideng merged commit 25ef706 into main Jul 27, 2026
4 checks passed

yuecideng deleted the yueci/curobo-integraion branch July 27, 2026 00:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(planner): integrate NVIDIA cuRobo V2#377

feat(planner): integrate NVIDIA cuRobo V2#377
yuecideng merged 26 commits into
mainfrom
yueci/curobo-integraion

yuecideng commented Jul 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

yuecideng commented Jul 11, 2026

Description

Type of change

Screenshots

Draft note

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants