Skip to content

feat(dataset): preserve depth and mask camera frames#422

Open
alectimison-maker wants to merge 3 commits into
DexForce:mainfrom
alectimison-maker:feat/lerobot-depth-mask
Open

feat(dataset): preserve depth and mask camera frames#422
alectimison-maker wants to merge 3 commits into
DexForce:mainfrom
alectimison-maker:feat/lerobot-depth-mask

Conversation

@alectimison-maker

@alectimison-maker alectimison-maker commented Jul 24, 2026

Copy link
Copy Markdown

Description

LeRobotRecorder currently exports RGB frames but drops camera depth and
segmentation-mask observations. This makes the generated dataset incomplete for
policies that consume geometry or semantic labels.

This PR adds first-class numeric LeRobot features for those modalities:

  • observation.depth.<sensor> and observation.mask.<sensor>
  • <sensor>_right variants for stereo-camera frames
  • source dtype and shape preservation (for example, float32 depth and int32
    masks)
  • RGB-only image/video handling, so auxiliary arrays are never passed through a
    lossy image codec
  • a warning and clean skip for camera frame types that the recorder does not
    support

The implementation derives both the feature schema and each recorded frame from
the observation-space frame names. It also removes the separate stereo-camera
branch, ensuring that schemas and frame conversion use the same key mapping.

No new runtime dependency is introduced.

Type of change

  • Enhancement (non-breaking change which improves existing functionality)
  • Documentation update

Screenshots

Not applicable; this change affects structured dataset output.

Validation

  • 11 passed for tests/gym/envs/managers/test_dataset_functors.py in an
    isolated Python 3.10 environment with LeRobot 0.4.4. The test module was
    executed through a minimal import harness because the repository's top-level
    pytest configuration imports the simulation engine before test collection.
  • black --check --target-version py310 embodichain/lab/gym/envs/managers/datasets.py tests/gym/envs/managers/test_dataset_functors.py
  • python -m py_compile for both changed Python files
  • git diff --check

The complete repository test suite could not be collected locally on this
development host:

  • macOS 26.5.1 (arm64)
  • Python 3.10.20
  • pip 26.1.2 and uv 0.11.7

Following the repository installation guide, I retried with:

python -m pip install dexsim_engine==0.4.3 \
  --extra-index-url http://pyp.open3dv.site:2345/simple/ \
  --trusted-host pyp.open3dv.site -v

The DexForce index contains CPython 3.10 and 3.11 wheels for
manylinux_2_31_x86_64, but no macOS arm64 wheel. pip therefore filters all
published files and reports:

ERROR: Could not find a version that satisfies the requirement dexsim_engine==0.4.3 (from versions: none)
ERROR: No matching distribution found for dexsim_engine==0.4.3

This is a local platform-wheel limitation, not an absence of version 0.4.3 from
the DexForce index. The Linux x86_64 CI run remains the authoritative full-suite
check.

Checklist

  • Changed Python files pass Black's check
  • I have made corresponding changes to the documentation
  • I have added tests that prove the enhancement works
  • No dependency update is required

@yuecideng
yuecideng requested review from Copilot and yhnsu July 24, 2026 09:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@yuecideng

Copy link
Copy Markdown
Contributor

Hi @alectimison-maker , thanks for your contribution.

The complete repository test suite could not be collected locally because the
declared dexsim_engine==0.4.3 package is unavailable from the configured
package index.

Could you please provide more details about the issue you encountered, including the environment, installation steps, and relevant error logs? This information would help us reproduce and investigate the problem.

@alectimison-maker

Copy link
Copy Markdown
Author

Thanks for asking—my original wording was imprecise. My initial attempt used the default PyPI index, while EmbodiChain's installation guide requires the DexForce index. I have now retried using the documented index and updated the PR description.

Environment:

  • macOS 26.5.1 (arm64)
  • Python 3.10.20
  • pip 26.1.2
  • uv 0.11.7
  • no global pip index configuration

Reproduction command:

python3.10 -m venv /tmp/embodichain-maintainer-repro
/tmp/embodichain-maintainer-repro/bin/python -m pip install --upgrade pip
/tmp/embodichain-maintainer-repro/bin/python -m pip install dexsim_engine==0.4.3 \
  --extra-index-url http://pyp.open3dv.site:2345/simple/ \
  --trusted-host pyp.open3dv.site -v

Relevant output:

Looking in indexes: https://pypi.org/simple, http://pyp.open3dv.site:2345/simple/
ERROR: Could not find a version that satisfies the requirement dexsim_engine==0.4.3 (from versions: none)
ERROR: No matching distribution found for dexsim_engine==0.4.3

The package index does contain version 0.4.3, but only these platform files:

dexsim_engine-0.4.3-cp310-cp310-manylinux_2_31_x86_64.whl
dexsim_engine-0.4.3-cp311-cp311-manylinux_2_31_x86_64.whl

So the corrected conclusion is that the official version exists, but there is no compatible macOS arm64 wheel for this local host. This is not an issue with the version being absent from the DexForce index. The Linux x86_64 CI build/test jobs are currently queued; I will use those as the authoritative full-environment validation and address any actionable failures.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

tests/gym/envs/managers/test_dataset_functors.py:28

  • The test module overwrites the imported LEROBOT_AVAILABLE flag with True inside the LeRobot import try: block. This makes @pytest.mark.skipif(not LEROBOT_AVAILABLE, ...) ineffective and can cause test failures on installs where lerobot is not present (e.g., patching ...datasets.LeRobotDataset will raise when the symbol is missing).
from tensordict import TensorDict
from unittest.mock import MagicMock, Mock, patch

# Skip all tests if LeRobot is not available
try:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants