cyy/Motrixlab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: release v0.1.0

Browse Source 
(cherry picked from commit 82525f882f3924a332d9ce40bf64255d0d14f6a4)
This commit is contained in:
motphys-developers
2026-01-04 04:43:04 +00:00
parent 13cfbce9a7
commit 62011bb24f
390 changed files with 18897 additions and 626 deletions
Download Patch File Download Diff File Expand all files Collapse all files

126
docs/source/en/index.md
View File

@@ -35,6 +35,76 @@ MotrixLab is a general-purpose machine learning architecture designed for robot
:::{grid-item-card}
```{video} _static/videos/go1_rough_terrain_walk.mp4
:poster: _static/images/poster/go1_rough_terrain_walk.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/go1_stairs_terrain_walk.mp4
:poster: _static/images/poster/go1_stairs_terrain_walk.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/franka_lift_cube.mp4
:poster: _static/images/poster/franka_lift_cube.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/franka_open_cabinet.mp4
:poster: _static/images/poster/franka_open_cabinet.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/anymal_c.mp4
:poster: _static/images/poster/anymal_c.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/dm_walker.mp4
:poster: _static/images/poster/dm_walker.jpg
:nocontrols:
@@ -45,6 +115,62 @@ MotrixLab is a general-purpose machine learning architecture designed for robot
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/dm_cheetah.mp4
:poster: _static/images/poster/dm_cheetah.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/dm_hopper.mp4
:poster: _static/images/poster/dm_hopper.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/dm_reacher.mp4
:poster: _static/images/poster/dm_reacher.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
:::{grid-item-card}
```{video} _static/videos/bounce_ball.mp4
:poster: _static/images/poster/bounce_ball.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
:::
::::

316
docs/source/en/user_guide/demo/anymal_c.md Normal file
View File

@@ -0,0 +1,316 @@
# ANYmal-C Locomotion
## Overview
This document describes in detail the navigation task environment based on the ANYmal-C quadruped robot. This environment is part of the navigation task collection in the MotrixLab project, providing a complete implementation for training quadruped robots to navigate to target positions and orientations using reinforcement learning.
```{video} /_static/videos/anymal_c.mp4
:poster: _static/images/poster/anymal_c.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
---
## Environment Description
The ANYmal-C navigation task environment is built based on the real ANYmal-C quadruped robot, designed to train robots to navigate to specified target positions and orientations on flat terrain. This environment uses the MotrixSim physics engine for simulation, providing high-fidelity dynamic simulation.
### Robot Structure
ANYmal-C is a quadruped robot composed of the following main parts:
- **Base**: The core torso of the robot, containing sensor modules such as IMU, camera, and lidar
- **Four Legs**: Each leg contains three joints
- HAA (Hip Abduction/Adduction): Hip abduction/adduction joint
- HFE (Hip Flexion/Extension): Hip flexion/extension joint
- KFE (Knee Flexion/Extension): Knee flexion/extension
- **Four Feet**: Spherical contact geometries that generate frictional contact with the ground
### Task Objective
The robot needs to complete the following navigation objectives:
1. **Position Navigation**: Move to the specified target position (XY plane coordinates)
2. **Orientation Control**: Adjust robot orientation to the target heading angle (yaw angle)
3. **Stable Stop**: Maintain stable standing after reaching the target, with linear and angular velocities approaching zero
The environment provides visualization markers:
- **Green Arrow**: Indicates target position and orientation
- **Green Arrow Above Robot**: Current actual movement direction
- **Blue Arrow Above Robot**: Desired movement direction
---
## Action Space
The action space is `Box(-1.0, 1.0, (12,), float32)`, representing position control commands applied to 12 joints (offsets relative to the default standing posture).
### Control Mode
The environment uses position control mode. Actions are converted to joint target positions as follows:
```
Target Joint Angle = Default Joint Angle + (Action Value × Action Scale)
```
Where the action scale is specified by the configuration parameter `control_config.action_scale`.
### Action Dimension Details
| Index | Action Description | Control Range | Joint Name | Joint Type |
| ----- | -------------------- | ------------- | ---------- | ---------- |
| 0 | Left Front Hip HAA | -1.0 ~ 1.0 | LF_HAA | hinge |
| 1 | Left Front Hip HFE | -1.0 ~ 1.0 | LF_HFE | hinge |
| 2 | Left Front Knee KFE | -1.0 ~ 1.0 | LF_KFE | hinge |
| 3 | Right Front Hip HAA | -1.0 ~ 1.0 | RF_HAA | hinge |
| 4 | Right Front Hip HFE | -1.0 ~ 1.0 | RF_HFE | hinge |
| 5 | Right Front Knee KFE | -1.0 ~ 1.0 | RF_KFE | hinge |
| 6 | Left Hind Hip HAA | -1.0 ~ 1.0 | LH_HAA | hinge |
| 7 | Left Hind Hip HFE | -1.0 ~ 1.0 | LH_HFE | hinge |
| 8 | Left Hind Knee KFE | -1.0 ~ 1.0 | LH_KFE | hinge |
| 9 | Right Hind Hip HAA | -1.0 ~ 1.0 | RH_HAA | hinge |
| 10 | Right Hind Hip HFE | -1.0 ~ 1.0 | RH_HFE | hinge |
| 11 | Right Hind Knee KFE | -1.0 ~ 1.0 | RH_KFE | hinge |
### PD Control Parameters
The underlying system uses position actuators with PD control parameters defined in the XML file:
- **kp (Proportional Gain)**: 200
- **kv (Derivative Gain)**: 1
- **Torque Limit**: -140 N·m ~ 140 N·m
---
## Observation Space
The observation space is `Box(-inf, inf, (54,), float32)`, containing the robot's proprioceptive information, task-related information, and action history.
### Observation Components
The observation vector consists of the following parts (in order):
1. **Proprioceptive State (33 dimensions)**
- Base Linear Velocity (3 dim): Linear velocity of robot base in world frame [vx, vy, vz]
- Angular Velocity (3 dim): Angular velocity from gyroscope [ωx, ωy, ωz]
- Projected Gravity (3 dim): Gravity vector projected in robot body frame
- Joint Angles (12 dim): 12 joint angle offsets relative to default standing posture
- Joint Velocities (12 dim): 12 joint angular velocities
2. **Action History (12 dimensions)**
- Action executed at previous timestep
3. **Velocity Commands (3 dimensions)**
- Desired Linear Velocity XY (2 dim): Desired linear velocity calculated from position error
- Desired Angular Velocity Z (1 dim): Desired angular velocity calculated from orientation error
4. **Task State (6 dimensions)**
- Position Error Vector (2 dim): XY plane error vector to target position (normalized)
- Orientation Error (1 dim): Angle difference to target orientation (normalized to [-1, 1])
- Distance (1 dim): Euclidean distance to target (normalized)
- Arrival Flag (1 dim): Whether both position and orientation arrival conditions are satisfied (0 or 1)
- Stop Ready Flag (1 dim): Whether stop criteria are met (arrived and angular velocity near zero)
### Observation Details
| Index | Observation Content | Min | Max | Normalization Coefficient | Unit |
| ----- | ------------------------------------------ | ---- | --- | ------------------------- | ------------- |
| 0-2 | Base Linear Velocity (vx, vy, vz) | -inf | inf | normalization.lin_vel | m/s |
| 3-5 | Angular Velocity (ωx, ωy, ωz) | -inf | inf | normalization.ang_vel | rad/s |
| 6-8 | Projected Gravity (gx, gy, gz) | -1 | 1 | 1.0 | Dimensionless |
| 9-20 | Joint Angle Offsets (12 joints) | -inf | inf | normalization.dof_pos | rad |
| 21-32 | Joint Angular Velocities (12 joints) | -inf | inf | normalization.dof_vel | rad/s |
| 33-44 | Previous Action | -1 | 1 | 1.0 | Dimensionless |
| 45-47 | Velocity Commands (vx_cmd, vy_cmd, ωz_cmd) | -inf | inf | commands_scale | m/s, rad/s |
| 48-49 | Position Error Vector (Δx, Δy) | -inf | inf | 1/5.0 | m |
| 50 | Orientation Error | -1 | 1 | 1/π | rad |
| 51 | Distance to Target | 0 | 1 | 1/5.0 (clipped) | m |
| 52 | Arrival Flag | 0 | 1 | 1.0 | Boolean |
| 53 | Stop Ready Flag | 0 | 1 | 1.0 | Boolean |
### Sensor Information
The environment uses the following sensors to obtain state:
- **framelinvel** (name: base_linvel): Base linear velocity sensor
- **gyro** (name: base_gyro): Gyroscope sensor, mounted at IMU site
---
## Reward Function
The reward function uses a composite design, employing different reward strategies based on whether the robot has reached the target.
### Rewards Before Reaching Target
Total Reward = Velocity Tracking Reward + Approach Reward - Penalty Terms
**Main Reward Terms:**
1. **Linear Velocity Tracking Reward** (Weight: 1.5)
- Formula: `1.5 × exp(-||v_xy - v_cmd||² / 0.25)`
- Encourages robot to track desired XY plane linear velocity
2. **Angular Velocity Tracking Reward** (Weight: 0.3)
- Formula: `0.3 × exp(-(ωz - ωz_cmd)² / 0.25)`
- Encourages robot to track desired yaw angular velocity
3. **Approach Reward**
- Formula: `clip((Historical Minimum Distance - Current Distance) × 4.0, -1.0, 1.0)`
- Rewards robot for progress when getting closer to target
**Penalty Terms:**
- Z-axis Linear Velocity Penalty (Weight: 2.0): `-2.0 × vz²`
- XY-axis Angular Velocity Penalty (Weight: 0.05): `-0.05 × (ωx² + ωy²)`
- Torque Penalty (Weight: 0.00001): `-0.00001 × ||τ||²`
- Action Rate Penalty (Weight: 0.001): `-0.001 × ||Δa||²`
### Rewards After Reaching Target
Total Reward = Stop Reward + First Arrival Reward - Penalty Terms
**Main Reward Terms:**
1. **Stop Base Reward**
- Formula: `2 × [0.8 × exp(-(v_xy/0.2)²) + 1.2 × exp(-(ωz/0.1)⁴)]`
- Encourages robot to maintain low velocity and angular velocity after arrival
2. **Zero Angular Velocity Reward** (Extra Reward: 6.0)
- Condition: Arrived at target and |ωz| < 0.05 rad/s
- Encourages robot to completely stop rotation
3. **First Arrival Reward** (One-time: 10.0)
- Condition: First time both position and orientation arrival conditions are satisfied
- Provides clear signal for reaching target
**Penalty Terms:** (Same as before arrival)
### Termination Condition Penalties
An additional penalty of -20.0 is applied in the following cases:
- Joint velocity exceeds limit (exceeds `max_dof_vel` configuration value)
- Joint velocity is NaN or Inf
- Robot base contacts ground
- Robot rollover (tilt angle exceeds 75°)
### Arrival Criteria
- **Position Arrival**: Distance to target < 0.3 meters
- **Orientation Arrival**: Orientation error < 15°
- **Complete Arrival**: Both position arrival and orientation arrival conditions satisfied
- **Stop Ready**: Complete arrival AND |ωz| < 0.05 rad/s
### info Return Content
The info dictionary returned each step contains the following debug information:
- `pose_commands`: Current target position and orientation [x, y, yaw]
- `last_actions`: Previous action
- `current_actions`: Current action
- `steps`: Current episode step count
- `ever_reached`: Whether target has ever been reached
- `min_distance`: Historical minimum distance (used to calculate approach reward)
---
## Initial State
### Robot Initialization
**Position Initialization:**
The robot's initial position in world coordinates is randomly sampled within the range defined by configuration parameter `init_state.pos_randomization_range`:
- X coordinate: Uniform random sampling in [x_min, x_max]
- Y coordinate: Uniform random sampling in [y_min, y_max]
- Z coordinate: Fixed at 0.56 meters (to avoid falling sensation)
**Orientation Initialization:**
- Robot orientation (quaternion): Initialized to unit quaternion [0, 0, 0, 1], indicating forward orientation
- No random noise added to quaternion (ensures initial stability)
**Joint Initialization:**
Joint angles are set to default standing posture, defined by configuration parameter `init_state.default_joint_angles`. No random noise added to joint angles (ensures stable standing initially).
**Velocity Initialization:**
All linear and angular velocities are initialized to zero, ensuring robot starts from stationary state.
### Target Generation
**Target Position:**
Target position is generated relative to robot's initial position:
```
Target Position = Robot Initial Position + Random Offset
```
Random offset is sampled within the range defined by configuration parameter `commands.pose_command_range`:
- X direction offset: [pose_command_range[0], pose_command_range[3]]
- Y direction offset: [pose_command_range[1], pose_command_range[4]]
**Target Orientation:**
Target orientation (yaw angle) is randomly generated in absolute reference frame:
- Orientation angle: [pose_command_range[2], pose_command_range[5]]
### Visualization Marker Initialization
- **Target Marker** (green arrow): Set to target position and orientation
- **Movement Direction Arrows**: Initialized at 0.76 meters above robot
---
## Episode Termination
### Termination Conditions
The environment terminates an episode when any of the following conditions are met:
1. **Timeout Termination**
- Condition: Episode reaches `max_episode_second` configuration value
- Description: Prevents infinite episodes
2. **Joint Velocity Anomaly**
- Condition: Absolute value of any joint velocity exceeds `max_dof_vel`
- Condition: Joint velocity is NaN, Inf, or exceeds 1e6
- Description: Prevents numerical divergence and physical instability
3. **Base Contacts Ground**
- Condition: Robot base (geometries defined by `terminate_after_contacts_on` configuration parameter) contacts ground
- Description: Robot fell or pose failure
4. **Rollover**
- Condition: Robot tilt angle exceeds 75°
- Calculation: Tilt angle calculated via projected gravity vector `arctan2(||g_xy||, |g_z|)`
- Description: Robot severely rolled over
### Success Conditions
Although the environment does not terminate upon success, task success is defined as:
- Robot reaches target position and orientation (both position threshold < 0.3m and orientation threshold < 15° satisfied)
- Robot maintains stable stop (linear velocity < 0.05 m/s, angular velocity < 0.05 rad/s)

130
docs/source/en/user_guide/demo/bounce_ball.md Normal file
View File

@@ -0,0 +1,130 @@
# Ping Pong Ball Bouncing
Train a single-arm robotic manipulator to control a paddle for continuous ball bouncing, maintaining the ball at a target height and position.
```{video} /_static/videos/bounce_ball.mp4
:poster: _static/images/poster/bounce_ball.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
## Task Description
Bounce Ball is a single-arm robotic manipulation task using a 6-DOF Peitian AIR4-560 industrial robotic arm to control the position of an end-effector paddle. The agent controls the position changes of the arm's 6 joints as actions, making the ping pong ball bounce continuously on the paddle and keeping it as close as possible to the target height and target horizontal position.
---
## Action Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-1.0, 1.0, (6,), float32)` |
| **Dimension** | 6 |
The joints correspond as follows:
| Index | Action Meaning (Joint Position Change) | Min Value | Max Value | Corresponding XML Name |
| ----: | --------------------------------------- | :-------: | :-------: | :--------------------: |
| 0 | Joint1 (Base Rotation) Position Change | -1 | 1 | `Joint1` |
| 1 | Joint2 (Upper Arm) Position Change | -1 | 1 | `Joint2` |
| 2 | Joint3 (Forearm) Position Change | -1 | 1 | `Joint3` |
| 3 | Joint4 (Wrist Rotation) Position Change | -1 | 1 | `Joint4` |
| 4 | Joint5 (Wrist Pitch) Position Change | -1 | 1 | `Joint5` |
| 5 | Joint6 (Wrist Rotation) Position Change | -1 | 1 | `Joint6` |
---
## Observation Space
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (25,), float32)` |
| **Dimension** | 25 |
The observation space consists of the following parts (in order):
| Part | Content Description | Dimension | Remarks |
| ----------- | ----------------------------------------------- | --------- | -------------------------------------------------------------------------------- |
| **dof_pos** | Position information for each degree of freedom | 13 | First 6 are arm joints, last 7 are ball's free joint (3 position + 4 quaternion) |
| **dof_vel** | Velocity information for each degree of freedom | 12 | Velocity is derivative of position |
| Index | Observation | Min Value | Max Value | XML Name | Type (Unit) |
| ----- | --------------------------- | --------- | --------- | ----------- | ------------------------- |
| 0-5 | Arm Joint Angles | -Inf | Inf | Joint1-6 | Angle (rad) |
| 6 | Ball x-coordinate | -Inf | Inf | ball_x | Position (m) |
| 7 | Ball y-coordinate | -Inf | Inf | ball_y | Position (m) |
| 8 | Ball z-coordinate | -Inf | Inf | ball_z | Position (m) |
| 9-12 | Ball Orientation Quaternion | -Inf | Inf | ball_qw/xyz | Quaternion (w,x,y,z) |
| 13-24 | Joint and Ball Velocities | -Inf | Inf | - | Velocity/Angular Velocity |
---
## Reward Function Design
The reward function consists of the following components:
```python
# Position Control Reward: Keep ball above paddle center
# Controlled Upward Velocity Reward: Reward moderate upward velocity when ball is well-positioned
# Height Accuracy Reward: Ball is close to target height
# Consecutive Bounces Reward: Reward consecutive successful bounces
# Total Reward = Weighted combination of all components
```
---
## Initial State
- **Arm Initial Position**: [0, 40, 110, 0, -60, 0] degrees, with random noise
- **Ball Initial Position**: Above paddle center, with random noise
- **Ball Initial Velocity**: [0.0, 0.0, 0.0] m/s
---
## Episode Termination Conditions
- **Ball Falls**: Ball z-coordinate < 0.05m (near ground)
- **Ball Too High**: Ball z-coordinate > target height + 1.0m (lost control)
- **Horizontal Deviation Too Far**: Absolute value of ball x-coordinate > 1.5m
---
## Usage Guide
### 1. Environment Preview
```bash
uv run scripts/view.py --env bounce_ball
```
### 2. Start Training
```bash
uv run scripts/train.py --env bounce_ball
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/bounce_ball
```
### 4. Test Training Results
```bash
uv run scripts/play.py --env bounce_ball
```
---
## Expected Training Results
1. Consecutive Bouncing: Capable of achieving 3 or more consecutive bounces
2. Position Control: Ball's horizontal position (x-coordinate) stable within target position ± 0.05m range
3. Height Control: Ball's height stable within target height 0.8 ± 0.1m range
4. Velocity Control: Ball's upward velocity maintained within reasonable range (0.1-1.5 m/s)
5. Stable Control: Capable of maintaining stable bouncing for 20 seconds without dropping

61
docs/source/en/user_guide/demo/cartpole.md
View File

@@ -1,4 +1,4 @@
# CartPole Training Example
# CartPole
CartPole is a classic control task in reinforcement learning. The goal is to keep the pole balanced by controlling the cart's left-right movement.
![cartpole](/_static/images/poster/cartpole.jpg)
@@ -21,14 +21,7 @@ uv run scripts/view.py --env cartpole
### 2. Start Training
```bash
# Train with default parameters
uv run scripts/train.py --env cartpole
# Customize number of environments
uv run scripts/train.py --env cartpole --num-envs 1024
# Enable rendering (visualize during training)
uv run scripts/train.py --env cartpole --render
```
### 3. View Training Progress
@@ -40,63 +33,11 @@ uv run tensorboard --logdir runs/cartpole
### 4. Test Training Results
```bash
# Automatically find best policy for testing (recommended)
uv run scripts/play.py --env cartpole
# Manually specify policy file for testing
uv run scripts/play.py --env cartpole --policy runs/cartpole/nn/best_policy.pickle
```
> **Tip**: The system will automatically find the latest and best policy files in the `runs/cartpole/` directory for testing. You can also manually specify specific policy files using the `--policy` parameter.
## Configuration Parameters
Main configuration parameters for the CartPole environment:
```python
@dataclass
class CartPoleEnvCfg(EnvCfg):
model_file: str = "path/to/inverted_pendulum.xml" # MJCF model file
reset_noise_scale: float = 0.01 # Reset noise
max_episode_seconds: float = 10.0 # Maximum episode length
```
Training configuration parameters:
```python
from dataclasses import dataclass
from motrix_rl.skrl.cfg import PPOCfg
from motrix_rl import registry
@registry.rlcfg("cartpole")
@dataclass
class CartPolePPO(PPOCfg):
max_env_steps: int = 10_000_000 # Maximum environment steps
check_point_interval: int = 500 # Checkpoint interval
# Network structure (small network suitable for simple tasks)
policy_hidden_layer_sizes: tuple[int, ...] = (32, 32)
value_hidden_layer_sizes: tuple[int, ...] = (32, 32)
# PPO parameters
rollouts: int = 32 # Experience replay rounds
learning_epochs: int = 5 # Training rounds
mini_batches: int = 4 # Number of mini-batches
```
**Note**: CartPole is a simple task and currently uses universal configuration. If you need to create specialized configurations for different training backends (JAX/Torch), refer to the environment configuration documentation examples.
## Custom Training
You can override default configurations through command line arguments:
```bash
uv run scripts/train.py --env cartpole \
--num-envs 1024 \
--train-backend jax \
--sim-backend np
```
## Expected Results
- Pole angle stays within ±5 degrees most of the time

131
docs/source/en/user_guide/demo/dm_cheetah.md Normal file
View File

@@ -0,0 +1,131 @@
# Half-Cheetah Robot
The Half-Cheetah robot is a classic continuous control task in the DeepMind Control Suite. The goal is to train a simulated bipedal robot to run at high speed and stably by controlling its joint torques.
```{video} /_static/videos/dm_cheetah.mp4
:poster: _static/images/poster/dm_cheetah.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
## Task Description
HalfCheetah is a 2D half-cheetah running task, composed of 7 main body parts (1 torso and 3 sections for each of the front and rear legs), with 6 controlled joints (front and rear thighs [connected to the torso], shins [connected to the thighs], and feet [connected to the shins]). The agent applies torques to these joints as actions, aiming to make the cheetah run forward as fast and stably as possible.
---
## Action Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-1.0, 1.0, (6,), float32)` |
| **Dimension** | 6 |
The joints correspond as follows:
| Index | Action Meaning (Torque applied to the joint) | Min Value | Max Value | Corresponding XML Name |
| ----: | -------------------------------------------- | :-------: | :-------: | :--------------------: |
| 0 | Rear Thigh Joint Drive Torque | -1 | 1 | `bthigh` |
| 1 | Rear Shin Joint Drive Torque | -1 | 1 | `bshin` |
| 2 | Rear Foot Joint Drive Torque | -1 | 1 | `bfoot` |
| 3 | Front Thigh Joint Drive Torque | -1 | 1 | `fthigh` |
| 4 | Front Shin Joint Drive Torque | -1 | 1 | `fshin` |
| 5 | Front Foot Joint Drive Torque | -1 | 1 | `ffoot` |
---
## Observation Space
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (17,), float32)` |
| **Dimension** | 17 |
The observation space of the HalfCheetah environment consists of the following parts (in order):
| Part | Content Description | Dimension | Remarks |
| -------- | ------------- | -- | ------------ |
| **qpos** | Position information of each body joint and the root | 8 | Root x-coordinate is excluded by default |
| **qvel** | Velocity information of each body joint and the root | 9 | Velocity is the derivative of position |
| Index | Observation | Min Value | Max Value | XML Name | Joint Type | Type (Unit) |
| -------- | ---------------------------- | --------- | --------- | -------- | ---------- | ------------------------ |
| 0 | Front z-coordinate | -Inf | Inf | rootz | slide | Position (m) |
| 1 | Front angle | -Inf | Inf | rooty | hinge | Angle (rad) |
| 2 | Rear Thigh Angle | -Inf | Inf | bthigh | hinge | Angle (rad) |
| 3 | Rear Shin Angle | -Inf | Inf | bshin | hinge | Angle (rad) |
| 4 | Rear Foot Angle | -Inf | Inf | bfoot | hinge | Angle (rad) |
| 5 | Front Thigh Angle | -Inf | Inf | fthigh | hinge | Angle (rad) |
| 6 | Front Shin Angle | -Inf | Inf | fshin | hinge | Angle (rad) |
| 7 | Front Foot Angle | -Inf | Inf | ffoot | hinge | Angle (rad) |
| 8 | Front x-coordinate Velocity | -Inf | Inf | rootx | slide | Velocity (m/s) |
| 9 | Front z-coordinate Velocity | -Inf | Inf | rootz | slide | Velocity (m/s) |
| 10 | Front Angular Velocity | -Inf | Inf | rooty | hinge | Angular Velocity (rad/s) |
| 11 | Rear Thigh Angular Velocity | -Inf | Inf | bthigh | hinge | Angular Velocity (rad/s) |
| 12 | Rear Shin Angular Velocity | -Inf | Inf | bshin | hinge | Angular Velocity (rad/s) |
| 13 | Rear Foot Angular Velocity | -Inf | Inf | bfoot | hinge | Angular Velocity (rad/s) |
| 14 | Front Thigh Angular Velocity | -Inf | Inf | fthigh | hinge | Angular Velocity (rad/s) |
| 15 | Front Shin Angular Velocity | -Inf | Inf | fshin | hinge | Angular Velocity (rad/s) |
| 16 | Front Foot Angular Velocity | -Inf | Inf | ffoot | hinge | Angular Velocity (rad/s) |
| excluded | Front x-coordinate | -Inf | Inf | rootx | slide | Position (m) |
---
## Reward Function Design
The cheetah's reward function consists of the following parts:
```python
# Velocity Reward: Tracking target speed
# Posture Reward: Maintaining a stable posture
# Total Reward = Velocity Reward + Posture Reward
```
---
## Initial State
- Reset all finite joint angles to random values within their allowed ranges, keeping infinite range joints in their default state.
- Generate the initial observation vector by stabilizing the torso and leg positions through multi-step physics simulation.
## Episode Termination Conditions
- **No Fall Termination Condition** (Does not end directly due to instability)
---
## Usage Guide
### 1. Environment Preview
```bash
uv run scripts/view.py --env dm-cheetah
```
### 2. Start Training
```bash
uv run scripts/train.py --env dm-cheetah
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/dm-cheetah
```
### 4. Test Training Results
```bash
uv run scripts/play.py --env dm-cheetah
```
---
## Expected Training Results
1. Stable horizontal speed approaching or exceeding 30.0 m/s
2. Maintaining an upright torso and coordinated gait

140
docs/source/en/user_guide/demo/dm_hopper.md Normal file
View File

@@ -0,0 +1,140 @@
# Single-Leg Hopping Robot
Hopper is a classic single-leg hopping control task in dm-control, simulating a 2D single-leg hopping robot.
```{video} /_static/videos/dm_hopper.mp4
:poster: _static/images/poster/dm_hopper.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
## Task Description
The 2D robot consists of four body segments: torso, pelvis, thigh, calf, and foot. Actions are generated through four articulated joints, including: waist, hip, knee, and ankle. Each joint is driven by a motor with different gear ratios, enabling behaviors such as standing, balancing, and hopping forward.
---
## Action Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-1.0, 1.0, (3,), float32)` |
| **Dimension** | 3 |
Joint mapping:
| Index | Action Description | Min | Max | XML Name |
| ----- | --------------------------------- | --- | --- | ----------- |
| 0 | Torque applied on the thigh rotor | -1 | 1 | thigh_joint |
| 1 | Torque applied on the leg rotor | -1 | 1 | leg_joint |
| 2 | Torque applied on the foot rotor | -1 | 1 | foot_joint |
---
## Observation Space
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (13,), float64)` |
| **Dimension** | 13 |
| Component | Description | Dim | Notes |
| ------------------- | ----------------------------- | --- | ------------------------------------ |
| **qpos** | Joint angles and torso height | 5 | torso x-position excluded by default |
| **qvel** | Joint and torso velocities | 6 | Velocity as derivative of position |
| **contact sensors** | Toe and heel ground sensors | 2 | Normalized by `log1p` |
The observation vector consists of joint positions (qpos), velocities (qvel), and contact force sensors. The full dimension is 13, including two contact sensors:
| Index | Observation | XML Name | Joint Type | Physical Meaning |
| ----: | ---------------------------- | ------------- | ---------- | ------------------------- |
| 0 | torso z-position | `rootz` | slide | torso height |
| 1 | torso angle | `rooty` | hinge | body pitch angle |
| 2 | thigh joint angle | `thigh_joint` | hinge | thigh rotation |
| 3 | leg joint angle | `leg_joint` | hinge | calf rotation |
| 4 | foot joint angle | `foot_joint` | hinge | foot rotation |
| 5 | torso x-velocity | `rootx` | slide | forward velocity |
| 6 | torso z-velocity | `rootz` | slide | vertical velocity |
| 7 | torso angular velocity | `rooty` | hinge | torso angular velocity |
| 8 | thigh joint angular velocity | `thigh_joint` | hinge | thigh angular velocity |
| 9 | leg joint angular velocity | `leg_joint` | hinge | calf angular velocity |
| 10 | foot joint angular velocity | `foot_joint` | hinge | foot angular velocity |
| 11 | toe touch sensor | `touch_toe` | sensor | toe ground contact force |
| 12 | heel touch sensor | `touch_heel` | sensor | heel ground contact force |
---
## Reward Function Design
The Hopper reward consists of the following terms:
### Stand Task
```python
# Stand reward: maintain stable height
```
### Hop Task
```python
# Stand reward: maintain stable height
# Hopping reward: achieve target forward velocity
# Leg movement reward: encourage moderate leg motion
# Knee extension reward: encourage proper knee extension
# Foot contact reward: encourage proper ground reaction forces
# Total reward = stand_reward + hop_reward + leg_motion_reward + knee_reward + contact_reward
```
---
## Initial State
- Randomize joint angles within allowed ranges during reset
---
## Episode Termination Conditions
- Observation values contain invalid numerical values (NaN)
---
## Usage Guide
### 1. Environment Preview
```bash
uv run scripts/view.py --env dm-hopper-stand
uv run scripts/view.py --env dm-hopper-hop
```
### 2. Start Training
```bash
uv run scripts/train.py --env dm-hopper-stand
uv run scripts/train.py --env dm-hopper-hop
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/dm-hopper-stand
```
### 4. Test Training Results
```bash
uv run scripts/play.py --env dm-hopper-stand
uv run scripts/play.py --env dm-hopper-hop
```
---
## Expected Training Results
1. Maintain stable standing behavior
2. Achieve a target hopping speed of **2.0**

128
docs/source/en/user_guide/demo/dm_reacher.md Normal file
View File

@@ -0,0 +1,128 @@
# Two-Joint Robotic Arm Control
Reacher is a classic robotic arm control task, simulating a robotic arm composed of two links. The goal is to bring the end effector (fingertip) as close as possible to a randomly generated target point.
```{video} /_static/videos/dm_reacher.mp4
:poster: _static/images/poster/dm_reacher.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
## Task Description
The Reacher consists of two joints, with two links connected by hinge joints. The objective of the task is to move the end of the robotic arm to the target position. The target point is randomly sampled at the beginning of each episode.
---
## Action Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-1.0, 1.0, (2,), float32)` |
| **Dimension** | 2 |
The actions correspond to:
| Index | Action Description | Min Control | Max Control | XML Name | Joint Type |
| ----- | ------------------------------------------------ | ----------- | ----------- | -------- | ---------- |
| 0 | Torque applied to the first joint (root link) | -1 | 1 | joint0 | hinge |
| 1 | Torque applied to the second joint (middle link) | -1 | 1 | joint1 | hinge |
---
## Observation Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-inf, inf, (6,), float32)` |
| **Dimension** | 6 |
The observation vector contains the following parts (in order):
- **qpos**: 2 joint angles
- **fingertip → target vector difference**: x and y dimensions
- **qvel**: 2 joint angular velocities
| Index | Observation | Min | Max | XML Name | Joint | Unit |
| ----- | ------------------------------- | ---- | --- | ---------- | ----- | ----- |
| 0 | First joint angle | -inf | inf | joint0_pos | hinge | rad |
| 1 | Second joint angle | -inf | inf | joint1_pos | hinge | rad |
| 2 | fingertip - target x difference | -inf | inf | NA | slide | m |
| 3 | fingertip - target y difference | -inf | inf | NA | slide | m |
| 4 | First joint angular velocity | -inf | inf | joint0_vel | hinge | rad/s |
| 5 | Second joint angular velocity | -inf | inf | joint1_vel | hinge | rad/s |
---
## Reward Function Design
The reward for this task is based on the **distance between the fingertip and the target**:
### Distance Reward (tolerance reward)
```text
reward = tolerance(|| fingertip - target ||)
```
- The closer the distance, the higher the reward
---
## Initial State
The initial state is sampled from random distributions:
- Arm angles: uniform distribution
- Arm angular velocities: small random values
- Target point position: random position in a circular area
---
## Episode Termination Conditions
### Termination
If `NaN` appears in the observations
### Termination Handling
```text
reward = 0
terminated = True
```
---
## Usage Guide
### 1. Environment Preview
```bash
uv run scripts/view.py --env dm-reacher
```
### 2. Start Training
```bash
uv run scripts/train.py --env dm-reacher
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/dm-reacher
```
### 4. Test Training Results
```bash
uv run scripts/play.py --env dm-reacher
```
## Expected Training Results
The robotic arm quickly and accurately reaches the target point

52
docs/source/en/user_guide/demo/dm_walker.md
View File

@@ -1,4 +1,4 @@
# 2D Walker Robot Training Example
# 2D Walker Robot
The 2D Walker Robot (Walker2D) is a classic robot control task from DeepMind Control Suite. The goal is to achieve standing, walking, and running by controlling the robot's joints.
@@ -46,33 +46,17 @@ uv run scripts/train.py --env dm-runner
### 1. Environment Preview
```bash
# View standing task
uv run scripts/view.py --env dm-stander
# View walking task
uv run scripts/view.py --env dm-walker
# View running task
uv run scripts/view.py --env dm-runner
```
### 2. Start Training
```bash
# Train standing task
uv run scripts/train.py --env dm-stander
# Train walking task (default)
uv run scripts/train.py --env dm-walker
# Train running task
uv run scripts/train.py --env dm-runner
# Customize number of environments
uv run scripts/train.py --env dm-walker --num-envs 512
# Enable rendering (visualize during training)
uv run scripts/train.py --env dm-walker --render
```
### 3. View Training Progress
@@ -84,39 +68,9 @@ uv run tensorboard --logdir runs/dm-walker
### 4. Test Training Results
```bash
# Automatically find best policy for testing (recommended)
uv run scripts/play.py --env dm-stander
uv run scripts/play.py --env dm-walker
# Manually specify policy file for testing
uv run scripts/play.py --env dm-walker --policy runs/dm-walker/nn/best_policy.pickle
```
> **Tip**: The system will automatically find the latest and best policy files in the `runs/dm-walker/` directory for testing. Supports dm-stander, dm-walker, dm-runner three task modes.
## Configuration Parameters
### Environment Configuration
```python
@dataclass
class WalkerEnvCfg(EnvCfg):
model_file: str = "walker.xml" # MJCF model file
max_episode_seconds: float = 25.0 # Maximum episode length
sim_dt: float = 0.0125 # Simulation time step
ctrl_dt: float = 0.025 # Control time step
move_speed: float = 1.0 # Target movement speed
stand_height: float = 1.2 # Target standing height
```
### Training Configuration
```python
@dataclass
class WalkerRLCfg(BaseRLCfg):
num_envs: int = 512 # Number of parallel environments
learning_rate: float = 3e-4 # Learning rate
batch_size: int = 512 # Batch size
max_epochs: int = 1000 # Maximum training epochs
uv run scripts/play.py --env dm-runner
```
## Reward Function Design

249
docs/source/en/user_guide/demo/franka_lift_cube.md Normal file
View File

@@ -0,0 +1,249 @@
# Franka Lift Cube
## Overview
This document describes in detail the cube grasping task environment based on the Franka Emika Panda robotic arm.
```{video} /_static/videos/franka_lift_cube.mp4
:poster: _static/images/poster/franka_lift_cube.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
---
## Environment Description
The Franka lift cube task environment is built based on the real Franka Emika Panda 7-DOF robotic arm, designed to train robots to grasp a cube on a table and lift it to a specified target position.
### Robot Structure
Franka Emika Panda is a 7-DOF robotic arm composed of the following main parts:
- **Base**: Robot base fixed to the table
- **7 Joints**:
- joint1 ~ joint4: Shoulder and arm rotation joints
- joint5 ~ joint7: Wrist rotation joints
- **Gripper**: Two-finger gripper, containing two finger joints
- finger_joint1: Left finger joint
- finger_joint2: Right finger joint
- **End Effector (TCP)**: Center point of gripper, used for grasping operations
### Task Objective
The robot needs to complete the following operation objectives:
1. **Approach Target**: Move from initial position to cube position
2. **Grasp Cube**: Close gripper to grasp cube
3. **Lift Cube**: Lift cube to target height
4. **Precise Positioning**: Move cube to specified target position (XYZ 3D coordinates)
The environment provides visualization aids:
- **Cube**: Red cube that can be grasped, initially at random position on table
- **Target Position**: 3D position where the cube should finally reach
---
## Action Space
The action space is `Box(-inf, inf, (8,), float32)`, representing position control commands applied to 8 joints (offsets relative to current joint positions).
### Control Mode
The environment uses position control mode. Actions are converted to joint target positions as follows:
```
Target Joint Angle = Current Joint Angle + Action Value
```
### Action Dimension Details
| Index | Action Description | Control Range | Joint Name | Joint Type |
| ----- | --------------------- | ------------- | -------------- | ---------- |
| 0 | Joint 1 Offset | -inf ~ inf | joint1 | revolve |
| 1 | Joint 2 Offset | -inf ~ inf | joint2 | hinge |
| 2 | Joint 3 Offset | -inf ~ inf | joint3 | hinge |
| 3 | Joint 4 Offset | -inf ~ inf | joint4 | hinge |
| 4 | Joint 5 Offset | -inf ~ inf | joint5 | hinge |
| 5 | Joint 6 Offset | -inf ~ inf | joint6 | hinge |
| 6 | Joint 7 Offset | -inf ~ inf | joint7 | hinge |
| 7 | Gripper Action (Prob) | -inf ~ inf | finger_joint\* | hinge |
### Gripper Control
The gripper action uses probabilistic control:
1. **Sigmoid Mapping**: Map action value to probability in [0, 1] interval
```
p = 1 / (1 + exp(-action))
```
2. **Bernoulli Sampling**: Random sampling based on probability p
- Sample result < p: Gripper closes (0.0)
- Sample result >= p: Gripper opens (0.04)
### Joint Position Limits
All joint positions are clamped to the following ranges after execution:
| Joint | Min | Max |
| ------- | ------- | ------- |
| 1 | -2.8973 | 2.8973 |
| 2 | -1.7628 | 1.7628 |
| 3 | -2.8973 | 2.8973 |
| 4 | -3.0718 | -0.0698 |
| 5 | -2.8973 | 2.8973 |
| 6 | -0.0175 | 3.7525 |
| 7 | -π/2 | π/2 |
| Gripper | 0 | 0.04 |
---
## Observation Space
The observation space is `Box(-inf, inf, (36,), float32)`, containing the robot's proprioceptive information, object state, and action history.
### Observation Components
The observation vector consists of the following parts (in order):
1. **Joint Angles (9 dimensions)**
- 7 robot arm joint angle offsets relative to default pose
- 2 gripper joint angles
2. **Joint Velocities (9 dimensions)**
- Angular velocities of 9 joints
3. **Cube Current Pose (9 dimensions)**
- Position (3 dim): [x, y, z]
- Quaternion (4 dim): [qx, qy, qz, qw]
- Rotation (Euler, 2 dim): [roll, pitch]
4. **Target Position Command (7 dimensions)**
- Target XYZ coordinates (3 dim)
- Target quaternion (4 dim)
5. **Previous Action (8 dimensions)**
### Observation Details
| Index | Observation Content | Dimensions | Unit |
| ----- | ----------------------------------------------- | ---------- | ------------- |
| 0-8 | Joint Angle Offsets (9 joints) | 9 | rad |
| 9-17 | Joint Angular Velocities (9 joints) | 9 | rad/s |
| 18-26 | Cube Current Pose (position + orientation) | 9 | rad |
| 27-33 | Target Position Command (position + quaternion) | 7 | Dimensionless |
| 34-41 | Previous Action (8 dimensions) | 8 | Dimensionless |
---
## Reward Function
The reward function uses a composite design with multiple reward and penalty terms.
### Main Reward Terms
1. **Approach Reward** (Weight: 1.5)
- Formula: `1.5 × (1 - tanh(d_hand_cube / 0.1))`
- Encourages robot end-effector to approach cube
- d_hand_cube: Euclidean distance from end-effector to cube
2. **Lifting Reward** (Weight: 30)
- Condition: Cube height > 0.04m AND end-effector to cube distance < 0.05m
- Encourages robot to grasp and lift cube
3. **Target Tracking Reward** (Variable Weight)
- **Coarse Tracking** (Weight: 10): Uses Sigmoid function, center distance 0.3m
- **Fine Tracking** (Weight: 20): Uses tanh function, scale factor 0.4m
- **Approach Reward** (Weight: 10): Used when distance < 0.2m, scale factor 0.05m
- **Approach Bonus** (Weight: 200): Extra reward, encourages approaching target
- All tracking rewards only active when cube height > 0.04m and grasp successful
### Penalty Terms
Penalty coefficients adjust with training progress:
| Penalty Term | Early Weight (steps < 10000) | Late Weight (steps >= 10000) |
| ---------------------------------- | ---------------------------- | ---------------------------- |
| Action Rate Penalty | 1e-4 | 1e-1 |
| Joint Velocity Squared Sum Penalty | 1e-4 | 1e-1 |
### Calculation Formulas
```
Action Rate = ||current_action - last_action||²
Joint Velocity Squared Sum = ||joint_vel||²
```
---
## Initial State
### Robot Initialization
**Position Initialization:**
The robot's initial position in world coordinates is fixed:
- Base position: Fixed on table
- Joint angles: Set to default pose with random noise added
**Joint Angle Noise:**
Each joint angle has uniform random noise added in range `[-0.125, 0.125]` radians.
**Velocity Initialization:**
All linear and angular velocities are initialized to zero.
### Cube Initialization
Cube position on table is randomly sampled:
- X coordinate: `[-0.1, 0.1]`
- Y coordinate: `[-0.25, 0.25]`
- Z coordinate: Fixed at 0.05 (above table)
### Target Position Generation
Target position is randomly sampled in the following range:
- X coordinate: `[0.4, 0.6]`
- Y coordinate: `[-0.25, 0.25]`
- Z coordinate: `[0.25, 0.5]`
---
## Usage
### Training
```bash
uv run scripts/train.py --env franka-lift-cube
```
### Policy Evaluation
```bash
uv run scripts/play.py --env franka-lift-cube
```
### TensorBoard
```bash
uv run tensorboard --logdir runs/franka-lift-cube
```

250
docs/source/en/user_guide/demo/franka_open_cabinet.md Normal file
View File

@@ -0,0 +1,250 @@
# Franka Open Cabinet
## Overview
This document describes in detail the cabinet opening task environment based on the Franka Emika Panda robotic arm.
```{video} /_static/videos/franka_open_cabinet.mp4
:poster: _static/images/poster/franka_open_cabinet.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
---
## Environment Description
The Franka open cabinet task environment is built based on the real Franka Emika Panda 7-DOF robotic arm, designed to train robots to approach cabinet door handles, grasp them, and pull open drawers.
### Robot Structure
Franka Emika Panda is a 7-DOF robotic arm composed of the following main parts:
- **Base**: Robot base fixed to the ground
- **7 Joints**:
- joint1 ~ joint4: Shoulder and arm rotation joints
- joint5 ~ joint7: Wrist rotation joints
- **Gripper**: Two-finger gripper, containing two finger joints
- finger_joint1: Left finger joint, with contact pad (left_finger_pad)
- finger_joint2: Right finger joint, with contact pad (right_finger_pad)
- **End Effector (TCP)**: Center point of gripper, used for grasping operations
### Environment Objects
- **Cabinet**: Contains one openable drawer
- **Drawer Handle (drawer_top_handle)**: Target part the robot needs to grasp
- **Drawer Joint (drawer_top_joint)**: Sliding joint of drawer, 1 DOF
### Task Objective
The robot needs to complete the following operation objectives:
1. **Approach Handle**: Move from initial position to drawer handle position
2. **Pose Alignment**: Adjust end-effector pose to align with handle
3. **Grasp Handle**: Close gripper to grasp drawer handle
4. **Open Drawer**: Pull backward to open drawer
---
## Action Space
The action space is `Box(-inf, inf, (8,), float32)`, representing position control commands applied to 8 joints (offsets relative to current joint positions).
### Control Mode
The environment uses position control mode. Actions are converted to joint target positions as follows:
```
Target Joint Angle = Current Joint Angle + Action Value
```
### Action Dimension Details
| Index | Action Description | Control Range | Joint Name | Joint Type |
| ----- | --------------------- | ------------- | -------------- | ---------- |
| 0 | Joint 1 Offset | -inf ~ inf | joint1 | revolve |
| 1 | Joint 2 Offset | -inf ~ inf | joint2 | hinge |
| 2 | Joint 3 Offset | -inf ~ inf | joint3 | hinge |
| 3 | Joint 4 Offset | -inf ~ inf | joint4 | hinge |
| 4 | Joint 5 Offset | -inf ~ inf | joint5 | hinge |
| 5 | Joint 6 Offset | -inf ~ inf | joint6 | hinge |
| 6 | Joint 7 Offset | -inf ~ inf | joint7 | hinge |
| 7 | Gripper Action (Prob) | -inf ~ inf | finger_joint\* | hinge |
### Gripper Control
The gripper action uses probabilistic control:
1. **Sigmoid Mapping**: Map action value to probability in [0, 1] interval
```
p = 1 / (1 + exp(-action))
```
2. **Bernoulli Sampling**: Random sampling based on probability p
- Sample result < p: Gripper closes (0.0)
- Sample result >= p: Gripper opens (0.04)
---
## Observation Space
The observation space is `Box(-5, 5, (25,), float32)`, containing the robot's proprioceptive information, task-related information, and drawer state.
### Observation Components
The observation vector consists of the following parts (in order):
1. **Joint Angles (8 dimensions)**
- 7 robot arm joint angles (normalized to [-1, 1])
- Normalization formula: `2 × (Joint Angle - Lower Bound) / (Upper Bound - Lower Bound) - 1`
2. **Joint Velocities (8 dimensions)**
- Angular velocities of 8 joints (divided by 2 for scaling)
3. **Target Relative Pose (7 dimensions)**
- Position Offset (3 dim): Handle position - End-effector position [Δx, Δy, Δz]
- Orientation Offset (4 dim): Handle orientation - End-effector orientation (quaternion)
4. **Drawer Joint Position (1 dimension)**
- Current open distance of drawer
5. **Drawer Joint Velocity (1 dimension)**
- Current opening velocity of drawer
### Observation Details
| Index | Observation Content | Dimensions | Range | Unit |
| ----- | ------------------------------------------- | ---------- | ------------ | ------------- |
| 0-7 | Normalized Joint Angles (8 joints) | 8 | [-1, 1] | Dimensionless |
| 8-15 | Normalized Joint Velocities (8 joints) | 8 | ≈[-π/2, π/2] | rad/s |
| 16-18 | Relative Position to Handle | 3 | [-5, 5] | m |
| 19-22 | Relative Orientation to Handle (quaternion) | 4 | [-5, 5] | Dimensionless |
| 23 | Drawer Joint Position | 1 | [-5, 5] | m |
| 24 | Drawer Joint Velocity | 1 | [-5, 5] | m/s |
All observation values are clipped to [-5, 5] range for numerical stability.
---
## Reward Function
The reward function uses a composite design with multiple reward and penalty terms.
### Main Reward Terms
1. **Distance Reward** (Weight: 10)
- Formula: `10 × (1 - tanh(d_gripper_handle / 0.1))`
- Encourages robot end-effector to approach drawer handle
- d_gripper_handle: Euclidean distance from end-effector to handle
2. **Orientation Matching Reward**
- Formula: Quaternion similarity function
- Encourages robot end-effector orientation to align with handle orientation
3. **Gripper Close Reward** (Conditional Reward)
- When distance < 0.025m: Closing gripper receives +100 reward
- When distance >= 0.025m: Closing gripper receives -20 penalty
- Opening gripper: No reward (0)
- Encourages robot to close gripper to grasp when approaching
4. **Open Drawer Reward** (Exponential Reward)
- Formula: `20 × (exp(open_dist) - 1)`
- open_dist: Drawer open distance (clipped to [0, 1] range)
- Reward grows exponentially as drawer opens more
5. **Prevent Illegal Opening**
- When drawer is already open (open_dist > 0) but end-effector not contacting handle (distance > 0.03m), cancel open reward
- Prevents robot from using other methods to force open drawer
### Penalty Terms
1. **Action Rate Penalty**
- Formula: `||current_action - last_action||²`
2. **Joint Velocity Penalty**
- Formula: `||joint_vel||²`
3. **Finger Position Penetration Penalty**
- Applied when finger contact pads are below handle surface
- Prevents finger model from penetrating drawer
### Penalty Coefficient Scheduling
Penalty coefficients adjust with training progress:
| Penalty Term | Early Weight (steps < 8000) | Late Weight (steps >= 8000) |
| ---------------------- | --------------------------- | --------------------------- |
| Action Rate | 1e-3 | 2e-3 |
| Joint Velocity Squared | 0 | 2e-7 |
### Termination Penalty
When termination condition is triggered, additional -10.0 penalty is applied.
---
## Initial State
### Robot Initialization
**Position Initialization:**
The robot's initial position in world coordinates is fixed:
- Base position: Fixed on ground
- Joint angles: Set to default pose
**Default Joint Pose:**
```
[0.0, -30°, 0°, -156°, 0.0, 186°, -45°, 0.04, 0.04] (radians)
```
**Joint Angle Noise:**
Each joint angle has uniform random noise added in range `[-0.125, 0.125]` radians.
**Velocity Initialization:**
All linear and angular velocities are initialized to zero.
### Cabinet Initialization
Cabinet is fixed on ground with drawer in closed state (joint position at 0).
---
## Usage
### Training
```bash
uv run scripts/train.py --env franka-open-cabinet
```
### Policy Evaluation
```bash
uv run scripts/play.py --env franka-open-cabinet
```
### TensorBoard
```bash
uv run tensorboard --logdir runs/franka-open-cabinet
```

86
docs/source/en/user_guide/demo/locomotion_unitree_go1.md
View File

@@ -1,4 +1,4 @@
# Unitree GO1 Robot Walking Training Example
# Unitree GO1 Locomotion
Unitree GO1 is a quadruped robot platform. This example demonstrates how to train GO1 to achieve stable gait walking on flat terrain.
@@ -21,82 +21,32 @@ The GO1 quadruped robot has 12 degrees of freedom (3 joints per leg) and needs t
- **Reward Function**: Composite reward including speed tracking, posture stability, energy efficiency, and other components
- **Termination Conditions**: Robot trunk contacts ground or other unstable states
### Training Task
---
## Usage Guide
### 1. Environment Preview
```bash
uv run scripts/view.py --env go1-flat-terrain-walk
```
### 2. Start Training
```bash
uv run scripts/train.py --env go1-flat-terrain-walk
```
## Configuration Parameters
### 3. View Training Progress
### Environment Configuration
```python
@dataclass
class Go1WalkNpEnvCfg(EnvCfg):
max_episode_seconds: float = 20.0 # Maximum episode length
model_file: str = "scene_motor_actuator.xml"
sim_dt: float = 0.01 # Simulation time step
ctrl_dt: float = 0.01 # Control time step
```bash
uv run tensorboard --logdir runs/go1-flat-terrain-walk
```
### Training Configuration
### 4. Test Training Results
```python
from dataclasses import dataclass
from motrix_rl.skrl.cfg import PPOCfg
from motrix_rl import registry
@registry.rlcfg("go1-flat-terrain-walk")
@dataclass
class Go1WalkPPO(PPOCfg):
"""
GO1 quadruped robot walking training configuration
"""
seed = 42
max_env_steps: int = 40960000 # Maximum training steps
num_envs: int = 2048 # Number of parallel environments
# Large network structure (suitable for complex robot control tasks)
policy_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
value_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
# PPO parameters (optimized for robot tasks)
learning_epochs: int = 2 # Training rounds
mini_batches: int = 32 # Number of mini-batches
learning_rate: float = 1e-3 # Learning rate
```
**Note**: GO1 is a complex task that uses large network structures. If you need to create specialized configurations for different training backends (JAX/Torch), refer to the environment configuration documentation examples.
### Control Configuration
```python
@dataclass
class ControlConfig:
stiffness = 80 # PD controller stiffness [N*m/rad]
damping = 1 # PD controller damping [N*m*s/rad]
action_scale = 0.1 # Action scaling factor
```
### Initial Joint Angles
```python
default_joint_angles = {
"FL_hip": 0.0, # Front left hip joint
"RL_hip": 0.0, # Rear left hip joint
"FR_hip": -0.0, # Front right hip joint
"RR_hip": -0.0, # Rear right hip joint
"FL_thigh": 0.9, # Front left thigh
"RL_thigh": 0.9, # Rear left thigh
"FR_thigh": 0.9, # Front right thigh
"RR_thigh": 0.9, # Rear right thigh
"FL_calf": -1.8, # Front left calf
"RL_calf": -1.8, # Rear left calf
"FR_calf": -1.8, # Front right calf
"RR_calf": -1.8, # Rear right calf
}
```bash
uv run scripts/play.py --env go1-flat-terrain-walk
```
## Reward Function Design

202
docs/source/en/user_guide/demo/locomotion_unitree_go1_rough_terrain.md Normal file
View File

@@ -0,0 +1,202 @@
# Unitree GO1 Complex Terrain Locomotion
The Unitree GO1 Complex Terrain Walking Environment is a quadruped robot reinforcement learning task designed to train robots to achieve stable walking on challenging terrain. This environment includes two main terrain types: rough terrain and stairs terrain.
```{video} /_static/videos/go1_rough_terrain_walk.mp4
:poster: _static/images/poster/go1_rough_terrain_walk.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
---
```{video} /_static/videos/go1_stairs_terrain_walk.mp4
:poster: _static/images/poster/go1_stairs_terrain_walk.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
## Task Description
Train the Unitree GO1 quadruped robot to achieve stable and efficient quadruped walking on complex terrain. This environment uses MotrixSim physics engine for simulation, providing high-fidelity dynamic simulation. The agent controls target positions of each joint to achieve velocity tracking and attitude stability while adapting to different terrain challenges.
### Task Objectives
- **Velocity Tracking**: Accurately track given linear and angular velocity commands
- **Attitude Stability**: Maintain body attitude stability under various terrain conditions
- **Energy Efficiency**: Achieve walking tasks with minimal energy consumption
- **Terrain Adaptability**: Adapt to different challenges of rough terrain and stairs terrain
---
## Action Space
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-1.0, 1.0, (12,), float32)` |
| **Dimension** | 12 |
Actions correspond to position control commands for 12 joints, including hip joints, thigh joints, and calf joints for all four legs.
---
## Observation Space
### Rough Terrain Observation Space (48-dimensional)
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (48,), float32)` |
| **Dimension** | 48 |
| Part | Content Description | Dim | Notes |
| --------------------- | ------------------------------------------- | --- | ------------------ |
| **noisy_linvel** | Linear velocity in body coordinate system | 3 | With noise |
| **noisy_gyro** | Angular velocity in body coordinate system | 3 | With noise |
| **local_gravity** | Local gravity direction | 3 | Gravity vector |
| **noisy_joint_angle** | Joint angle deviation from default angles | 12 | 12 joints |
| **noisy_joint_vel** | Joint angular velocities | 12 | With noise |
| **last_actions** | Control actions from previous time step | 12 | Historical actions |
| **command** | Target linear velocity and angular velocity | 3 | [vx, vy, vyaw] |
### Stairs Terrain Observation Space (60-dimensional)
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (60,), float32)` |
| **Dimension** | 60 |
In addition to the rough terrain observations, it includes:
| Part | Content Description | Dim | Notes |
| ---------------------- | ----------------------------------- | --- | ------------------------ |
| **feet_contact_force** | Contact force vectors for four feet | 12 | 3D per foot (Fx, Fy, Fz) |
---
## Reward Function Design
GO1 complex terrain reward function adopts multi-objective weighted design:
```python
# Core reward components
reward_config.scales = {
"tracking_lin_vel": 1.0, # Linear velocity tracking accuracy
"tracking_ang_vel": 0.5, # Angular velocity tracking accuracy
"orientation": -0.0, # Body attitude stability penalty
"torques": -0.00001, # Joint torque penalty (energy efficiency)
"dof_acc": -2.5e-7, # Joint acceleration penalty
"action_rate": -0.001, # Action smoothness penalty
"feet_air_time": 1.0, # Foot air time reward (encourage large strides)
"stand_still": 0.0, # Joint position maintenance for stationary commands
"hip_pos": -1, # Hip joint position preference
"calf_pos": -0.3, # Calf joint position preference
"feet_stumble": -0.5, # Penalty when foot laterally touches obstacle
}
# Total reward = weighted combination of all above terms
```
---
## Initial State
### Rough Terrain Initialization
- **Terrain Generation**: Use height maps to generate random terrain
- **Terrain Height Levels**: Three preset height levels: -2.5m, 0.5m, 2.0m
- **Position Randomization**: At basic training level, robot position is fixed; at advanced level, randomly selects from 25 preset positions in cycles
### Stairs Terrain Initialization
- **Terrain Type**: Various step-based small terrain blocks arranged on continuous terrain
- **Position Randomization**: Similar position randomization strategy as rough terrain
### Robot Initialization
- **Joint Angles**: Set to default standing posture with [-0.125, 0.125] radian noise
- **Velocity Initialization**: All linear and angular velocities initialized to zero
---
## Episode Termination Conditions
- **Body Contact**: Robot trunk makes unexpected contact with ground
- **Velocity Anomaly**: Sum of squared linear velocities exceeds threshold (1e8)
---
## Usage Guide
### 1. Environment Preview
```bash
uv run scripts/view.py --env go1-rough-terrain-walk
uv run scripts/view.py --env go1-stairs-terrain-walk
```
### 2. Start Training
```bash
uv run scripts/train.py --env go1-rough-terrain-walk
uv run scripts/train.py --env go1-stairs-terrain-walk
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/go1-rough-terrain-walk
```
### 4. Test Training Results
Since the rough terrain scene generates both an infinite plane and a rugged terrain height field, when testing training results, the agent will first be spawned on the flat terrain following the training process, complete one round of walking, and then be spawned onto the rugged terrain. Users need to actively adjust camera perspective and position to observe the agent's status.
```bash
uv run scripts/play.py --env go1-rough-terrain-walk
uv run scripts/play.py --env go1-stairs-terrain-walk
```
---
## Expected Training Results
### Rough Terrain Task (go1-rough-terrain-walk)
1. Capable of adapting to different rough terrain heights
2. High velocity tracking accuracy and stable posture
3. Coordinated gait with minimal foot slipping
### Stairs Terrain Task (go1-stairs-terrain-walk)
1. Capable of stably going up and down stairs
2. Capable of adapting to stairs of different heights and widths
3. Smooth movements without obvious stuttering
---
## Training Performance Reference
### go1-rough-terrain-walk
| Operating System | Training Backend | CPU | GPU | Num Environments | Training Time (30000 steps) |
| ---------------- | ---------------- | ----------------- | ----------- | ---------------- | --------------------------- |
| Ubuntu 22.04 | JAX | AMD Ryzen 7 9700X | RTX 5070 Ti | 2048 | 7m20s |
| Ubuntu 22.04 | PyTorch | AMD Ryzen 7 9700X | RTX 5070 Ti | 2048 | 8m30s |
| Windows 11 | PyTorch | AMD Ryzen 7 9700X | RTX 5070 Ti | 2048 | 10m42s |
### go1-stairs-terrain-walk
| Operating System | Training Backend | CPU | GPU | Num Environments | Training Time (30000 steps) |
| ---------------- | ---------------- | ----------------- | ----------- | ---------------- | --------------------------- |
| Ubuntu 22.04 | JAX | AMD Ryzen 7 9700X | RTX 5070 Ti | 2048 | 7m18s |
| Ubuntu 22.04 | PyTorch | AMD Ryzen 7 9700X | RTX 5070 Ti | 2048 | 8m41s |
| Windows 11 | PyTorch | AMD Ryzen 7 9700X | RTX 5070 Ti | 2048 | 10m52s |

8
docs/source/en/user_guide/index.md
View File

@@ -26,7 +26,15 @@ tutorial/training_and_result
:maxdepth: 1
demo/cartpole
demo/bounce_ball
demo/dm_walker
demo/dm_cheetah
demo/dm_hopper
demo/dm_reacher
demo/locomotion_unitree_go1
demo/locomotion_unitree_go1_rough_terrain
demo/anymal_c
demo/franka_lift_cube
demo/franka_open_cabinet
```