cyy/Motrixlab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: release v0.2.0

Browse Source
This commit is contained in:
motphys-developers
2026-02-10 08:08:11 +00:00
parent dbfa9e31fa
commit b568ac5600
123 changed files with 9732 additions and 497 deletions
Download Patch File Download Diff File Expand all files Collapse all files

154
docs/source/en/user_guide/demo/acrobot.md Normal file
View File

@@ -0,0 +1,154 @@
# Acrobot
Acrobot is a two-link swing-up and balance task. The goal is to swing both arms up and reach a target position using one motor torque.
```{video} /_static/videos/acrobot.mp4
:poster: _static/images/poster/acrobot.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
---
## Task Description
A two-link acrobot with one hinge joint is driven by a single motor. The motor is installed at the elbow joint, which is the only actuated joint in the system. The motor's torque rotates the rods in a plane, enabling swing-up from arbitrary initial angles and reaching a target position. Torque is limited by the actuator ctrlrange; by modulating its magnitude and direction, the policy must accumulate energy to swing up and reach the target while maintaining stability.
## Action Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-1.0, 1.0, (1,), float32)` |
| **Dimension** | 1 |
---
## Observation Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-inf, inf, (6,), float32)` |
| **Dimension** | 6 |
Order: `upper_arm_horizontal, lower_arm_horizontal, upper_arm_vertical, lower_arm_vertical, shoulder_velocity, elbow_velocity`.
---
## Reward Function Design
- Base sparse reward: encourages the tip to enter the target region (radius = 0.2)
- Continuous reward: provides 0.1 reward per step for staying in the target region
- Distance shaping: 0.3 \* (1.0 - clip(distance / 2.0, 0, 1.0)) to encourage movement towards target
- Velocity penalty: 0.01 \* max(0, velocity_magnitude - 2.0) to penalize excessive velocities
---
## Initial State
- Shoulder angle randomized in `[-pi, pi]`
- Elbow angle randomized in `[-pi, pi]`
- Angular velocities initialized to zero
## Episode Termination Conditions
- Episode length limited by `max_episode_seconds`
- NaN check for observation values
---
### 1. Environment Preview
```bash
uv run scripts/view.py --env acrobot
```
### 2. Start Training
```bash
# Train with default parameters
uv run scripts/train.py --env acrobot
# Customize parallel environments
uv run scripts/train.py --env acrobot --num-envs 1024
# Enable rendering during training
uv run scripts/train.py --env acrobot --render
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/acrobot
```
### 4. Test Training Results
```bash
# Auto-discover best policy (recommended)
uv run scripts/play.py --env acrobot
# Manually specify a policy file
uv run scripts/play.py --env acrobot --policy runs/acrobot/nn/best_policy.pickle
```
> **Tip**: Policies are auto-selected from `runs/acrobot/`. You can override with `--policy`.
---
## Configuration Parameters
### Environment Configuration
```python
@dataclass
class AcrobotEnvCfg(EnvCfg):
model_file: str = ".../acrobot.xml" # MJCF model
max_episode_seconds: float = 10.0
sim_dt: float = 0.01
ctrl_dt: float = 0.02
reset_noise_scale: float = 0.1
render_spacing: float = 2.0
```
### Training Configuration (PPO example)
```python
@rlcfg("acrobot", backend="jax")
@dataclass
class AcrobotPPO(PPOCfg):
max_env_steps: int = 60_000_000
check_point_interval: int = 500
# Override PPO configuration
policy_hidden_layer_sizes: tuple[int, ...] = (32, 32)
value_hidden_layer_sizes: tuple[int, ...] = (32, 32)
rollouts: int = 64
learning_epochs: int = 5
mini_batches: int = 8
learning_rate: float = 3e-4
grad_norm_clip: float = 0.1
clip_predicted_values: bool = False
value_clip: float = 10.0
entropy_loss_scale: float = 0.1
learning_rate_scheduler_kl_threshold: float = 0.02
discount_factor: float = 0.995
lambda_param: float = 0.97
ratio_clip: float = 0.2
value_loss_scale: float = 0.5
random_timesteps: int = 0
learning_starts: int = 0
kl_threshold: float = 0.03
```
---
## Expected Training Results
1. Acrobot can swing up both arms to reach the target position
2. The tip can stay within the target region with stability
3. Excessive oscillations are reduced by velocity penalty
4. The policy efficiently approaches the target with smooth movements

2
docs/source/en/user_guide/demo/anymal_c.md
View File

@@ -1,4 +1,4 @@
# ANYmal-C Locomotion
# ANYmal-C Navigation
## Overview

373
docs/source/en/user_guide/demo/bounce_ball.md
View File

@@ -42,46 +42,357 @@ The joints correspond as follows:
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (25,), float32)` |
| **Dimension** | 25 |
| **Type** | `Box(-inf, inf, (29,), float32)` |
| **Dimension** | 29 |
The observation space consists of the following parts (in order):
| Part | Content Description | Dimension | Remarks |
| ----------- | ----------------------------------------------- | --------- | -------------------------------------------------------------------------------- |
| **dof_pos** | Position information for each degree of freedom | 13 | First 6 are arm joints, last 7 are ball's free joint (3 position + 4 quaternion) |
| **dof_vel** | Velocity information for each degree of freedom | 12 | Velocity is derivative of position |
| Part | Content Description | Dimension | Remarks |
| ----------------- | ----------------------------------------------- | --------- | -------------------------------------------------------------------------------- |
| **dof_pos** | Position information for each degree of freedom | 13 | First 6 are arm joints, last 7 are ball's free joint (3 position + 4 quaternion) |
| **dof_vel** | Velocity information for each degree of freedom | 12 | Velocity is derivative of position |
| **paddle_pos** | Paddle position information | 3 | x, y, z coordinates of paddle center |
| **target_height** | Target height | 1 | Target height for current environment |
| Index | Observation | Min Value | Max Value | XML Name | Type (Unit) |
| ----- | --------------------------- | --------- | --------- | ----------- | ------------------------- |
| 0-5 | Arm Joint Angles | -Inf | Inf | Joint1-6 | Angle (rad) |
| 6 | Ball x-coordinate | -Inf | Inf | ball_x | Position (m) |
| 7 | Ball y-coordinate | -Inf | Inf | ball_y | Position (m) |
| 8 | Ball z-coordinate | -Inf | Inf | ball_z | Position (m) |
| 9-12 | Ball Orientation Quaternion | -Inf | Inf | ball_qw/xyz | Quaternion (w,x,y,z) |
| 13-24 | Joint and Ball Velocities | -Inf | Inf | - | Velocity/Angular Velocity |
| Index | Observation | Min Value | Max Value | XML Name | Type (Unit) |
| ----- | ------------------------------------- | --------- | --------- | --------- | ------------------------ |
| 0-5 | Arm Joint Angles | -Inf | Inf | Joint1-6 | Angle (rad) |
| 6-8 | Ball Position [x, y, z] | -Inf | Inf | ball_link | Position (m) |
| 9-12 | Ball Orientation Quaternion [w,x,y,z] | -Inf | Inf | ball_link | Quaternion |
| 13-18 | Arm Joint Angular Velocities | -Inf | Inf | Joint1-6 | Angular Velocity (rad/s) |
| 19-24 | Ball Velocity [vx,vy,vz,wx,wy,wz] | -Inf | Inf | ball_link | Velocity (m/s, rad/s) |
| 25-27 | Paddle Position [x, y, z] | -Inf | Inf | blocker | Position (m) |
| 28 | Target Height | -Inf | Inf | - | Position (m) |
---
## Reward Function Design
## Reward Function
The reward function consists of the following components:
The reward function uses a composite design with multiple reward and penalty terms to guide the robot to learn a stable ball bouncing strategy. All reward parameters can be adjusted through the configuration file.
```python
# Position Control Reward: Keep ball above paddle center
# Controlled Upward Velocity Reward: Reward moderate upward velocity when ball is well-positioned
# Height Accuracy Reward: Ball is close to target height
# Consecutive Bounces Reward: Reward consecutive successful bounces
# Total Reward = Weighted combination of all components
```
### Main Reward Terms
#### 1. Horizontal Position Reward
**Design Rationale**: This is the core reward term, ensuring the ball stays directly above the paddle. Through a vertical distance weighting mechanism, when the ball is close to the paddle (about to hit), the horizontal position requirement is stricter, guiding the strategy to align precisely at critical moments.
**Formula**:
$$
\begin{aligned}
\text{err}_{xy} &= \sqrt{(x_{ball} - x_{target})^2 + (y_{ball} - y_{target})^2} &&\text{(Horizontal position error)} \\
d_{vert} &= |z_{ball} - z_{paddle}| &&\text{(Vertical distance)} \\
w_{vert} &= e^{-d_{vert} / \sigma_{vert}} &&\text{(Vertical distance weight)} \\
\sigma_{pos} &= \sigma_{base} \times (1.0 + k_{weight} \times w_{vert}) &&\text{(Adaptive scale)} \\
r_{pos} &= e^{-\frac{\text{err}_{xy}^2}{2\sigma_{pos}^2}} &&\text{(Gaussian reward)} \\
\\
\text{Where:} \quad &\sigma_{vert} = 0.15 \text{ m} &&\text{(Vertical distance scale)} \\
&\sigma_{base} = 0.1 \text{ m} &&\text{(Base horizontal scale)} \\
&k_{weight} = 3.0 &&\text{(Weight factor)} \\
&x_{target} = 0.58856 \text{ m}, \, y_{target} = 0.0 \text{ m} &&\text{(Target position)}
\end{aligned}
$$
**Weight**: 2.0
#### 2. Out of Position Penalty
**Design Rationale**: Applies strong penalty for severe deviation from target position to prevent the ball from flying out of control range. Uses sigmoid function for smooth transition, avoiding discontinuous reward function.
**Formula**:
$$
r_{out} = -\frac{2.0}{1 + e^{-(\text{err}_{xy} - 0.05) / 0.03}} \qquad \text{(Sigmoid penalty)}
$$
**Weight**: 1.0
#### 3. Velocity Matching Reward
**Design Rationale**: Based on projectile motion physics, encourages the ball's trajectory to have the desired velocity (0.5 m/s) at target height. This ensures the ball doesn't pass through the target height too fast or too slow, facilitating stable control.
**Formula**:
$$
\begin{aligned}
\Delta h &= h_{target} - z_{ball} &&\text{(Height difference)} \\
v_{desired} &= 0.5 \text{ m/s} &&\text{(Desired velocity)} \\
\\
\text{Case 1:}&\text{ Ball moving upward and below target height} \\
v_{z,up}^2 &= v_z^2 - 2g\Delta h &&\text{(Energy conservation)} \\
v_{at\_target,up} &= \sqrt{\max(0, v_{z,up}^2)} &&\text{(Upward arrival velocity)} \\
\\
\text{Case 2:}&\text{ Ball moving downward and above target height} \\
v_{z,down}^2 &= v_z^2 + 2g|\Delta h| &&\text{(Energy conservation)} \\
v_{at\_target,down} &= -\sqrt{\max(0, v_{z,down}^2)} &&\text{(Downward arrival velocity)} \\
\\
\text{Case 3:}&\text{ Ball near target height (} |\Delta h| < 0.05 \text{ m)} \\
v_{at\_target,near} &= v_z &&\text{(Current velocity)} \\
\\
\text{Smooth combination:}& \\
\sigma_{up} &= \frac{1}{1 + e^{-v_z / 0.2}} &&\text{(Upward motion weight)} \\
\sigma_{below} &= \frac{1}{1 + e^{-\Delta h / 0.02}} &&\text{(Below target weight)} \\
\sigma_{down} &= 1 - \sigma_{up} &&\text{(Downward motion weight)} \\
\sigma_{above} &= 1 - \sigma_{below} &&\text{(Above target weight)} \\
w_{near} &= e^{-\frac{\Delta h^2}{2 \times 0.01^2}} &&\text{(Near target weight)} \\
\\
v_{at\_target} &= v_{at\_target,up} \cdot \sigma_{up} \cdot \sigma_{below} \\
&\quad + v_{at\_target,down} \cdot \sigma_{down} \cdot \sigma_{above} \\
&\quad + v_{at\_target,near} \cdot w_{near} &&\text{(Weighted combination)} \\
\\
\text{err}_{vel} &= |v_{at\_target} - v_{desired}| &&\text{(Velocity error)} \\
r_{vel} &= e^{-\frac{\text{err}_{vel}^2}{2 \times 0.8^2}} &&\text{(Gaussian reward)}
\end{aligned}
$$
**Weight**: 2.0
#### 4. Height Reward
**Design Rationale**: Directly encourages the ball to approach target height, one of the core task objectives. Higher weight (4.5) ensures the strategy prioritizes height control. Target height is randomly sampled (0.3-0.6 m) in each environment to improve policy generalization.
**Formula**:
$$
\begin{aligned}
\text{err}_h &= |z_{ball} - h_{target}| &&\text{(Height error)} \\
r_h &= e^{-\frac{\text{err}_h^2}{2 \times 0.15^2}} &&\text{(Gaussian reward)}
\end{aligned}
$$
**Weight**: 4.5
#### 5. Height Progress Reward
**Design Rationale**: Encourages the ball to reach higher positions, helping the strategy quickly learn to hit the ball upward in early training, avoiding the "no-hit" local optimum.
**Formula**:
$$
r_{progress} = \max(0, z_{ball} - 0.2) \times 2.0 \qquad \text{(Linear reward)}
$$
**Weight**: 1.0
#### 6. Controlled Upward Velocity Reward
**Design Rationale**: Only rewards upward velocity when the ball's horizontal position is good, avoiding "random hitting" behavior. Ideal velocity is calculated from physics formula, ensuring the ball can exactly reach target height. This reward guides the strategy to learn precise hitting force.
**Formula**:
$$
\begin{aligned}
q_{pos} &= e^{-\frac{\text{err}_{xy}^2}{2 \times 0.02^2}} &&\text{(Position quality)} \\
v_{ideal} &= \sqrt{2g \times \max(0, \Delta h)} &&\text{(Ideal launch velocity)} \\
v_{ideal} &\in [0.5, 3.0] \text{ m/s} &&\text{(Limit range)} \\
q_{vel} &= e^{-\frac{(v_z - v_{ideal})^2}{2 \times 0.5^2}} &&\text{(Velocity quality)} \\
\sigma_{up} &= \frac{1}{1 + e^{-v_z / 0.1}} &&\text{(Upward mask)} \\
r_{controlled} &= q_{pos} \times q_{vel} \times \sigma_{up} \times \text{clip}(v_z, 0, 1.5) &&\text{(Combined reward)}
\end{aligned}
$$
**Weight**: 1.5
#### 7. Consecutive Bounces Reward
**Design Rationale**: Encourages multiple consecutive successful bounces, guiding the strategy to learn stable long-term control. Uses logarithmic function to avoid infinite reward growth, while requiring good ball position to give reward.
**Formula**:
$$
\begin{aligned}
q_{bounce} &= e^{-\frac{\text{err}_{xy}^2}{2 \times 0.05^2}} &&\text{(Bounce position quality)} \\
r_{bounce\_log} &= 0.5 \times \log(n_{bounces} + 1) &&\text{(Logarithmic reward)} \\
r_{bounce} &= r_{bounce\_log} \times q_{bounce} \times \mathbb{1}_{n_{bounces} > 0} &&\text{(Conditional reward)}
\end{aligned}
$$
**Weight**: 0.8
#### 8. High Bounce Count Reward
**Design Rationale**: Gives extra reward for high bounce counts (≥3), further encouraging long-term stable control. Uses sigmoid activation function for smooth reward growth.
**Formula**:
$$
\begin{aligned}
\sigma_{high} &= \frac{1}{1 + e^{-(n_{bounces} - 2.0) / 0.5}} &&\text{(High bounce activation)} \\
r_{high} &= n_{bounces} \times 0.15 \times q_{bounce} \times \sigma_{high} &&\text{(Extra reward)}
\end{aligned}
$$
**Weight**: 0.3
#### 9. Paddle-Ball Horizontal Alignment Reward
**Design Rationale**: Encourages the paddle to actively move directly below the ball rather than waiting for the ball to fall. The closer the vertical distance, the greater the weight, requiring more precise alignment at the moment of hitting. Extra reward (boost) is given at bounce moment to reinforce correct hitting behavior.
**Formula**:
$$
\begin{aligned}
\text{err}_{align} &= \|(\mathbf{x}_{ball}, \mathbf{y}_{ball}) - (\mathbf{x}_{paddle}, \mathbf{y}_{paddle})\|_2 &&\text{(Alignment error)} \\
w_{prox} &= e^{-d_{vert} / 0.1} &&\text{(Vertical proximity weight)} \\
q_{align} &= e^{-\frac{\text{err}_{align}^2}{2 \times 0.03^2}} \times (1.0 + 2.0 \times w_{prox}) &&\text{(Alignment quality)} \\
k_{boost} &= \begin{cases} 3.0 & \text{if bounce detected} \\ 1.0 & \text{otherwise} \end{cases} &&\text{(Bounce boost)} \\
r_{align} &= q_{align} \times k_{boost} \times 0.3 &&\text{(Alignment reward)}
\end{aligned}
$$
**Weight**: 0.6
#### 10. Paddle Home Position Reward
**Design Rationale**: Encourages the paddle to return to home position when the ball is far away, avoiding the paddle staying at high position for long time. Uses distance dynamic factor: when ball is far from paddle ($d_{vert} > 0.15$ m), increase reward to encourage quick return; when ball is close, decrease reward to allow paddle to move up for hitting. This design makes paddle motion more energy-efficient and natural.
**Formula**:
$$
\begin{aligned}
\text{err}_{home} &= |z_{paddle} - z_{home}| &&\text{(Home deviation)} \\
k_{dist} &= 1.0 + \frac{0.5}{1 + e^{-(d_{vert} - 0.15) / 0.03}} &&\text{(Distance factor)} \\
q_{home} &= e^{-\frac{\text{err}_{home}^2}{2 \times 0.05^2}} &&\text{(Home quality)} \\
r_{home} &= q_{home} \times k_{dist} &&\text{(Home reward)} \\
\\
\text{Where:} \quad &z_{home} = 0.05 \text{ m} &&\text{(Paddle home height)}
\end{aligned}
$$
**Weight**: 1.5
### Penalty Terms
#### 1. Excessive Upward Velocity Penalty
**Design Rationale**: Prevents ball velocity from being too fast (>3.5 m/s) and losing control, ensuring ball motion stays within controllable range.
**Formula**:
$$
r_{excess} = -\frac{1.0}{1 + e^{-(v_z - 3.5) / 0.3}} \qquad \text{(Sigmoid penalty)}
$$
**Weight**: 1.0
#### 2. Downward Velocity Penalty
**Design Rationale**: Penalizes ball moving downward ($v_z < -0.2$ m/s), encouraging the strategy to hit the ball in time, avoiding free fall.
**Formula**:
$$
\begin{aligned}
\text{mag}_{down} &= -v_z \times \text{clip}(-v_z \times 0.3, 0, 0.5) &&\text{(Downward magnitude)} \\
\sigma_{down} &= \frac{1}{1 + e^{(v_z + 0.2) / 0.2}} &&\text{(Downward activation)} \\
r_{down} &= \text{mag}_{down} \times \sigma_{down} &&\text{(Downward penalty)}
\end{aligned}
$$
**Weight**: 1.0
#### 3. Paddle Height Violation Penalty
**Design Rationale**: Applies strong penalty when paddle deviates too far from home position (>0.1 m), ensuring paddle doesn't stay at high position for long time.
**Formula**:
$$
\begin{aligned}
\text{viol}_{height} &= \max(0, \text{err}_{home} - 0.1) &&\text{(Violation amount)} \\
r_{violation} &= -20.0 \times \text{viol}_{height} &&\text{(Strong penalty)}
\end{aligned}
$$
**Weight**: 1.0
#### 4. Action Change Rate Penalty
**Design Rationale**: Penalizes drastic action changes, encouraging smooth control strategy.
**Formula**:
$$
\begin{aligned}
r_{action} &= -\|\mathbf{a}_t - \mathbf{a}_{t-1}\|^2 &&\text{(L2 penalty)} \\
\\
\text{Where:} \quad &\mathbf{a}_t \in \mathbb{R}^6 &&\text{(Current action vector)}
\end{aligned}
$$
**Weight**: $10^{-4}$
#### 5. Joint Velocity Penalty
**Design Rationale**: Penalizes excessive joint velocities, encouraging energy-efficient and smooth motion.
**Formula**:
$$
\begin{aligned}
r_{joint\_vel} &= -\sum_{i=1}^{6} \dot{q}_i^2 &&\text{(Velocity squared sum penalty)} \\
\\
\text{Where:} \quad &\dot{\mathbf{q}} = [\dot{q}_1, \dot{q}_2, \dot{q}_3, \dot{q}_4, \dot{q}_5, \dot{q}_6]^T &&\text{(Joint angular velocity vector)}
\end{aligned}
$$
**Weight**: $10^{-4}$
### Total Reward Calculation
$$
\begin{aligned}
R_{total} = &\ 2.0 \cdot r_{pos} \\
&+ 1.0 \cdot r_{out} \\
&+ 2.0 \cdot r_{vel} \\
&+ 4.5 \cdot r_h \\
&+ 1.0 \cdot r_{progress} \\
&+ 1.5 \cdot r_{controlled} \\
&+ 0.8 \cdot r_{bounce} \\
&+ 0.3 \cdot r_{high} \\
&+ 0.6 \cdot r_{align} \\
&+ 1.5 \cdot r_{home} \\
&+ 1.0 \cdot r_{excess} \\
&+ 1.0 \cdot r_{down} \\
&+ 1.0 \cdot r_{violation} \\
&+ 10^{-4} \cdot r_{action} \\
&+ 10^{-4} \cdot r_{joint\_vel}
\end{aligned}
$$
---
## Initial State
- **Arm Initial Position**: [0, 40, 110, 0, -60, 0] degrees, with random noise
- **Ball Initial Position**: Above paddle center, with random noise
- **Ball Initial Velocity**: [0.0, 0.0, 0.0] m/s
### Robot Initialization
**Joint Angles**:
- Default angles: [0°, 40°, 110°, 0°, -60°, 0°]
- Random noise: Uniform random noise in [-0.1, 0.1] radians added to each joint
**Joint Velocities**:
- Initialized to zero with small random noise
### Ball Initialization
**Position**:
- Base position: `ball_init_pos` from config file (default [0.58856, 0, 0.45] m)
- Random noise: Uniform random noise in [-0.01, 0.01] m
**Velocity**:
- Initialized to zero
**Orientation**:
- Quaternion: [0, 0, 0, 1] (identity quaternion, no rotation)
### Target Height
Target height for each environment is randomly sampled in [0.4, 0.6] m range to improve policy generalization.
---
@@ -89,7 +400,9 @@ The reward function consists of the following components:
- **Ball Falls**: Ball z-coordinate < 0.05m (near ground)
- **Ball Too High**: Ball z-coordinate > target height + 1.0m (lost control)
- **Horizontal Deviation Too Far**: Absolute value of ball x-coordinate > 1.5m
- **Horizontal Deviation Too Far**: Ball x or y coordinate absolute value > 1.5m
- **Joint Velocity Too High**: Any joint angular velocity > 2π rad/s (360°/s)
- **Timeout**: Episode duration exceeds maximum allowed time
---
@@ -128,3 +441,9 @@ uv run scripts/play.py --env bounce_ball
3. Height Control: Ball's height stable within target height 0.8 ± 0.1m range
4. Velocity Control: Ball's upward velocity maintained within reasonable range (0.1-1.5 m/s)
5. Stable Control: Capable of maintaining stable bouncing for 20 seconds without dropping
---
## Known Issues
- **JAX Backend Training Performance**: The JAX version currently shows suboptimal training performance. For better results, it is recommended to use the PyTorch backend for this environment.

167
docs/source/en/user_guide/demo/bring_ball.md Normal file
View File

@@ -0,0 +1,167 @@
# Manipulator Bring Ball
Bring Ball is a classic Manipulator task from the DeepMind Control Suite. A planar hand with a thumb and finger must grasp a ball and move it to a target ball position. MotrixLab currently provides one Bring Ball environment:
- `dm-manipulator-bring-ball`: grasp the ball and move it to the target position
```{video} /_static/videos/bring-ball.mp4
:poster: _static/images/poster/bring-ball.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
---
## Task Description
Bring Ball is a planar (x-z) grasp-and-transport task:
- The hand has 4 arm joints (`arm_root`, `arm_shoulder`, `arm_elbow`, `arm_wrist`) plus thumb/finger joints
- Grasping is driven by the `grasp` tendon, coupling the `thumb` and `finger` joints
- The ball can slide in the plane (`ball_x`, `ball_z`) and rotate about the y-axis (`ball_y`)
- The target ball is a mocap body, sampled at reset
---
## Action Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-1.0, 1.0, (5,), float32)` |
| **Dimension** | 5 |
Actions correspond to the following actuators:
| Index | Action Meaning | Min Control | Max Control | XML Name |
| ----: | ---------------------------------- | :---------: | :---------: | :--------: |
| 0 | Root joint drive | -1 | 1 | `root` |
| 1 | Shoulder joint drive | -1 | 1 | `shoulder` |
| 2 | Elbow joint drive | -1 | 1 | `elbow` |
| 3 | Wrist joint drive | -1 | 1 | `wrist` |
| 4 | Grasp drive (thumb/finger coupled) | -1 | 1 | `grasp` |
---
## Observation Space
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (41,), float32)` |
| **Dimension** | 41 |
The observation vector is composed of the following parts (in order):
| Part | Content Description | Dim | Notes |
| -------------- | ----------------------------- | --- | ------------------------------------ |
| **arm_pos** | `sin`/`cos` of 8 joint angles | 16 | Joint order listed below |
| **arm_vel** | 8 joint velocities | 8 | Same order as arm_pos |
| **touch** | `log(1 + touch)` sensors | 5 | palm/finger/thumb/fingertip/thumbtip |
| **hand_pos** | Grasp site world position | 3 | x, y, z |
| **object_pos** | Ball position | 3 | x, y, z |
| **target_pos** | Target ball position | 3 | x, y, z |
| **rel** | `object_pos - target_pos` | 3 | Relative position |
| Index | Observation Range | Dim | Notes |
| ----- | ----------------------------------------------- | --- | --------------------------------- |
| 0-15 | `sin`/`cos` of 8 joint angles | 16 | Joint order: arm_root -> thumbtip |
| 16-23 | 8 joint velocities | 8 | Same order as above |
| 24-28 | Touch: palm, finger, thumb, fingertip, thumbtip | 5 | `log(1 + touch)` |
| 29-31 | Grasp position (x, y, z) | 3 | hand_pos |
| 32-34 | Ball position (x, y, z) | 3 | object_pos |
| 35-37 | Target ball position (x, y, z) | 3 | target_pos |
| 38-40 | Ball relative position (x, y, z) | 3 | rel |
Joint order: `arm_root`, `arm_shoulder`, `arm_elbow`, `arm_wrist`, `finger`, `fingertip`, `thumb`, `thumbtip`.
---
## Reward Function Design
Bring Ball uses shaped rewards with multiple components and penalties:
```python
# R1: Reach - fingertips approach the ball
r_reach = tolerance(avg_tip_dist)
# R2: Orient - palm points toward the ball
r_orient = clip(1 - orient_bound + dot(hand_dir, unit_vec_to_ball), 0..1)
# R3: Pause - reduce arm jitter when close to the ball
r_pause = tolerance(arm_speed_step) * is_close_to_ball
# R4: Close - grasp intent and contact together
r_close = r_close_intent * (approach_or_grasp)
# R5: Lift & Transport - height and target distance
r_lift_height = tolerance(ball_z)
r_transport = tolerance(move_dist_to_target)
r_lift = mix(r_lift_height, r_transport)
# Precision/Progress
r_precision = tolerance(move_dist_to_target, gaussian)
r_progress = (prev_dist - curr_dist) * scale
# Penalties
penalty_side + penalty_hover
```
Default weights (from `BringBallCfg`):
- reach 1.0, orient 1.5, pause 0.5, close 2.0, lift 6.0, precision 1.0
- lift mixes `lift_height_weight` and `transport_weight`
- progress reward is controlled by `transport_progress_scale`
---
## Initial State
- **Arm initialization**: uses the default model pose (`randomize_arm=False`), thumb/finger are symmetric
- **Target position**: `x in [-0.4, 0.4]`, `z in [0.1, 0.4]`, `y = 0.001`
- **Ball position**: `x in [-0.4, 0.4]`, `z in [0.2, 0.7]`, with a minimum hand distance
- **Physics settling**: performs settle steps after reset (`settle_steps=300`)
---
## Episode Termination Conditions
- Terminate if any `NaN` appears in observations
---
## Usage Guide
### 1. Environment Preview (random actions)
```bash
uv run scripts/view.py --env dm-manipulator-bring-ball
```
### 2. Start Training
```bash
uv run scripts/train.py --env dm-manipulator-bring-ball --train-backend torch
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/dm-manipulator-bring-ball
```
### 4. Test Training Results
```bash
uv run scripts/play.py --env dm-manipulator-bring-ball
```
---
## Expected Training Results
1. The hand consistently reaches and grasps the ball
2. The ball is lifted off the ground and held at a stable height
3. The ball is reliably moved close to the target ball position

5
docs/source/en/user_guide/demo/dm_cheetah.md
View File

@@ -127,5 +127,6 @@ uv run scripts/play.py --env dm-cheetah
## Expected Training Results
1. Stable horizontal speed approaching or exceeding 30.0 m/s
2. Maintaining an upright torso and coordinated gait
1. Run at a stable horizontal speed close to or exceeding 10.0 m/s
2. Maintain torso stability and coordinated gait, running long distances without falling
3. Running posture close to that of a real cheetah, with a sense of extension during the run

179
docs/source/en/user_guide/demo/dm_finger.md Normal file
View File

@@ -0,0 +1,179 @@
# Finger Manipulation
Finger is a classic manipulation task from the DeepMind Control Suite. A two-link “finger” applies torques to interact with a rotating spinner. MotrixLab currently provides three Finger environments:
```{video} /_static/videos/dm_finger_spin.mp4
:poster: _static/images/poster/dm_finger_spin.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
```{video} /_static/videos/dm_finger_turn.mp4
:poster: _static/images/poster/dm_finger_turn.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
- `dm-finger-spin`: make the spinner rotate continuously in the target direction
- `dm-finger-turn-easy`: align the spinner tip (`tip`) with a target point (larger target radius)
- `dm-finger-turn-hard`: same as Turn, but with a smaller target radius
---
## Task Description
Finger is a planar (x-z) interaction task:
- The finger has 2 actuated hinge joints: `proximal` and `distal`
- The spinner rotates around joint `hinge`, and `tip` denotes the spinner tip position
- For Turn tasks, a target point is sampled around the spinner at the beginning of each episode
---
## Action Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-1.0, 1.0, (2,), float32)` |
| **Dimension** | 2 |
The actions correspond to:
| Index | Action Description | Min Control | Max Control | XML Name | Joint Type |
| ----- | ---------------------------------- | ----------- | ----------- | -------- | ---------- |
| 0 | Torque applied to `proximal` joint | -1 | 1 | proximal | hinge |
| 1 | Torque applied to `distal` joint | -1 | 1 | distal | hinge |
---
## Observation Space
MotrixLab follows dm_control-style observations, but flattens them into a single vector.
### Spin Observation Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-inf, inf, (9,), float32)` |
| **Dimension** | 9 |
The observation vector contains (in order):
- **position (4)**: `qpos(proximal, distal)` + `tip_xz` (tip position relative to the spinner in x-z)
- **velocity (3)**: `qvel(proximal, distal, hinge)` (hinge velocity is used by Spin reward)
- **touch (2)**: `log(1 + touchtop)`, `log(1 + touchbottom)`
### Turn Observation Space
| Item | Details |
| ------------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (12,), float32)` |
| **Dimension** | 12 |
Compared to Spin, Turn adds:
- **target_position (2)**: target position relative to the spinner in x-z
- **dist_to_target (1)**: signed distance from tip to the target sphere surface (negative means “inside”)
---
## Reward Function Design
### Spin
In dm_control, Spin is typically defined with a sparse threshold on spinner angular velocity. MotrixLab defaults to a dense/shaped reward for easier training, while also logging the sparse version:
```text
spin_sparse = 1 if hinge_velocity <= -15 else 0
spin = clip(-hinge_velocity / 15, 0, 1)
```
### Turn (Easy / Hard)
Turn aims to bring the spinner tip into a target sphere around the spinner:
- `turn_sparse = 1` when `dist_to_target <= 0`
- MotrixLab defaults to a shaped reward based on distance-to-target (exponential decay), and adds auxiliary terms to reduce “no-contact” failure modes and action jitter:
- approach-to-spinner shaping
- touch bonus
- action magnitude / action change penalties
The final shaped reward is clipped to `[0, 1]`.
---
## Initial State
- `proximal`, `distal` joint angles are sampled uniformly within joint limits
- spinner `hinge` angle is sampled uniformly in `[-pi, pi]`
- for Turn tasks, the target is sampled around the spinner on the x-z plane at reset
---
## Episode Termination Conditions
### Termination
If `NaN` appears in the observations
---
## Usage Guide
### 1. Environment Preview (random actions)
```bash
uv run scripts/view.py --env dm-finger-spin
```
```bash
uv run scripts/view.py --env dm-finger-turn-easy
```
```bash
uv run scripts/view.py --env dm-finger-turn-hard
```
### 2. Start Training
```bash
uv run scripts/train.py --env dm-finger-spin --train-backend torch
```
```bash
uv run scripts/train.py --env dm-finger-turn-easy --train-backend torch
```
```bash
uv run scripts/train.py --env dm-finger-turn-hard --train-backend torch
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/dm-finger-spin
```
### 4. Test Training Results
`scripts/play.py` will auto-discover the latest `best_agent.*` under `runs/{env-name}/` (or you can pass `--policy` explicitly):
```bash
uv run scripts/play.py --env dm-finger-turn-hard
```
---
## Expected Training Results
1. `dm-finger-spin`: stable continuous rotation in the target direction
2. `dm-finger-turn-easy`: consistent contact and alignment with the (larger) target region, with reduced jitter
3. `dm-finger-turn-hard`: successful alignment with a smaller target region, typically requiring more training and better contact behavior

236
docs/source/en/user_guide/demo/dm_humanoid.md Normal file
View File

@@ -0,0 +1,236 @@
# 3D Humanoid Robot
The 3D Humanoid Robot (Humanoid) is a classic bipedal locomotion task from DeepMind Control Suite. The goal is to train a simulated 3D humanoid robot to achieve standing, walking, and running by controlling joint torques.
```{video} /_static/videos/dm_humanoid_run.mp4
:poster: _static/images/poster/dm_humanoid_run.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
## Task Description
Humanoid is a 3D bipedal humanoid robot task. The robot consists of a head, torso, two arms, and two legs, with 21 controlled joints (actuators). The agent controls the robot by applying torques to these joints to achieve standing balance, forward walking, or fast running. This task requires coordinated bipedal gait, balance control, and 3D spatial posture stability.
---
## Action Space
| Item | Details |
| -------- | -------------------------------- |
| **Type** | `Box(-1.0, 1.0, (21,), float32)` |
| **Dim** | 21 |
Actions correspond to the following:
| Index | Action (Joint Torque) | Min | Max | XML Name |
| ----: | ------------------------------ | :--: | :-: | :---------------- |
| 0 | Abdomen Y-axis rotation torque | -1.0 | 1.0 | `abdomen_y` |
| 1 | Abdomen Z-axis rotation torque | -1.0 | 1.0 | `abdomen_z` |
| 2 | Abdomen X-axis rotation torque | -1.0 | 1.0 | `abdomen_x` |
| 3 | Right hip X-axis torque | -1.0 | 1.0 | `right_hip_x` |
| 4 | Right hip Z-axis torque | -1.0 | 1.0 | `right_hip_z` |
| 5 | Right hip Y-axis torque | -1.0 | 1.0 | `right_hip_y` |
| 6 | Right knee torque | -1.0 | 1.0 | `right_knee` |
| 7 | Right ankle X-axis torque | -1.0 | 1.0 | `right_ankle_x` |
| 8 | Right ankle Y-axis torque | -1.0 | 1.0 | `right_ankle_y` |
| 9 | Left hip X-axis torque | -1.0 | 1.0 | `left_hip_x` |
| 10 | Left hip Z-axis torque | -1.0 | 1.0 | `left_hip_z` |
| 11 | Left hip Y-axis torque | -1.0 | 1.0 | `left_hip_y` |
| 12 | Left knee torque | -1.0 | 1.0 | `left_knee` |
| 13 | Left ankle X-axis torque | -1.0 | 1.0 | `left_ankle_x` |
| 14 | Left ankle Y-axis torque | -1.0 | 1.0 | `left_ankle_y` |
| 15 | Right shoulder 1 torque | -1.0 | 1.0 | `right_shoulder1` |
| 16 | Right shoulder 2 torque | -1.0 | 1.0 | `right_shoulder2` |
| 17 | Right elbow torque | -1.0 | 1.0 | `right_elbow` |
| 18 | Left shoulder 1 torque | -1.0 | 1.0 | `left_shoulder1` |
| 19 | Left shoulder 2 torque | -1.0 | 1.0 | `left_shoulder2` |
| 20 | Left elbow torque | -1.0 | 1.0 | `left_elbow` |
---
## Observation Space
| Item | Details |
| -------- | -------------------------------- |
| **Type** | `Box(-inf, inf, (73,), float32)` |
| **Dim** | 73 |
The observation space of the Humanoid environment consists of the following components (in order):
| Component | Description | Dim | Notes |
| ------------------ | --------------------------------------- | --- | --------------------------------------------------------------------- |
| **joint_angles** | Joint angles (excluding root's 7 DOF) | 22 | Angles of 22 joints |
| **head_height** | Head height | 1 | Height of head relative to ground |
| **extremities** | Extremity positions (relative to torso) | 12 | Left hand, left foot, right hand, right foot (3D each, in that order) |
| **torso_vertical** | Torso vertical direction vector | 3 | Vertical direction in local coordinates |
| **com_vel** | Center of mass linear velocity | 3 | Linear velocity of torso subtree |
| **qvel** | Velocities of all joints and root | 29 | Including root's 6 DOF |
| **target_local** | Target direction (local coordinates) | 3 | Target direction in torso local frame |
---
## Reward Function Design
The Humanoid reward function varies according to task type (standing, walking, running), but all include the following core components:
### Posture Reward
```python
# Head height reward: keep head above target height (95% of stand_height, ~1.33m)
stand_reward = tolerance(head_height, bounds=(stand_height * 0.95, inf), margin=0.5)
# Torso upright reward: keep torso upright
upright_reward = tolerance(torso_upright, bounds=(0.9, inf), sigmoid="linear", margin=0.9)
# Pelvis height reward: keep pelvis at reasonable height (60% of stand_height, ~0.84m)
pelvis_height_reward = tolerance(pelvis_height, bounds=(stand_height * 0.6, inf), sigmoid="linear", margin=stand_height * 0.6)
# Posture reward = head height reward × torso upright reward × pelvis height reward
posture_reward = stand_reward * upright_reward * pelvis_height_reward
```
### Speed Reward
The speed reward calculation differs by task type:
**Standing Task (move_speed <= 0)**:
```python
# Speed reward: maintain near-zero speed
speed_reward = tolerance(actual_speed, bounds=(0, 0), margin=1.0, value_at_margin=0.01)
```
**Walking Task (0 < move_speed <= 3.0)**:
```python
# Speed reward: achieve target speed (default 1.0 m/s) in target direction (positive X-axis)
actual_speed = dot(com_vel[:2], target_direction[:2]) # Projection of velocity onto target direction
speed_reward = tolerance(actual_speed, bounds=(move_speed, move_speed), margin=move_speed, sigmoid="linear")
```
**Running Task (move_speed > 3.0)**:
```python
# Speed reward: achieve target speed (default 10.0 m/s) or above in target direction
actual_speed = dot(com_vel[:2], target_direction[:2])
speed_reward = tolerance(actual_speed, bounds=(move_speed, inf), margin=move_speed, sigmoid="linear")
```
### Energy Reward
```python
energy_reward = exp(-energy_coef * mean(ctrls ^ 2))
```
### Gait Reward
```python
# Torso heading reward: torso faces target direction
torso_heading_reward = tolerance(dot(torso_forward, target_dir), bounds=(0.9, 1.0), margin=0.3, sigmoid="linear")
# Head heading reward: head faces target direction
head_heading_reward = tolerance(dot(head_forward, target_dir), bounds=(0.9, 1.0), margin=0.3, sigmoid="linear")
# Pelvis yaw reward: pelvis faces target direction
pelvis_yaw_reward = tolerance(dot(pelvis_forward, target_dir), bounds=(0.9, 1.0), margin=0.3, sigmoid="linear")
# Pelvis level reward: pelvis remains level
pelvis_level_reward = tolerance(pelvis_up, bounds=(0.9, 1.0), margin=0.3, sigmoid="linear")
# Feet height reward: feet stay close to ground
feet_height_reward = tolerance(max_foot_height, bounds=(0.0, 0.3), margin=0.5, sigmoid="quadratic")
# Gait reward = product of all heading and posture rewards
gait_reward = torso_heading_reward * head_heading_reward * pelvis_yaw_reward * pelvis_level_reward * feet_height_reward
```
### Total Reward
```python
total_reward = posture_reward * speed_reward * energy_reward * gait_reward
```
---
## Initial State
- **Robot Position**: Torso initial height is 1.33 meters (95% of standard stand height)
- **Robot Orientation**: Torso remains upright, quaternion set to (1.0, 0.0, 0.0, 0.0)
- **Joint Angles**: Randomly initialized within joint limits
- Torso/hip base joints: Randomized in small range (±15 degrees)
- Leg joints: Symmetrically initialized, ensuring left and right legs are independently randomized, knees initially bent
- Arm joints: Symmetrically initialized, using middle 80% of joint limit ranges
- **Initial Velocities**: All joint velocities and linear velocities initialized to small random values near zero (-0.01 to 0.01)
- **Initial Controls**: All actuator controls initialized to small random values near zero (-0.02 to 0.02)
## Episode Termination Conditions
- Robot state observations contain abnormal values (NaN or Inf)
- Head too low: Head height below 50% of standard stand height (0.7 meters)
- Torso too tilted: Torso vertical component less than 0.2 (severe torso tilt)
- Extreme velocity: Absolute value of any joint velocity exceeds 200.0 rad/s or m/s
- Maximum episode duration: 25 seconds
---
## Usage Guide
### 1. Environment Preview
```bash
uv run scripts/view.py --env dm-humanoid-stand
uv run scripts/view.py --env dm-humanoid-walk
uv run scripts/view.py --env dm-humanoid-run
```
### 2. Start Training
```bash
uv run scripts/train.py --env dm-humanoid-stand
uv run scripts/train.py --env dm-humanoid-walk
uv run scripts/train.py --env dm-humanoid-run
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/dm-humanoid-walk
```
### 4. Test Training Results
```bash
uv run scripts/play.py --env dm-humanoid-stand
uv run scripts/play.py --env dm-humanoid-walk
uv run scripts/play.py --env dm-humanoid-run
```
---
## Expected Training Results
### Standing Task (dm-humanoid-stand)
1. Head height maintained in 1.3-1.5m range
2. Torso upright angle deviation less than 15 degrees
3. Able to stand stably without falling
4. Speed near zero, no significant movement
### Walking Task (dm-humanoid-walk)
1. Actual walking speed close to 1.0 m/s
2. Coordinated gait, no obvious falls
3. Able to walk continuously and stably
4. Torso and head facing target direction
### Running Task (dm-humanoid-run)
1. Running speed reaches 5.0-10.0 m/s
2. Flight phase appears (both feet off ground simultaneously)
3. Coordinated and stable gait
4. Able to maintain high-speed running posture

136
docs/source/en/user_guide/demo/pendulum.md Normal file
View File

@@ -0,0 +1,136 @@
# Pendulum
Pendulum is a single-joint swing-up and balance task. The goal is to swing the pole up and keep it inverted using one motor torque.
```{video} /_static/videos/pendulum.mp4
:poster: _static/images/poster/pendulum.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
---
## Task Description
A single-link pendulum with one hinge joint is driven by a single motor (configurable gear). The motor’s torque rotates the rod in a plane, enabling swing-up from arbitrary initial angles, inverted balance, and maintenance. Torque is limited by the actuator ctrlrange; by modulating its magnitude and direction, the policy must accumulate energy to swing up and stabilize near the inverted position while damping angular-velocity-induced oscillations.
## Action Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-1.0, 1.0, (1,), float32)` |
| **Dimension** | 1 |
---
## Observation Space
| Item | Details |
| ------------- | ------------------------------- |
| **Type** | `Box(-inf, inf, (3,), float32)` |
| **Dimension** | 3 |
Order: `cos(theta), sin(theta), angular velocity`.
---
## Reward Function Design
- Upright reward: encourages angle near π (inverted)
- Energy shaping: target energy near inverted position
- Penalties: `ang_vel^2`, `ctrl^2`, `(ctrl - prev_ctrl)^2` to reduce oscillation and aggressive actuation
---
## Initial State
- Angle randomized in `[-pi, pi]`
- Angular velocity small random noise (if configured)
- Control history (`prev_ctrl`) reset to zero
## Episode Termination Conditions
- No fall/angle termination; only NaN check
- Episode length limited by `max_episode_seconds`
---
### 1. Environment Preview
```bash
uv run scripts/view.py --env pendulum
```
### 2. Start Training
```bash
# Train with default parameters
uv run scripts/train.py --env pendulum
# Customize parallel environments
uv run scripts/train.py --env pendulum --num-envs 1024
# Enable rendering during training
uv run scripts/train.py --env pendulum --render
```
### 3. View Training Progress
```bash
uv run tensorboard --logdir runs/pendulum
```
### 4. Test Training Results
```bash
# Auto-discover best policy (recommended)
uv run scripts/play.py --env pendulum
# Manually specify a policy file
uv run scripts/play.py --env pendulum --policy runs/pendulum/nn/best_policy.pickle
```
> **Tip**: Policies are auto-selected from `runs/pendulum/`. You can override with `--policy`.
---
## Configuration Parameters
### Environment Configuration
```python
@dataclass
class PendulumEnvCfg(EnvCfg):
model_file: str = ".../pendulum.xml" # MJCF model (gear=5)
max_episode_seconds: float = 20.0
sim_dt: float = 0.0125
ctrl_dt: float = 0.025
```
### Training Configuration (PPO example)
```python
@rlcfg("pendulum")
@dataclass
class PendulumPPO(PPOCfg):
seed: int = 42
max_env_steps: int = 10_000_000
num_envs: int = 1024
learning_rate: float = 3e-4
rollouts: int = 32
learning_epochs: int = 5
mini_batches: int = 4
policy_hidden_layer_sizes: tuple[int, ...] = (64, 64)
value_hidden_layer_sizes: tuple[int, ...] = (64, 64)
```
---
## Expected Training Results
1. Pendulum can swing up and stay near inverted
2. Oscillation around upright is reduced by angular-velocity and control-change penalties

381
docs/source/en/user_guide/demo/shadow_hand_repose.md Normal file
View File

@@ -0,0 +1,381 @@
# Shadow Hand Cube Repose
## Overview
This document describes in detail the Shadow Hand dexterous manipulation cube reorientation task environment. This is a classic benchmark test in the field of robotic manipulation, requiring the robot to reorient a cube in-hand to match a randomly sampled target orientation.
```{video} /_static/videos/shadow_hand_repose.mp4
:poster: _static/images/poster/shadow_hand_repose.jpg
:nocontrols:
:autoplay:
:playsinline:
:muted:
:loop:
:width: 100%
```
---
## Environment Description
The Shadow Hand cube reorientation task is built based on the realistic Shadow Hand 24-DOF dexterous hand, designed to train robots to rotate a cube in-hand to a target pose through fine multi-finger coordination.
### Robot Structure
Shadow Hand is a high-fidelity 24-DOF dexterous hand composed of the following main parts:
- **Palm**: The base structure of the hand, fixed to the base
- **5 Fingers**:
- **Thumb**: 5 degrees of freedom, including CMC rotation, MCP, IP joints
- **Index**: 4 degrees of freedom, including MCP, PIP, DIP joints
- **Middle**: 4 degrees of freedom, including MCP, PIP, DIP joints
- **Ring**: 4 degrees of freedom, including MCP, PIP, DIP joints
- **Little**: 4 degrees of freedom, including MCP, PIP, DIP joints
- **Actuator Configuration**: 20 actuated joints (4 coupled joints driven by other joints)
### Finger Joint Details
Each finger (except thumb) contains:
- **MCP Joint**: Metacarpophalangeal joint, 2 degrees of freedom (abduction/adduction + flexion)
- **PIP Joint**: Proximal interphalangeal joint, 1 degree of freedom (flexion)
- **DIP Joint**: Distal interphalangeal joint, 1 degree of freedom (flexion)
Thumb contains:
- **CMC Joint**: Carpometacarpal joint, 2 degrees of freedom
- **MCP Joint**: Metacarpophalangeal joint, 1 degree of freedom
- **IP Joint**: Interphalangeal joint, 1 degree of freedom
### Environment Objects
- **Cube**: 50mm \* 50mm \* 50mm cube
- Mass: approximately 0.028 kg
- Friction coefficient: 1.2
- Initial position: above palm center `(0.33, 0.00, 0.295)` meters
- **Target Visualization**: Semi-transparent target pose indicator (mocap body)
### Task Goals
The robot needs to complete the following operation goals:
1. **Maintain Grasp**: Maintain stable grip of the cube in-hand
2. **Perceive Goal**: Observe target pose (visualization indicator)
3. **Fine Manipulation**: Rotate cube through multi-finger coordination
4. **Pose Alignment**: Rotate cube pose to target pose (tolerance <20>0.1 radian)
---
## Action Space
Action space is `Box(-1, 1, (20,), float32)`, representing position control commands applied to 20 actuated joints (normalized).
### Control Mode
The environment uses position control mode, actions are converted to joint target positions through:
```
1. Scale from [-1, 1] to actuator control range
2. Optional: action smoothing (moving average filter)
3. Clip to joint limits
4. Apply to simulator actuators
```
### Action Processing Flow
```python
# 1. Scale to actuator limits
targets = scale(actions, lower_limits, upper_limits)
# 2. Action smoothing (optional)
if act_moving_average < 1.0:
targets = α * targets + (1-α) * prev_actions
# 3. Clip to limits
targets = clip(targets, lower_limits, upper_limits)
# 4. Apply control
actuator_ctrls = targets
```
### Action Dimension Details
| Index | Finger | Joint | DOF | Description |
| ----- | ------ | ----- | --- | ------------- |
| 0-4 | Thumb | J0-J4 | 5 | CMC, MCP, IP |
| 5-8 | Index | J0-J3 | 4 | MCP, PIP, DIP |
| 9-12 | Middle | J0-J3 | 4 | MCP, PIP, DIP |
| 13-16 | Ring | J0-J3 | 4 | MCP, PIP, DIP |
| 17-20 | Little | J0-J3 | 4 | MCP, PIP, DIP |
---
## Observation Space
Observation space is `Box(-inf, inf, (157,), float32)`, containing robot proprioceptive information, cube state, target state, and fingertip state.
### Observation Components
Observation vector consists of the following parts (in order):
#### 1. Hand Joint State (48 dimensions)
- **Joint Position (24 dims)**: Unscaled raw joint angles
- **Joint Velocity (24 dims)**: Joint angular velocities scaled by 0.2
#### 2. Cube State (17 dimensions)
- **Position (3 dims)**: Cube position in world coordinates `(x, y, z)`
- **Orientation (4 dims)**: Quaternion `(x, y, z, w)`
- **Linear Velocity (3 dims)**: Cube linear velocity
- **Angular Velocity (3 dims)**: Angular velocity scaled by 0.2
- **Normalization Factor**: Velocity observations multiplied by `vel_obs_scale = 0.2`
#### 3. Goal State (11 dimensions)
- **Goal Position (3 dims)**: Fixed at `(0.33, 0.00, 0.295)`
- **Goal Orientation (4 dims)**: Randomly sampled target quaternion
- **Relative Rotation (4 dims)**: Relative quaternion from cube to goal
#### 4. Fingertip State (65 dimensions)
State of 5 fingertips, 13 dimensions per fingertip:
- **Position (3 dims)**: Fingertip position in Cartesian space
- **Orientation (4 dims)**: Fingertip quaternion
- **Velocity (6 dims)**: Linear and angular velocities
**Fingertip Link Names**:
- `rh_ffdistal`: Index fingertip
- `rh_mfdistal`: Middle fingertip
- `rh_rfdistal`: Ring fingertip
- `rh_lfdistal`: Little fingertip
- `rh_thdistal`: Thumb fingertip
#### 5. Action History (20 dimensions)
- Previous action values, for temporal context in the policy
### Observation Details
| Index | Observation Content | Dimension | Range | Unit |
| ------- | -------------------------------- | --------- | ------------ | ----- |
| 0-23 | Hand joint position (unscaled) | 24 | Joint limits | rad |
| 24-47 | Hand joint velocity (0.2) | 24 | ±π/2 | rad/s |
| 48-50 | Cube position | 3 | Real | m |
| 51-54 | Cube orientation (quat w,x,y,z) | 4 | Unit norm | - |
| 55-57 | Cube linear velocity | 3 | Real | m/s |
| 58-60 | Cube angular velocity (0.2) | 3 | Real | rad/s |
| 61-63 | Goal position | 3 | Fixed | m |
| 64-67 | Goal orientation (quat w,x,y,z) | 4 | Unit norm | - |
| 68-71 | Relative rotation (quat w,x,y,z) | 4 | Unit norm | - |
| 72-136 | Fingertip state (5\*13) | 65 | - | - |
| 137-156 | Previous action | 20 | [-1, 1] | - |
---
## Reward Function
The reward function uses a composite design with multiple reward and penalty terms.
### Main Reward Terms
1. **Rotation Alignment Reward** (core objective)
```
rot_reward = rot_reward_scale / (|rot_dist| + rot_eps)
```
- **Scaling factor**: `1.0`
- **Epsilon**: `0.1`
- **Rotation distance calculation**: Using quaternion rotation distance formula
- **Incentive**: Reward grows inversely as cube orientation approaches target
2. **Position Distance Penalty**
```
dist_reward = dist_reward_scale <20> goal_dist
```
- **Scaling factor**: `-10.0`
- **Distance calculation**: Euclidean distance from cube to goal position
- **Incentive**: Prevent cube from dropping, keep near target position
3. **Action Regularization Penalty**
```
action_penalty = action_penalty_scale <20> ||actions||<7C>
```
- **Scaling factor**: `-0.0002`
- **Purpose**: Encourage smooth, energy-efficient motion
### Conditional Rewards
4. **Success Reward**
```
if |rot_dist| d success_tolerance:
reward += reach_goal_bonus
```
- **Reward value**: `2.0`
- **Tolerance**: `0.1` radian (approximately 5.7<EFBFBD>)
- **Purpose**: Sparse reward for achieving goal alignment
5. **Drop Penalty**
```
if goal_dist e fall_dist:
reward += fall_penalty
terminated = True
```
- **Penalty value**: `0.0` (termination only, no additional penalty)
- **Distance threshold**: `0.24` meters
- **Purpose**: Terminate episode when cube is dropped
---
## Initial State
### Hand Initialization
**Position Initialization:**
The palm is fixed in the world coordinate system, position determined by the model file.
**Joint Angle Initialization:**
- Use model default joint positions
- Add uniform random noise: `[-0.2, 0.2]` radians
- Range: All 24 hand degrees of freedom
**Velocity Initialization:**
All joint velocities are initialized to zero.
### Cube Initialization
**Position Initialization:**
- Fixed position: `(0.33, 0.00, 0.295)` above palm center
- Add uniform random noise: `[-0.01, 0.01]` (<28>1cm)
**Orientation Initialization:**
- Use Shoemake method to generate uniformly distributed random quaternions
- Ensure uniform sampling on SO(3) space
**Velocity Initialization:**
All linear and angular velocities are initialized to zero.
### Goal Initialization
**Position Initialization:**
- Fixed position: `(0.33, 0.00, 0.295)` (same as cube initial position)
**Orientation Initialization:**
- Use Shoemake method to generate uniformly distributed random target quaternions
- Resampled each reset
---
## Termination Conditions
Episodes terminate under the following conditions:
1. **Drop Termination**: Cube distance from goal position e `fall_dist` (0.24m)
2. **Timeout Termination**: Reaching `max_episode_steps` (default 1000 steps)
3. **NaN Protection**: Detecting rotation distance or position distance as NaN
### Success Holding Mechanism
The environment uses a consecutive success counter:
- When rotation tolerance is satisfied, counter increments
- When `max_consecutive_successes` (50) is reached, trigger success termination and reset goal
- Rotation tolerance: `0.1` radian
---
## Usage
### Training
```bash
uv run scripts/train.py --env shadow-hand-repose
```
### Policy Evaluation
```bash
uv run scripts/play.py --env shadow-hand-repose
```
### Environment Visualization
```bash
uv run scripts/view.py --env shadow-hand-repose
```
### TensorBoard
```bash
uv run tensorboard --logdir runs/shadow-hand-repose
```
---
## Configuration Parameters
### Environment Parameters
| Parameter | Default | Description |
| --------------------- | ------- | -------------------------- |
| `max_episode_seconds` | 10.0 | Maximum episode length (s) |
| `ctrl_dt` | 0.01 | Control timestep (s) |
| `max_episode_steps` | 1000 | Maximum episode steps |
| `num_hand_dofs` | 24 | Total hand DOFs |
| `num_actuators` | 20 | Number of actuated joints |
### Reward Parameters
| Parameter | Default | Description |
| ---------------------- | ------- | ------------------------- |
| `dist_reward_scale` | -10.0 | Position distance reward |
| `rot_reward_scale` | 1.0 | Rotation alignment reward |
| `rot_eps` | 0.1 | Rotation reward epsilon |
| `action_penalty_scale` | -0.0002 | Action regularization |
| `success_tolerance` | 0.1 | Success tolerance (rad) |
| `reach_goal_bonus` | 2.0 | Success reward |
| `fall_dist` | 0.24 | Drop distance threshold |
| `fall_penalty` | 0.0 | Drop penalty |
### Reset Noise Parameters
| Parameter | Default | Description |
| ---------------------- | ------- | ---------------------------- |
| `reset_position_noise` | 0.01 | Cube position noise (m) |
| `reset_dof_pos_noise` | 0.2 | Joint position noise (rad) |
| `reset_dof_vel_noise` | 0.0 | Joint velocity noise (rad/s) |
### Observation Scaling Parameters
| Parameter | Default | Description |
| --------------- | ------- | -------------------------- |
| `vel_obs_scale` | 0.2 | Velocity observation scale |
---
## References
This environment is based on the following classic works:
- **OpenAI Dactyl** (2018): First successful in-hand manipulation sim-to-real transfer
- **Isaac Gym** (2021): High-performance GPU-accelerated physics simulation
- **Isaac Lab** (2023): Modular robot learning framework

189
docs/source/en/user_guide/getting_started/container_deployment.md Normal file
View File

@@ -0,0 +1,189 @@
# Container Deployment
This document describes how to deploy MotrixLab using Docker containers to simplify environment configuration and enable rapid deployment.
## Prerequisites
- **Docker** and **Docker Compose**: [Installation Docs](https://docs.docker.com/engine/install/)
- **NVIDIA Container Toolkit** (for GPU support): [Installation Docs](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation)
- **NVIDIA GPU**: Graphics card supporting CUDA 12.8
## Quick Start
### 1. Clone the Repository
```bash
git clone https://github.com/Motphys/MotrixLab.git
cd MotrixLab/docker
```
### 2. Start with Docker Compose
We provide a complete `docker-compose.yml` configuration file that supports one-click startup of training and TensorBoard visualization services.
```bash
# Start training and TensorBoard services
docker compose up -d
```
This will start the following services:
- **motrixlab-training**: Training container that executes reinforcement learning training tasks
- **motrixlab-tensorboard**: TensorBoard visualization service accessible via browser at http://localhost:6006
### 3. Configure Training Parameters
You can customize training configurations through environment variables:
```bash
# Set training backend (jax or torch)
export MOTRIX_TRAIN_BACKEND=jax
# Set number of parallel environments
export MOTRIX_NUM_ENVS=2048
# Set training environment name
export MOTRIX_ENV=cartpole
# Start services
docker compose up -d
```
| Environment Variable | Default | Description |
| :--------------------- | :--------- | :--------------------------------- |
| `MOTRIX_TRAIN_BACKEND` | `jax` | Training backend: `jax` or `torch` |
| `MOTRIX_NUM_ENVS` | `2048` | Number of parallel environments |
| `MOTRIX_ENV` | `cartpole` | Training environment name |
### 4. Monitor Training Progress
Training logs are automatically saved to the Docker Volume `motrixlab-data`. You can view them through:
```bash
# View training container logs
docker logs -f motrixlab-training
# Access TensorBoard
# Open in browser: http://localhost:6006
```
### 5. Stop Services
```bash
# Stop all services
docker compose down
# Stop services and remove data volumes
docker compose down -v
```
## Advanced Usage
### Building Docker Images
If you need to customize the image, you can build from source:
```bash
# Execute from project root directory
cd docker
docker build -t motphys/motrixlab:latest .
```
### Running Single Container
If you only want to run the training container without using Docker Compose:
```bash
docker run --gpus all \
-v $(pwd)/runs:/root/motrixlab/runs \
motphys/motrixlab:latest \
scripts/train.py --train-backend jax --num-envs 2048 --env cartpole
```
### Persisting Training Results
The default configuration uses the Docker Volume `motrixlab-data` to save training results. You can mount it to a host directory:
```bash
# Modify volumes configuration in docker-compose.yml
volumes:
- ./runs:/root/motrixlab/runs
```
## Image Details
Our Docker image is based on {bdg-primary-line}`NVIDIA CUDA 12.8.1` runtime environment and comes pre-installed with the following components:
- **UV Package Manager**: Fast, reliable dependency management
- **MotrixLab**: Complete reinforcement learning training framework
- **SKRL**: Reinforcement learning library supporting both JAX and PyTorch backends
- **TensorBoard**: Training process visualization tool
- **MotrixSim**: High-performance physics simulation engine
Image layer build process:
1. **Base Environment**: NVIDIA CUDA 12.8.1 Runtime + Ubuntu 24.04
2. **System Dependencies**: Install UV package manager and necessary system tools
3. **Python Dependencies**: Use UV cache mechanism for fast Python package installation
4. **Project Code**: Copy MotrixLab source code and complete dependency installation
## Troubleshooting
### GPU Not Available
If the container cannot access the GPU:
```bash
# Check NVIDIA Docker Runtime
docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi
# Verify NVIDIA Container Toolkit is correctly installed
which nvidia-container-cli
```
### Insufficient Storage Space
Clean Docker cache and unused images:
```bash
# Clean build cache
docker builder prune
# Delete unused images
docker image prune -a
# Clean all unused resources
docker system prune -a
```
## Performance Optimization
### Accelerate Builds with UV Cache
The Dockerfile uses UV's cache mount feature, which can significantly speed up rebuilds:
```bash
# Rebuild using UV cache
docker build --cache-from motphys/motrixlab:latest -t motphys/motrixlab:latest .
```
### GPU Resource Allocation
You can specify the number of GPUs to use in `docker-compose.yml`:
```yaml
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["0", "1"] # Use GPU 0 and 1
capabilities: [gpu]
```
## Next Steps
- Check out the [Quick Start Tutorial](hello_motrixlab.md) to learn basic MotrixLab usage
- Read [Training Examples](../demo/cartpole.md) for more training tasks
- Explore [Basic Framework](../tutorial/basic_frame.md) for in-depth understanding of the framework architecture

7
docs/source/en/user_guide/index.md
View File

@@ -5,6 +5,7 @@
:maxdepth: 1
getting_started/installation
getting_started/container_deployment
getting_started/hello_motrixlab
```
@@ -25,16 +26,22 @@ tutorial/training_and_result
:caption: Training Examples
:maxdepth: 1
demo/acrobot
demo/cartpole
demo/pendulum
demo/bounce_ball
demo/dm_walker
demo/dm_cheetah
demo/dm_hopper
demo/dm_reacher
demo/dm_finger
demo/dm_humanoid
demo/bring_ball
demo/locomotion_unitree_go1
demo/locomotion_unitree_go1_rough_terrain
demo/anymal_c
demo/franka_lift_cube
demo/franka_open_cabinet
demo/shadow_hand_repose
```