chore: release v0.3.0
This commit is contained in:
motphys-developers
3
docs/source/_static/images/poster/dm_lqr_2_1.jpg
Normal file
3
docs/source/_static/images/poster/dm_lqr_2_1.jpg
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:48e8e5cecf80a0b8c640a70d51c64ef8416e4c46b1ac66e7594f30819fbf71d7
|
||||
size 2525404
|
||||
3
docs/source/_static/images/poster/dm_lqr_6_2.jpg
Normal file
3
docs/source/_static/images/poster/dm_lqr_6_2.jpg
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e68b01048e54591da250d636cb7b8be1d5bb6cf8bebb3509484bf2ee4c61d386
|
||||
size 212340
|
||||
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d0e769091ae74cf540c68dd531200f776553f09f0a15eaa51d349216e5053ec2
|
||||
size 2288132
|
||||
3
docs/source/_static/images/poster/dm_quadruped_fetch.png
Normal file
3
docs/source/_static/images/poster/dm_quadruped_fetch.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:06cbce23b71c6d8ec00e67e41c57e83e75b47ac9330998c03fa6734da49a848d
|
||||
size 1788861
|
||||
3
docs/source/_static/images/poster/dm_quadruped_run.png
Normal file
3
docs/source/_static/images/poster/dm_quadruped_run.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:43e3894b9dc3a286e9983a2f4786cf47caaf4fbbf00bfb75ea61661d47b92258
|
||||
size 2066473
|
||||
3
docs/source/_static/images/poster/dm_quadruped_walk.png
Normal file
3
docs/source/_static/images/poster/dm_quadruped_walk.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:9de7df58544554f1a7b4c523490e52e016e4057794ac27112b28ebaef704534e
|
||||
size 2678511
|
||||
3
docs/source/_static/images/poster/point_mass.jpg
Normal file
3
docs/source/_static/images/poster/point_mass.jpg
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4c306cf5e29c1380b808ecb327d34fc68fead362695b241632b4858f893ab4de
|
||||
size 777697
|
||||
3
docs/source/_static/images/poster/rm65_open_cabinet.png
Normal file
3
docs/source/_static/images/poster/rm65_open_cabinet.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fe0d085abcf2e8acc9c17ac5bd9ae4019c8f625971633295d5ac78278c84cecd
|
||||
size 2923711
|
||||
3
docs/source/_static/videos/dm_lqr_2_1.mp4
Normal file
3
docs/source/_static/videos/dm_lqr_2_1.mp4
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8392355f75fef17266bb6b40169aa2fd868cd42a3649c63ab241008ab9993374
|
||||
size 519681
|
||||
3
docs/source/_static/videos/dm_lqr_6_2.mp4
Normal file
3
docs/source/_static/videos/dm_lqr_6_2.mp4
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a8b679d72551575db86e836823416be9f9e7d06a44210dacb17803467676a1a4
|
||||
size 1377670
|
||||
3
docs/source/_static/videos/dm_quadruped_escape.mp4
Normal file
3
docs/source/_static/videos/dm_quadruped_escape.mp4
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:078a36bafb869cef0244ef3e33b9eaa5c661d49eaa9d42a8534f711993ec2fd5
|
||||
size 1738757
|
||||
3
docs/source/_static/videos/dm_quadruped_fetch.mp4
Normal file
3
docs/source/_static/videos/dm_quadruped_fetch.mp4
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8b8b4097ccf1c2ac509baf2a29e37c1e288e966472cde220eae77fc47644c324
|
||||
size 508065
|
||||
3
docs/source/_static/videos/dm_quadruped_run.mp4
Normal file
3
docs/source/_static/videos/dm_quadruped_run.mp4
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:29f418ee33792ecf4621a3d9b68341d6b0addfc5a248f5556bf6b310bfa4675c
|
||||
size 3177164
|
||||
3
docs/source/_static/videos/dm_quadruped_walk.mp4
Normal file
3
docs/source/_static/videos/dm_quadruped_walk.mp4
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ecc0db612a958bcc20be30da04c131fe920818bd052eebeb3c8918551a28fe29
|
||||
size 11380668
|
||||
3
docs/source/_static/videos/point_mass.mp4
Normal file
3
docs/source/_static/videos/point_mass.mp4
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:388d28839dfa4792a9b8f5778cb7991ea00fd522990a77e6bf0b9e4b9c4580c8
|
||||
size 3663098
|
||||
3
docs/source/_static/videos/rm65_open_cabinet.mp4
Normal file
3
docs/source/_static/videos/rm65_open_cabinet.mp4
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:163f1c45d3cda32837e1f08dd403b96c8061ff1c38860e60421b7c5dd6c06222
|
||||
size 5637813
|
||||
@@ -7,8 +7,8 @@ MotrixLab is a general-purpose machine learning architecture designed for robot
|
||||
|
||||
:::{grid-item-card}
|
||||
|
||||
```{video} _static/videos/cartpole.mp4
|
||||
:poster: _static/images/poster/cartpole.jpg
|
||||
```{video} _static/videos/shadow_hand_repose.mp4
|
||||
:poster: _static/images/poster/shadow_hand_repose.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
@@ -147,8 +147,8 @@ MotrixLab is a general-purpose machine learning architecture designed for robot
|
||||
|
||||
:::{grid-item-card}
|
||||
|
||||
```{video} _static/videos/dm_reacher.mp4
|
||||
:poster: _static/images/poster/dm_reacher.jpg
|
||||
```{video} _static/videos/cartpole.mp4
|
||||
:poster: _static/images/poster/cartpole.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
@@ -185,6 +185,32 @@ MotrixLab is a general-purpose machine learning architecture designed for robot
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
:::
|
||||
:::{grid-item-card}
|
||||
|
||||
```{video} _static/videos/dm_quadruped_fetch.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_fetch.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
:::
|
||||
:::{grid-item-card}
|
||||
|
||||
```{video} _static/videos/dm_quadruped_run.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_run.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
|
||||
199
docs/source/en/user_guide/demo/dm_lqr.md
Normal file
199
docs/source/en/user_guide/demo/dm_lqr.md
Normal file
@@ -0,0 +1,199 @@
|
||||
# Linear Quadratic Regulator
|
||||
|
||||
LQR (Linear Quadratic Regulator) is a classic continuous control and stabilization task. This repository currently provides two variants:
|
||||
|
||||
- `dm-lqr-2-1`: two masses connected by a rope, with only the last mass actuated
|
||||
- `dm-lqr-6-2`: six masses connected as a chain, with only the last two masses actuated
|
||||
|
||||
The goal is to drive the whole system back to the center and keep it near equilibrium with minimal control effort.
|
||||
|
||||
```{video} /_static/videos/dm_lqr_2_1.mp4
|
||||
:poster: _static/images/poster/dm_lqr_2_1.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
```{video} /_static/videos/dm_lqr_6_2.mp4
|
||||
:poster: _static/images/poster/dm_lqr_6_2.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task Description
|
||||
|
||||
Both tasks can be viewed as one-dimensional spring-damper chain stabilization problems. Each mass has a single translational degree of freedom along the x-axis. Neighboring masses are coupled by rope-like spring forces, and the system is affected by:
|
||||
|
||||
- body damping on each mass
|
||||
- spring forces and relative damping between neighboring masses
|
||||
- a center-restoring force pulling the system toward the origin
|
||||
- control inputs applied only to the actuated terminal degrees of freedom
|
||||
|
||||
In practice:
|
||||
|
||||
- `dm-lqr-2-1` is the simpler version and is useful for verifying whether the policy can learn a stable equilibrium
|
||||
- `dm-lqr-6-2` is more difficult because the controller must propagate its effect through a longer chain
|
||||
|
||||
---
|
||||
|
||||
## Action Space
|
||||
|
||||
### dm-lqr-2-1
|
||||
|
||||
| Item | Details |
|
||||
| ------------- | ------------------------------- |
|
||||
| **Type** | `Box(-1.0, 1.0, (1,), float32)` |
|
||||
| **Dimension** | 1 |
|
||||
|
||||
| Index | Action Description | Min | Max | XML Joint |
|
||||
| ----- | -------------------------------------- | ---- | --- | --------- |
|
||||
| 0 | Control input applied to the last mass | -1.0 | 1.0 | `q1` |
|
||||
|
||||
### dm-lqr-6-2
|
||||
|
||||
| Item | Details |
|
||||
| ------------- | ------------------------------- |
|
||||
| **Type** | `Box(-1.0, 1.0, (2,), float32)` |
|
||||
| **Dimension** | 2 |
|
||||
|
||||
| Index | Action Description | Min | Max | XML Joint |
|
||||
| ----- | --------------------------------------------- | ---- | --- | --------- |
|
||||
| 0 | Control input applied to the second-last mass | -1.0 | 1.0 | `q4` |
|
||||
| 1 | Control input applied to the last mass | -1.0 | 1.0 | `q5` |
|
||||
|
||||
---
|
||||
|
||||
## Observation Space
|
||||
|
||||
The observation is formed by concatenating all positions `qpos` and velocities `qvel`.
|
||||
|
||||
### dm-lqr-2-1
|
||||
|
||||
| Item | Details |
|
||||
| ------------- | ------------------------------- |
|
||||
| **Type** | `Box(-inf, inf, (4,), float32)` |
|
||||
| **Dimension** | 4 |
|
||||
|
||||
| Index | Observation | Meaning |
|
||||
| ----- | ----------- | --------------------------- |
|
||||
| 0 | `q0` | Position of the first mass |
|
||||
| 1 | `q1` | Position of the second mass |
|
||||
| 2 | `dq0` | Velocity of the first mass |
|
||||
| 3 | `dq1` | Velocity of the second mass |
|
||||
|
||||
### dm-lqr-6-2
|
||||
|
||||
| Item | Details |
|
||||
| ------------- | -------------------------------- |
|
||||
| **Type** | `Box(-inf, inf, (12,), float32)` |
|
||||
| **Dimension** | 12 |
|
||||
|
||||
The first 6 dimensions are `q0 ~ q5`, and the last 6 dimensions are `dq0 ~ dq5`.
|
||||
|
||||
---
|
||||
|
||||
## Reward Function Design
|
||||
|
||||
The current reward is composed of state cost, velocity cost, control cost, success bonus, and out-of-bounds penalty:
|
||||
|
||||
```python
|
||||
state_cost = 0.5 * sum(qpos ** 2)
|
||||
velocity_cost = 0.5 * velocity_cost_coef * sum(qvel ** 2)
|
||||
control_cost = 0.5 * control_cost_coef * sum(action ** 2)
|
||||
|
||||
reward = 1.0 - (state_cost + velocity_cost + control_cost)
|
||||
reward += success_bonus
|
||||
reward -= out_of_bounds_penalty
|
||||
```
|
||||
|
||||
Intuitively:
|
||||
|
||||
- the farther the system is from the origin, the lower the reward
|
||||
- larger velocities reduce the reward
|
||||
- aggressive control inputs reduce the reward
|
||||
- entering a small stable region around the origin yields a success bonus
|
||||
- leaving the valid state boundary triggers an additional penalty
|
||||
|
||||
---
|
||||
|
||||
## Initial State
|
||||
|
||||
At reset:
|
||||
|
||||
- the position vector is sampled in a random direction and normalized to a fixed norm
|
||||
- all initial velocities are set to zero
|
||||
|
||||
With the current configuration:
|
||||
|
||||
- `dm-lqr-2-1` starts with position norm around `0.8`
|
||||
- `dm-lqr-6-2` starts with position norm around `1.0`
|
||||
|
||||
---
|
||||
|
||||
## Episode Termination Conditions
|
||||
|
||||
An episode terminates and resets when any of the following conditions is met:
|
||||
|
||||
- success condition is reached:
|
||||
the position norm is below the success distance threshold and the velocity norm is below the success velocity threshold
|
||||
- out-of-bounds condition is reached:
|
||||
any position exceeds the position boundary or any velocity exceeds the velocity boundary
|
||||
- the full state is sufficiently close to zero
|
||||
- `NaN` appears in the observation or action
|
||||
|
||||
---
|
||||
|
||||
## Usage Guide
|
||||
|
||||
### 1. Environment Preview
|
||||
|
||||
```bash
|
||||
uv run scripts/view.py --env dm-lqr-2-1
|
||||
uv run scripts/view.py --env dm-lqr-6-2
|
||||
```
|
||||
|
||||
### 2. Start Training
|
||||
|
||||
```bash
|
||||
uv run scripts/train.py --env dm-lqr-2-1
|
||||
uv run scripts/train.py --env dm-lqr-6-2
|
||||
```
|
||||
|
||||
### 3. View Training Progress
|
||||
|
||||
```bash
|
||||
uv run tensorboard --logdir runs/dm-lqr-2-1
|
||||
uv run tensorboard --logdir runs/dm-lqr-6-2
|
||||
```
|
||||
|
||||
### 4. Test Training Results
|
||||
|
||||
```bash
|
||||
uv run scripts/play.py --env dm-lqr-2-1
|
||||
uv run scripts/play.py --env dm-lqr-6-2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Training Results
|
||||
|
||||
### dm-lqr-2-1
|
||||
|
||||
1. The actuated mass pulls the unactuated mass back toward the center.
|
||||
2. Both positions and velocities converge to a small neighborhood of zero.
|
||||
3. The learned policy does not settle at a biased off-center equilibrium.
|
||||
|
||||
### dm-lqr-6-2
|
||||
|
||||
1. The last two actuated masses gradually pull the entire chain back toward the center.
|
||||
2. The chain remains stable without obvious divergence or persistent oscillation.
|
||||
3. Success rate increases during training while the out-of-bounds rate decreases.
|
||||
159
docs/source/en/user_guide/demo/dm_point_mass.md
Normal file
159
docs/source/en/user_guide/demo/dm_point_mass.md
Normal file
@@ -0,0 +1,159 @@
|
||||
# Point Mass Environment
|
||||
|
||||
The Point Mass environment is a simple yet fundamental 2D navigation task where an agent controls a point mass to reach a target position. This environment serves as an excellent introduction to reinforcement learning concepts and continuous action spaces.
|
||||
|
||||
```{video} /_static/videos/point_mass.mp4
|
||||
:poster: _static/images/poster/point_mass.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
## Task Description
|
||||
|
||||
The Point Mass environment is a 2D navigation task. The agent needs to control a point mass by applying forces to move it to a randomly generated target position. This task requires the agent to learn efficient navigation strategies to reach the target with minimal control cost.
|
||||
|
||||
---
|
||||
|
||||
## Action Space (Action Space)
|
||||
|
||||
| Item | Details |
|
||||
| ------------- | ------------------------------- |
|
||||
| **Type** | `Box(-1.0, 1.0, (2,), float32)` |
|
||||
| **Dimension** | 2 |
|
||||
|
||||
Actions correspond to:
|
||||
|
||||
| Index | Action Meaning (Applied Force) | Min | Max | XML Name |
|
||||
| ----: | ------------------------------ | :-: | :-: | :-------: |
|
||||
| 0 | x-direction force | -1 | 1 | `x_force` |
|
||||
| 1 | y-direction force | -1 | 1 | `y_force` |
|
||||
|
||||
---
|
||||
|
||||
## Observation Space
|
||||
|
||||
| Item | Details |
|
||||
| ------------- | ------------------------------- |
|
||||
| **Type** | `Box(-inf, inf, (9,), float32)` |
|
||||
| **Dimension** | 9 |
|
||||
|
||||
The observation space of the Point Mass environment consists of the following components (in order):
|
||||
|
||||
| Component | Description | Dimension | Notes |
|
||||
| ------------ | ---------------------------- | --------- | ----- |
|
||||
| **Position** | Point mass x, y coordinates | 2 | |
|
||||
| **Velocity** | Point mass x, y velocities | 2 | |
|
||||
| **Target** | Target x, y coordinates | 2 | |
|
||||
| **Distance** | Distance vector to target | 2 | |
|
||||
| **Distance** | Euclidean distance to target | 1 | |
|
||||
|
||||
---
|
||||
|
||||
## Reward Function Design
|
||||
|
||||
The Point Mass environment's reward function consists of the following components:
|
||||
|
||||
### Distance Reward
|
||||
|
||||
```python
|
||||
# Exponential distance reward - stronger as agent gets closer
|
||||
distance_reward = np.exp(-10 * dist_to_target)
|
||||
```
|
||||
|
||||
### Target Arrival and Stay Reward
|
||||
|
||||
```python
|
||||
# Large bonus for reaching target
|
||||
target_bonus = 100.0 * in_target
|
||||
|
||||
# Continuous reward for staying in target
|
||||
continuous_reward = 30.0 * in_target
|
||||
```
|
||||
|
||||
### Control and Path Optimization
|
||||
|
||||
```python
|
||||
# Penalty for distance from target center when inside target
|
||||
center_penalty = np.where(in_target, 10.0 * dist_to_target, 0.0)
|
||||
|
||||
# Control penalty to encourage smooth movement
|
||||
control_penalty = 0.1 * vel_magnitude
|
||||
|
||||
# Path optimization reward for straight-line movement
|
||||
path_reward = 0.5 * direction_alignment
|
||||
```
|
||||
|
||||
### Total Reward Calculation
|
||||
|
||||
```python
|
||||
# Combine all reward components
|
||||
rwd = distance_reward + target_bonus + continuous_reward + path_reward - center_penalty - control_penalty
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Initial State
|
||||
|
||||
- Point mass position randomly initialized within [-1.0, 1.0]
|
||||
- Target position randomly initialized within [-1.5, 1.5]
|
||||
- Point mass velocity initialized to 0
|
||||
|
||||
## Episode Termination Conditions
|
||||
|
||||
- Point mass reaches target and stays for 0.5 seconds
|
||||
- Simulation time reaches 10 seconds
|
||||
- Observation contains abnormal values (NaN)
|
||||
|
||||
---
|
||||
|
||||
## Usage Guide
|
||||
|
||||
### 1. Environment Preview
|
||||
|
||||
```bash
|
||||
uv run scripts/view.py --env point_mass
|
||||
```
|
||||
|
||||
### 2. Start Training
|
||||
|
||||
```bash
|
||||
uv run scripts/train.py --env point_mass
|
||||
```
|
||||
|
||||
### 3. View Training Progress
|
||||
|
||||
```bash
|
||||
uv run tensorboard --logdir runs/point_mass
|
||||
```
|
||||
|
||||
### 4. Test Training Results
|
||||
|
||||
```bash
|
||||
uv run scripts/play.py --env point_mass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Training Results
|
||||
|
||||
### Navigation Performance
|
||||
|
||||
1. Agent learns to move directly towards the target
|
||||
2. Smooth movement with minimal control effort
|
||||
3. Consistent target reaching within episode duration
|
||||
|
||||
### Learning Progress
|
||||
|
||||
1. Rapid initial learning phase as agent discovers basic navigation
|
||||
2. Gradual refinement of control strategy
|
||||
3. Stable performance across different target positions
|
||||
|
||||
### Behavior Characteristics
|
||||
|
||||
1. Efficient path planning towards target
|
||||
2. Smooth approach to target center
|
||||
3. Minimal overshooting or oscillatory behavior
|
||||
290
docs/source/en/user_guide/demo/dm_quadruped.md
Normal file
290
docs/source/en/user_guide/demo/dm_quadruped.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Quadruped Robot
|
||||
|
||||
The Quadruped robot is a classic continuous control task in the DeepMind Control Suite. In MotrixLab, the `motrix_envs/src/motrix_envs/basic/quadruped` directory currently registers four directly trainable tasks: flat-ground walking `dm-quadruped-walk`, flat-ground running `dm-quadruped-run`, rough-terrain escape `dm-quadruped-escape`, and flat-ground ball pushing `dm-quadruped-fetch`.
|
||||
|
||||
## Task Preview
|
||||
|
||||
### Walk
|
||||
|
||||
```{video} /_static/videos/dm_quadruped_walk.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_walk.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
### Run
|
||||
|
||||
```{video} /_static/videos/dm_quadruped_run.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_run.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
### Escape
|
||||
|
||||
```{video} /_static/videos/dm_quadruped_escape.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_escape.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
### Fetch
|
||||
|
||||
```{video} /_static/videos/dm_quadruped_fetch.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_fetch.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
## Task Overview
|
||||
|
||||
| Environment ID | Task Goal | Model File | Target Speed | Observation Dimension |
|
||||
| --------------------- | ---------------------------------------------------------------------------- | ---------------------- | ------------ | --------------------- |
|
||||
| `dm-quadruped-walk` | Walk forward stably on flat ground while maintaining heading | `quadruped_walk.xml` | 0.5 m/s | 54 |
|
||||
| `dm-quadruped-run` | Run at high speed on flat ground while maintaining stable posture | `quadruped_walk.xml` | 5.0 m/s | 54 |
|
||||
| `dm-quadruped-escape` | Escape outward from the origin area as quickly as possible on uneven terrain | `quadruped_escape.xml` | 3.0 m/s | 57 |
|
||||
| `dm-quadruped-fetch` | Push a ball into the target area on flat ground | `quadruped_fetch.xml` | 2.0 m/s | 66 |
|
||||
|
||||
## Task Description
|
||||
|
||||
Quadruped is a 3D quadruped robot task. The robot consists of one torso and four legs, and each leg has control dimensions related to yaw, lift, and extension. The underlying XML defines the hip, knee, and ankle joint structure, while the action layer uses a coupled actuator design with 3 actuators per leg:
|
||||
|
||||
- `yaw`: controls leg yaw
|
||||
- `lift`: controls leg lifting through tendon coupling
|
||||
- `extend`: controls leg extension/retraction through tendon coupling
|
||||
|
||||
`walk` and `run` use the same flat-ground model, with the main difference being the target speed. `escape` uses `quadruped_escape.xml` with a heightfield terrain and requires the robot to move away from the world origin quickly while maintaining an upright torso and stable locomotion. `fetch` uses `quadruped_fetch.xml`, which adds a free ball and a target region to the scene, requiring the robot to first approach a suitable position and then push the ball toward the goal.
|
||||
|
||||
---
|
||||
|
||||
## Action Space
|
||||
|
||||
| Item | Details |
|
||||
| ------------- | -------------------------------- |
|
||||
| **Type** | `Box(low, high, (12,), float32)` |
|
||||
| **Dimension** | 12 |
|
||||
|
||||
Actions are arranged leg by leg, and each leg contains three actuators: `yaw / lift / extend`.
|
||||
|
||||
| Index | Action Meaning | Min Value | Max Value | Corresponding Actuator |
|
||||
| ----: | --------------------------------- | :-------: | :-------: | ---------------------- |
|
||||
| 0 | Front-left leg yaw control | -1.0 | 1.0 | `yaw_front_left` |
|
||||
| 1 | Front-left leg lift control | -1.0 | 1.1 | `lift_front_left` |
|
||||
| 2 | Front-left leg extension control | -0.8 | 0.8 | `extend_front_left` |
|
||||
| 3 | Front-right leg yaw control | -1.0 | 1.0 | `yaw_front_right` |
|
||||
| 4 | Front-right leg lift control | -1.0 | 1.1 | `lift_front_right` |
|
||||
| 5 | Front-right leg extension control | -0.8 | 0.8 | `extend_front_right` |
|
||||
| 6 | Rear-right leg yaw control | -1.0 | 1.0 | `yaw_back_right` |
|
||||
| 7 | Rear-right leg lift control | -1.0 | 1.1 | `lift_back_right` |
|
||||
| 8 | Rear-right leg extension control | -0.8 | 0.8 | `extend_back_right` |
|
||||
| 9 | Rear-left leg yaw control | -1.0 | 1.0 | `yaw_back_left` |
|
||||
| 10 | Rear-left leg lift control | -1.0 | 1.1 | `lift_back_left` |
|
||||
| 11 | Rear-left leg extension control | -0.8 | 0.8 | `extend_back_left` |
|
||||
|
||||
---
|
||||
|
||||
## Observation Space
|
||||
|
||||
| Environment | Details |
|
||||
| --------------------- | -------------------------------- |
|
||||
| `dm-quadruped-walk` | `Box(-inf, inf, (54,), float32)` |
|
||||
| `dm-quadruped-run` | `Box(-inf, inf, (54,), float32)` |
|
||||
| `dm-quadruped-escape` | `Box(-inf, inf, (57,), float32)` |
|
||||
| `dm-quadruped-fetch` | `Box(-inf, inf, (66,), float32)` |
|
||||
|
||||
All four tasks share nearly the same proprioceptive observations. `escape` adds 3 task-related dimensions associated with the origin, while `fetch` adds ball state and target position information:
|
||||
|
||||
| Part | Description | Dimension | `walk/run` | `escape` | `fetch` |
|
||||
| ---------------------- | --------------------------------------------------------- | --------- | ---------- | -------- | ------- |
|
||||
| **egocentric dof pos** | Body generalized position state | 16 | Yes | Yes | Yes |
|
||||
| **egocentric dof vel** | Body generalized velocity state | 16 | Yes | Yes | Yes |
|
||||
| **actuator ctrl** | Current 12-dimensional actuator controls | 12 | Yes | Yes | Yes |
|
||||
| **torso velocity** | Torso linear velocity sensor `velocimeter` | 3 | Yes | Yes | Yes |
|
||||
| **torso upright** | Scalar representing torso uprightness | 1 | Yes | Yes | Yes |
|
||||
| **imu** | IMU acceleration and angular velocity | 6 | Yes | Yes | Yes |
|
||||
| **origin** | World origin position in the body frame | 3 | No | Yes | No |
|
||||
| **ball state** | Ball position, relative linear velocity, angular velocity | 9 | No | No | Yes |
|
||||
| **target** | Relative target position in the body frame | 3 | No | No | Yes |
|
||||
|
||||
The XML also defines foot force/torque sensors and a center-of-mass sensor, but these values are not directly concatenated into the default observation in the current implementation.
|
||||
|
||||
---
|
||||
|
||||
## Reward Function Design
|
||||
|
||||
All four tasks use torso uprightness as the core constraint. In the implementation, `upright_reward` is computed first from `torso_upright`, encouraging the robot to keep the body close to upright.
|
||||
|
||||
### Walk / Run
|
||||
|
||||
`dm-quadruped-walk` and `dm-quadruped-run` use the same reward structure, with different target speeds:
|
||||
|
||||
- `walk` tracks `0.5 m/s`
|
||||
- `run` tracks `5.0 m/s`
|
||||
|
||||
The total reward is composed of:
|
||||
|
||||
```python
|
||||
# Speed reward: reach the target forward speed
|
||||
# Posture reward: keep the torso upright
|
||||
# Auxiliary rewards: height, lateral stability, heading alignment, action smoothness
|
||||
# Penalties: backward motion, excessive vertical speed, excessive roll/pitch angular velocity, deviation from default posture
|
||||
total_reward = upright_reward * move_reward + shaping_terms - penalty_terms
|
||||
```
|
||||
|
||||
The main shaping and penalty terms include:
|
||||
|
||||
- `height_reward`: encourages the torso to stay near the standing height
|
||||
- `lateral_reward`: suppresses excessive lateral velocity
|
||||
- `heading_reward`: encourages forward motion along the +X direction
|
||||
- `smooth_reward`: penalizes large action changes between consecutive timesteps
|
||||
- `backward_penalty`: suppresses backward movement
|
||||
- `lin_vel_z_penalty` and `ang_vel_xy_penalty`: suppress vertical bouncing and excessive torso roll/pitch
|
||||
- `similar_to_default_penalty`: encourages joint posture to stay reasonably close to the default standing pose
|
||||
|
||||
### Escape
|
||||
|
||||
`dm-quadruped-escape` adds a task term for escaping away from the origin area on top of the locomotion reward. This task uses the heightfield terrain in `quadruped_escape.xml`:
|
||||
|
||||
```python
|
||||
# Base locomotion reward
|
||||
# + reward for getting farther from the origin
|
||||
# + reward for outward radial speed
|
||||
total_reward = locomotion_reward + upright_reward * escape_reward + radial_speed_reward
|
||||
```
|
||||
|
||||
Additional task terms include:
|
||||
|
||||
- `escape_reward`: rewards the robot based on its distance from the origin area
|
||||
- `radial_speed_reward`: encourages acceleration along the outward direction away from the origin
|
||||
|
||||
This makes `escape` require not only fast locomotion, but also correct outward motion on rough terrain.
|
||||
|
||||
### Fetch
|
||||
|
||||
`dm-quadruped-fetch` uses a task-specific shaping structure centered on positioning and ball pushing. In the current implementation, the reward is mainly based on the geometric relationship between the robot, the ball, and the target:
|
||||
|
||||
```python
|
||||
# Positioning stage: encourage the robot to move behind or slightly behind the ball
|
||||
# Ready stage: encourage facing the ball, getting close to it, and aligning with the ball-target line
|
||||
# Pushing stage: encourage the ball to roll toward the target and eventually enter the target area
|
||||
# Penalties: moving in the wrong direction, pushing the ball away from the target, getting the legs too close to the ball
|
||||
total_reward = stage_terms + ready_terms + push_terms - penalty_terms
|
||||
```
|
||||
|
||||
The main terms include:
|
||||
|
||||
- `stage_move`: encourages movement toward the current stage waypoint
|
||||
- `stage_reach`: encourages reaching a suitable waypoint behind or to the side of the ball
|
||||
- `behind_align`: encourages the robot to position itself behind the ball relative to the target
|
||||
- `face_ball`: encourages the torso heading to point toward the ball
|
||||
- `near_ball`: encourages the robot to approach the ball
|
||||
- `ready` and `ready_gate`: combine position, orientation, and distance into a readiness signal for active pushing
|
||||
- `fetch`: encourages the ball to get closer to the target region
|
||||
- `push`: encourages the ball to move along the target direction
|
||||
- `backward`: penalizes moving opposite to the current stage target
|
||||
- `away`: penalizes pushing the ball away from the target
|
||||
- `leg_ball`: penalizes leg geometry getting too close to the ball, reducing ball trapping and squeezing behavior
|
||||
|
||||
In addition, `fetch` uses a `stability` gate on torso uprightness and torso height so the agent cannot easily exploit obviously collapsed poses to collect task reward.
|
||||
|
||||
---
|
||||
|
||||
## Initial State
|
||||
|
||||
- `walk`, `run`, and `escape` reset from the default quadruped standing pose defined in the XML
|
||||
- For these three tasks, the root orientation is fixed to the initial heading instead of being randomly rotated
|
||||
- `fetch` randomizes the robot position and yaw on the plane, and also randomizes the ball position on the ground
|
||||
- All tasks initialize joint velocities, and in `fetch` also ball velocities, to zero
|
||||
- During reset, the robot is automatically lifted until there is no initial penetration/contact with the ground
|
||||
|
||||
## Episode Termination Conditions
|
||||
|
||||
- Maximum episode duration is 20 seconds
|
||||
- The episode terminates when `NaN` appears in the observation
|
||||
- `walk`, `run`, and `escape` do not currently define a separate fall termination condition
|
||||
- `fetch` terminates early when the robot has clearly fallen, based on low torso uprightness or low torso height
|
||||
- The current implementation does not yet define a separate success termination condition for "ball enters the target area"
|
||||
|
||||
---
|
||||
|
||||
## Usage Guide
|
||||
|
||||
### 1. Environment Preview
|
||||
|
||||
```bash
|
||||
uv run scripts/view.py --env dm-quadruped-walk
|
||||
uv run scripts/view.py --env dm-quadruped-run
|
||||
uv run scripts/view.py --env dm-quadruped-escape
|
||||
uv run scripts/view.py --env dm-quadruped-fetch
|
||||
```
|
||||
|
||||
### 2. Start Training
|
||||
|
||||
```bash
|
||||
uv run scripts/train.py --env dm-quadruped-walk
|
||||
uv run scripts/train.py --env dm-quadruped-run
|
||||
uv run scripts/train.py --env dm-quadruped-escape
|
||||
uv run scripts/train.py --env dm-quadruped-fetch
|
||||
```
|
||||
|
||||
### 3. View Training Progress
|
||||
|
||||
```bash
|
||||
uv run tensorboard --logdir runs/dm-quadruped-walk
|
||||
uv run tensorboard --logdir runs/dm-quadruped-run
|
||||
uv run tensorboard --logdir runs/dm-quadruped-escape
|
||||
uv run tensorboard --logdir runs/dm-quadruped-fetch
|
||||
```
|
||||
|
||||
### 4. Test Training Results
|
||||
|
||||
```bash
|
||||
uv run scripts/play.py --env dm-quadruped-walk
|
||||
uv run scripts/play.py --env dm-quadruped-run
|
||||
uv run scripts/play.py --env dm-quadruped-escape
|
||||
uv run scripts/play.py --env dm-quadruped-fetch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Expected Training Results
|
||||
|
||||
### Walking Task (`dm-quadruped-walk`)
|
||||
|
||||
1. Maintain a stable forward speed close to `0.5 m/s`
|
||||
2. Keep body posture stable with small lateral sway
|
||||
3. Sustain walking along the +X direction
|
||||
|
||||
### Running Task (`dm-quadruped-run`)
|
||||
|
||||
1. Increase speed to near or above `5.0 m/s`
|
||||
2. Produce larger stride length and more explosive motions
|
||||
3. Maintain good torso stability under high-speed locomotion
|
||||
|
||||
### Escape Task (`dm-quadruped-escape`)
|
||||
|
||||
1. Move away from the origin region quickly
|
||||
2. Maintain stable footholds on the heightfield terrain without easily tipping over
|
||||
3. Move primarily outward rather than spinning in place
|
||||
|
||||
### Fetch Task (`dm-quadruped-fetch`)
|
||||
|
||||
1. First move into a reasonable position along the ball-target line instead of randomly colliding with the ball from the side
|
||||
2. Push the ball toward the target area consistently instead of kicking it away or repeatedly sending it off course
|
||||
3. Maintain better body stability during pushing, with fewer collapsed poses, flips, or leg-ball entanglement behaviors
|
||||
257
docs/source/en/user_guide/demo/rm65_open_cabinet.md
Normal file
257
docs/source/en/user_guide/demo/rm65_open_cabinet.md
Normal file
@@ -0,0 +1,257 @@
|
||||
# RM65 Open Cabinet
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the `rm65-open-cabinet` manipulation task environment. The environment uses an RM65 6-DOF robotic arm with a parallel gripper. The goal is to approach the bottom drawer handle, establish a stable grasp, and pull the drawer open.
|
||||
|
||||
```{video} /_static/videos/rm65_open_cabinet.mp4
|
||||
:poster: _static/images/poster/rm65_open_cabinet.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment Description
|
||||
|
||||
This task is built on an RM65 arm and a cabinet drawer scene. Although the registered environment name is `rm65-open-cabinet`, the current implementation actually targets the bottom drawer, using `drawer_bottom_handle` and `drawer_bottom_joint` in code.
|
||||
|
||||
### Robot Structure
|
||||
|
||||
The RM65 robot in this environment contains the following major components:
|
||||
|
||||
- **Base (`base_link`)**: Fixed in front of the workspace
|
||||
- **6 arm joints**: `joint_1` to `joint_6`
|
||||
- **Parallel gripper**: The main driven gripper joint is `gripper_Left_1_Joint`, while the other 5 gripper joints follow through mimic linkage
|
||||
- **End effector (TCP)**: The `gripper` site, used to compute the relative pose between the tool center point and the drawer handle
|
||||
- **Finger contact sites**: `left_finger_pad` and `right_finger_pad`, used to evaluate handle alignment and penetration
|
||||
|
||||
### Scene Objects
|
||||
|
||||
- **Cabinet**: Contains multiple doors and drawers
|
||||
- **Target handle**: `drawer_bottom_handle`
|
||||
- **Target joint**: `drawer_bottom_joint`, with a sliding range of `0.0 ~ 0.4 m`
|
||||
|
||||
### Task Objective
|
||||
|
||||
The robot is expected to complete the following stages:
|
||||
|
||||
1. **Approach the handle**: Move the TCP close to the drawer handle
|
||||
2. **Align the pose**: Match the gripper pose to the handle pose
|
||||
3. **Secure the grasp**: Close the gripper and maintain a stable grasp
|
||||
4. **Open the drawer**: Pull the bottom drawer along its sliding direction
|
||||
|
||||
---
|
||||
|
||||
## Action Space
|
||||
|
||||
The action space is `Box(-inf, inf, (7,), float32)`.
|
||||
|
||||
The first 6 dimensions control arm joint targets, and the last dimension controls gripper opening and closing.
|
||||
|
||||
### Control Mode
|
||||
|
||||
- **Arm**: Uses `joint_target` mode by default, with normalized target actions
|
||||
Arm actions are clipped to `[-1, 1]` first, then linearly mapped to each joint control range
|
||||
- **Gripper**: Uses `binary` mode by default
|
||||
The raw action is converted to a closing probability through a Sigmoid function, then turned into a binary open/close command with hysteresis
|
||||
|
||||
### Action Dimension Details
|
||||
|
||||
| Index | Action Description | Raw Input Range | Controlled Target |
|
||||
| ----- | ------------------ | --------------- | ---------------------- |
|
||||
| 0 | Joint 1 target | `(-inf, inf)` | `joint_1` |
|
||||
| 1 | Joint 2 target | `(-inf, inf)` | `joint_2` |
|
||||
| 2 | Joint 3 target | `(-inf, inf)` | `joint_3` |
|
||||
| 3 | Joint 4 target | `(-inf, inf)` | `joint_4` |
|
||||
| 4 | Joint 5 target | `(-inf, inf)` | `joint_5` |
|
||||
| 5 | Joint 6 target | `(-inf, inf)` | `joint_6` |
|
||||
| 6 | Gripper open/close | `(-inf, inf)` | `gripper_Left_1_Joint` |
|
||||
|
||||
### Control Constraints
|
||||
|
||||
- Control period: `ctrl_dt = 0.025s`, corresponding to 40 Hz
|
||||
- The arm uses speed limits, acceleration limits, action delay, and first-order actuator lag by default
|
||||
- During training, arm delay, lag, speed limits, and acceleration limits are randomized per episode to improve sim-to-real robustness
|
||||
- The gripper uses hysteresis thresholds:
|
||||
- Open-to-close threshold: `0.78`
|
||||
- Close-to-open threshold: `0.62`
|
||||
- Minimum switching interval: `0.25s`
|
||||
|
||||
---
|
||||
|
||||
## Observation Space
|
||||
|
||||
The observation space is `Box(-inf, inf, (84,), float32)`, and the final observation is clipped to `[-5, 5]`.
|
||||
|
||||
### Observation Components
|
||||
|
||||
The observation is composed of the following 4 parts:
|
||||
|
||||
1. **Joint positions (7 dimensions)**
|
||||
- 6 arm joints
|
||||
- 1 primary gripper joint
|
||||
- All normalized to `[-1, 1]`
|
||||
2. **Joint velocities (7 dimensions)**
|
||||
- Estimated by finite differences between consecutive joint positions
|
||||
- Then divided by `2` for scaling
|
||||
3. **Target relative pose (7 dimensions)**
|
||||
- Relative position from TCP to handle
|
||||
- Relative orientation from TCP to handle in quaternion form
|
||||
4. **Action history (63 dimensions)**
|
||||
- Raw actions from the most recent `9` steps
|
||||
- `7` values per step, for `9 × 7 = 63` dimensions
|
||||
|
||||
### Observation Dimension Details
|
||||
|
||||
| Index Range | Description | Dimension |
|
||||
| ----------- | ------------------------------------ | --------- |
|
||||
| 0-6 | Normalized joint positions | 7 |
|
||||
| 7-13 | Joint velocities | 7 |
|
||||
| 14-16 | Relative position from TCP to handle | 3 |
|
||||
| 17-20 | Relative orientation quaternion | 4 |
|
||||
| 21-83 | Recent 9-step action history | 63 |
|
||||
|
||||
### Observation Noise
|
||||
|
||||
Sim-to-real observation perturbations are enabled by default, including:
|
||||
|
||||
- Joint position and velocity noise
|
||||
- Handle position and orientation noise
|
||||
- Persistent handle observation bias
|
||||
- Random handle observation dropout, optionally holding the previous observation on dropout
|
||||
|
||||
---
|
||||
|
||||
## Reward Function
|
||||
|
||||
The reward is a staged composite design that encourages approach, alignment, stable grasping, and continuous drawer opening.
|
||||
|
||||
### Main Reward Terms
|
||||
|
||||
1. **Distance reward**
|
||||
|
||||
```python
|
||||
dist_reward = 15.0 * (1 - tanh(distance / 0.4))
|
||||
```
|
||||
|
||||
Encourages the TCP to stay close to the handle.
|
||||
|
||||
2. **Orientation reward**
|
||||
|
||||
Computed from quaternion similarity between the TCP pose and the handle pose, and only applied when the TCP is sufficiently close to the handle.
|
||||
|
||||
3. **Gripper closing reward**
|
||||
|
||||
When the TCP is within `0.035m` and the two fingers are vertically aligned around the handle, closing the gripper is rewarded; otherwise it is penalized. This term is also scaled by the gripper closing amount.
|
||||
|
||||
4. **Drawer opening reward**
|
||||
|
||||
```python
|
||||
open_reward = (exp(open_dist) - 1.0) * 420.0
|
||||
```
|
||||
|
||||
This reward is only active when the robot has already grasped the handle, or has entered the grasp-maintenance phase while still staying near the handle.
|
||||
|
||||
5. **Open-distance delta reward**
|
||||
|
||||
Provides extra reward for newly gained drawer displacement at the current step, encouraging stable and continuous pulling.
|
||||
|
||||
6. **Stable grasp reward**
|
||||
|
||||
When the TCP is within `0.03m`, the gripper close ratio exceeds `0.7`, and this condition is maintained for `6` consecutive steps, the environment considers the handle grasped and provides persistent reward.
|
||||
|
||||
7. **Milestone rewards**
|
||||
|
||||
- Reward `35` when drawer opening exceeds `0.15m`
|
||||
- Additional reward `70` when drawer opening exceeds `0.22m`
|
||||
|
||||
### Penalty Terms
|
||||
|
||||
1. **Slip penalty**
|
||||
|
||||
If the environment has entered the grasping phase but the robot later loses the grasp while the drawer is already open, an additional penalty is applied.
|
||||
|
||||
2. **Finger penetration penalty**
|
||||
|
||||
Applied when the finger contact points cross the upper or lower handle boundary, discouraging unrealistic penetration.
|
||||
|
||||
3. **Gripper switching penalty**
|
||||
|
||||
Penalizes frequent gripper toggling when the TCP is near the handle.
|
||||
|
||||
4. **Action-change penalty**
|
||||
|
||||
Penalizes the squared difference between consecutive actions.
|
||||
|
||||
5. **Joint-velocity penalty**
|
||||
|
||||
Penalizes the squared sum of joint velocities, with a larger weight in later training.
|
||||
|
||||
6. **Termination penalty**
|
||||
|
||||
Applies an additional `-10.0` penalty when a termination condition is triggered.
|
||||
|
||||
---
|
||||
|
||||
## Initial State
|
||||
|
||||
### Robot Initialization
|
||||
|
||||
- The arm starts from its default zero pose
|
||||
- The gripper starts in the open state
|
||||
- No joint-position reset noise is added by default
|
||||
- All joint velocities are initialized to zero
|
||||
|
||||
### Scene Initialization
|
||||
|
||||
- The target drawer starts fully closed
|
||||
- The drawer and the rest of the cabinet remain at their default scene poses
|
||||
|
||||
### Randomized Factors
|
||||
|
||||
At reset, the environment re-samples a subset of sim-to-real parameters, including:
|
||||
|
||||
- Arm action delay
|
||||
- Arm actuator lag
|
||||
- Arm speed and acceleration limits
|
||||
- Handle observation bias
|
||||
|
||||
---
|
||||
|
||||
## Episode Termination Conditions
|
||||
|
||||
The episode terminates early if any of the following conditions is met:
|
||||
|
||||
1. **TCP moves too far behind the handle**
|
||||
- Threshold: `tcp_x - handle_x < -0.02`
|
||||
2. **Joint velocity becomes too large**
|
||||
- Any robot joint velocity magnitude exceeds `3.93 rad/s`
|
||||
|
||||
In addition, the maximum episode length is `30s`.
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### Training
|
||||
|
||||
```bash
|
||||
uv run scripts/train.py --env rm65-open-cabinet --train-backend torch
|
||||
```
|
||||
|
||||
### Policy Evaluation
|
||||
|
||||
```bash
|
||||
uv run scripts/play.py --env rm65-open-cabinet
|
||||
```
|
||||
|
||||
### TensorBoard
|
||||
|
||||
```bash
|
||||
uv run tensorboard --logdir runs/rm65_open_cabinet
|
||||
```
|
||||
@@ -1,9 +1,13 @@
|
||||
# Installation Environment
|
||||
|
||||
## Installation Requirements
|
||||
This document will guide you through the installation and configuration of MotrixLab. Please read the system requirements carefully and choose the appropriate installation method based on your use case.
|
||||
|
||||
## System Requirements
|
||||
|
||||
- **Python Version**: {bdg-danger-line}`3.10.*`
|
||||
|
||||
This project requires a specific Python version, other versions are not currently supported:
|
||||
|
||||
| Python Version | Support Status |
|
||||
| :------------: | :------------: |
|
||||
| ≤ 3.9 | ❌ |
|
||||
@@ -12,6 +16,8 @@
|
||||
|
||||
- **Package Manager**: {bdg-danger-line}`UV`
|
||||
|
||||
This project uses UV as the exclusive package management tool to provide fast, reproducible dependency management environment. For UV installation, please refer to the [official documentation](https://docs.astral.sh/uv/getting-started/installation/).
|
||||
|
||||
- **System and Architecture**:
|
||||
|
||||
- {bdg-danger-line}`Windows(x86_64)`
|
||||
@@ -26,30 +32,34 @@
|
||||
| Windows | ✅ | ✅ | 🛠️ In Development |
|
||||
```
|
||||
|
||||
## Installation Method
|
||||
## Installation Steps
|
||||
|
||||
### Clone Project
|
||||
### Clone Project Repository
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Motphys/MotrixLab.git
|
||||
cd MotrixLab
|
||||
```
|
||||
|
||||
### Install Dependencies
|
||||
### Configure Dependencies
|
||||
|
||||
Use UV to install project dependencies:
|
||||
Execute the following command to install complete dependencies:
|
||||
|
||||
```bash
|
||||
# Install all dependencies
|
||||
uv sync --all-packages --all-extras
|
||||
```
|
||||
|
||||
If you only need to install one training backend, you can choose to install a specific backend type:
|
||||
If you only need specific training frameworks, you can selectively install to reduce dependency size:
|
||||
|
||||
```bash
|
||||
# Install SKRL JAX (support Linux only)
|
||||
|
||||
# Install SKRL JAX (Linux only)
|
||||
uv sync --all-packages --extra skrl-jax
|
||||
|
||||
# Install SKRL PyTorch
|
||||
uv sync --all-packages --extra skrl-torch
|
||||
|
||||
# Install RSLRL (PyTorch only)
|
||||
uv sync --all-packages --extra rslrl
|
||||
```
|
||||
|
||||
@@ -34,14 +34,18 @@ demo/dm_walker
|
||||
demo/dm_cheetah
|
||||
demo/dm_hopper
|
||||
demo/dm_reacher
|
||||
demo/dm_lqr
|
||||
demo/dm_finger
|
||||
demo/dm_humanoid
|
||||
demo/dm_quadruped
|
||||
demo/dm_point_mass
|
||||
demo/bring_ball
|
||||
demo/locomotion_unitree_go1
|
||||
demo/locomotion_unitree_go1_rough_terrain
|
||||
demo/anymal_c
|
||||
demo/franka_lift_cube
|
||||
demo/franka_open_cabinet
|
||||
demo/rm65_open_cabinet
|
||||
demo/shadow_hand_repose
|
||||
|
||||
```
|
||||
|
||||
@@ -16,6 +16,7 @@ MotrixLab/
|
||||
│ └── registry.py # Environment registry system
|
||||
├── motrix_rl/ # Training layer: RL algorithms and configuration
|
||||
│ ├── skrl/ # SKRL framework integration (JAX/PyTorch)
|
||||
│ ├── rslrl/ # RSLRL framework integration (PyTorch)
|
||||
│ ├── base.py # RL configuration base class
|
||||
│ └── registry.py # RL configuration registry system
|
||||
└── scripts
|
||||
@@ -34,7 +35,7 @@ MotrixLab/
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Training Algorithm Layer (SKRL) │
|
||||
│ Training Algorithm Layer (SKRL / RSLRL) │
|
||||
│ PPO Trainer │ Network Architecture │ Optimizer │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
@@ -128,13 +129,15 @@ Configuration parameters play a key connecting role throughout the process:
|
||||
- **Reward Configuration** affects learning signals (reward weights, calculation methods, etc.)
|
||||
- **Training Configuration** controls algorithm behavior (network structure, learning rate, batch size, etc.)
|
||||
|
||||
## Multi-Backend Support
|
||||
## Multi-Framework Support
|
||||
|
||||
MotrixLab's layered design naturally supports multiple backends:
|
||||
MotrixLab's layered design naturally supports multiple RL frameworks:
|
||||
|
||||
- **Simulation Backends**: MotrixSim (CPU)
|
||||
- **Training Backends**: JAX and PyTorch, supporting GPU acceleration
|
||||
- **Algorithm Framework**: Mainly integrates SKRL, easy to extend to other algorithms
|
||||
- **Training Frameworks**:
|
||||
- **SKRL**: Supports JAX and PyTorch backends with GPU acceleration
|
||||
- **RSLRL**: Supports PyTorch backend with GPU acceleration
|
||||
- **Framework Selection**: Use `--rllib` parameter to choose between `skrl` (default) and `rslrl`
|
||||
|
||||
## Design Advantages
|
||||
|
||||
|
||||
@@ -7,56 +7,64 @@ This section introduces how to execute reinforcement learning training and how t
|
||||
### Basic Training Commands
|
||||
|
||||
```bash
|
||||
# Train with default parameters
|
||||
# Train with default parameters (SKRL framework)
|
||||
uv run scripts/train.py --env cartpole
|
||||
|
||||
# Specify RL framework
|
||||
uv run scripts/train.py --env cartpole --rllib skrl
|
||||
uv run scripts/train.py --env cartpole --rllib rslrl
|
||||
|
||||
# Specify simulation backend
|
||||
uv run scripts/train.py --env cartpole --sim-backend np
|
||||
|
||||
# Specify training backend
|
||||
uv run scripts/train.py --env cartpole --train-backend jax
|
||||
uv run scripts/train.py --env cartpole --train-backend torch
|
||||
# Specify training backend (SKRL only)
|
||||
uv run scripts/train.py --env cartpole --rllib skrl --train-backend jax
|
||||
uv run scripts/train.py --env cartpole --rllib skrl --train-backend torch
|
||||
```
|
||||
|
||||
### Advanced Training Configuration
|
||||
|
||||
```bash
|
||||
# Customize training parameters
|
||||
# Customize training parameters with SKRL
|
||||
uv run scripts/train.py --env cartpole \
|
||||
--rllib skrl \
|
||||
--num-envs 1024 \
|
||||
--train-backend jax \
|
||||
--sim-backend np
|
||||
|
||||
# Customize training parameters with RSLRL
|
||||
uv run scripts/train.py --env cartpole \
|
||||
--rllib rslrl \
|
||||
--num-envs 1024 \
|
||||
--sim-backend np
|
||||
|
||||
# Note: Parameters like learning rate need to be set through configuration files or code override
|
||||
|
||||
# Enable rendering to monitor training process
|
||||
uv run scripts/train.py --env cartpole --render
|
||||
```
|
||||
|
||||
### Different Backend Configuration Differences
|
||||
### Different Framework Configuration
|
||||
|
||||
The system supports configuring different reinforcement learning parameters for different training backends (JAX/Torch). For example:
|
||||
The system supports different RL frameworks with different configuration systems:
|
||||
|
||||
- **dm-walker environment**:
|
||||
- **SKRL Framework**: Supports JAX and PyTorch training backends with configurable parameters per backend (via Python dataclasses)
|
||||
- **RSLRL Framework**: Supports PyTorch backend with configuration via Python dataclasses (RslrlCfg)
|
||||
|
||||
- JAX backend: `mini_batches: 4`
|
||||
- Torch backend: `mini_batches: 32`
|
||||
|
||||
- **dm-runner environment**:
|
||||
- JAX backend: `learning_epochs: 4`
|
||||
- Torch backend: `learning_epochs: 2`
|
||||
|
||||
These differences are implemented through the `@rlcfg(env_name, backend="jax/torch")` decorator in configuration classes. The system automatically applies the corresponding configuration based on the selected training backend.
|
||||
For SKRL, the system supports configuring different reinforcement learning parameters for different training backends (JAX/Torch).
|
||||
|
||||
### Supported Command Line Parameters
|
||||
|
||||
| Parameter | Description | Default Value |
|
||||
| ----------------- | ------------------------------- | ------------- |
|
||||
| `--env` | Environment name | `cartpole` |
|
||||
| `--sim-backend` | Simulation backend (np) | Auto select |
|
||||
| `--train-backend` | Training backend (jax/torch) | Auto select |
|
||||
| `--num-envs` | Number of parallel environments | 2048 |
|
||||
| `--render` | Enable rendering | False |
|
||||
| Parameter | Description | Default Value |
|
||||
| ----------------- | --------------------------------------- | ------------- |
|
||||
| `--env` | Environment name | `cartpole` |
|
||||
| `--rllib` | RL framework (skrl/rslrl) | `skrl` |
|
||||
| `--sim-backend` | Simulation backend (np) | Auto select |
|
||||
| `--train-backend` | Training backend (jax/torch, SKRL only) | Auto select |
|
||||
| `--num-envs` | Number of parallel environments | 2048 |
|
||||
| `--render` | Enable rendering | False |
|
||||
|
||||
> **Note**: Other parameters such as learning rate, network structure, etc., can be set in configuration files. Some environments support configuring different parameters for different training backends.
|
||||
> **Note**: Other parameters such as learning rate, network structure, etc., need to be set through separate configuration files.
|
||||
|
||||
## Training Process Monitoring
|
||||
|
||||
@@ -83,7 +91,7 @@ uv run tensorboard --logdir runs/cartpole
|
||||
uv run scripts/play.py --env cartpole
|
||||
|
||||
# Manually specify policy file for testing
|
||||
uv run scripts/play.py --env cartpole --policy runs/cartpole/nn/best_policy.pickle
|
||||
uv run scripts/play.py --env cartpole --policy runs/cartpole/nn/best_agent.pickle
|
||||
|
||||
# Specify number of test environments
|
||||
uv run scripts/play.py --env cartpole --num-envs 100
|
||||
|
||||
@@ -2,141 +2,80 @@
|
||||
|
||||
MotrixLab provides a flexible configuration system that allows users to customize reinforcement learning training parameters. This section introduces how to configure training environments and reinforcement learning algorithm parameters.
|
||||
|
||||
## RL Training Configuration (PPOCfg)
|
||||
## RL Training Configuration
|
||||
|
||||
Training configuration defines parameters for reinforcement learning algorithms. MotrixLab now supports configuring different parameters for different training backends.
|
||||
MotrixLab supports multiple RL frameworks with different configuration systems:
|
||||
|
||||
### Basic Training Configuration
|
||||
- **SKRL Framework**: Uses Python dataclass configuration (SkrlCfg)
|
||||
- **RSLRL Framework**: Uses Python dataclass configuration (RslrlCfg)
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from motrix_rl.skrl.cfg import PPOCfg
|
||||
from motrix_rl import registry
|
||||
### SKRL Configuration (SkrlCfg)
|
||||
|
||||
# Universal configuration (applies to all backends)
|
||||
@registry.rlcfg("my-task")
|
||||
@dataclass
|
||||
class MyTaskRLCfg(PPOCfg):
|
||||
# Environment parameters
|
||||
num_envs: int = 2048 # Number of parallel environments during training
|
||||
play_num_envs: int = 16 # Number of parallel environments during evaluation
|
||||
Training configuration defines parameters for reinforcement learning algorithms based on PPO. MotrixLab now supports configuring different parameters for different training backends.
|
||||
|
||||
# PPO algorithm parameters
|
||||
learning_rate: float = 3e-4 # Learning rate
|
||||
rollouts: int = 32 # Experience replay rounds
|
||||
learning_epochs: int = 10 # Number of epochs per update
|
||||
mini_batches: int = 32 # Number of mini-batches
|
||||
discount_factor: float = 0.99 # Discount factor
|
||||
grad_norm_clip: float = 1.0 # Gradient clipping
|
||||
#### Complete Configuration Example
|
||||
|
||||
# Network structure parameters
|
||||
policy_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64) # Policy network hidden layers
|
||||
value_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64) # Value network hidden layers
|
||||
The following is the actual `CartPoleSkrlPpo` configuration, demonstrating complete explicit parameter filling. This configuration uses a smaller network `[32, 32]` suitable for simple tasks like CartPole.
|
||||
|
||||
# Training control parameters
|
||||
max_env_steps: int = 1_000_000 # Maximum environment steps
|
||||
check_point_interval: int = 10_000 # Checkpoint interval
|
||||
```{literalinclude} ../../../../motrix_rl/src/motrix_rl/tasks/cartpole.py
|
||||
:language: python
|
||||
:start-after: docs-start: cartpole-skrl-config
|
||||
:end-before: docs-end: cartpole-skrl-config
|
||||
```
|
||||
|
||||
### Different Backend Configurations
|
||||
**Key Configuration Notes:**
|
||||
|
||||
```python
|
||||
# JAX backend specific configuration
|
||||
@registry.rlcfg("my-task", backend="jax")
|
||||
@dataclass
|
||||
class MyTaskJAXCfg(PPOCfg):
|
||||
# JAX optimized configuration
|
||||
learning_rate: float = 2e-4 # JAX backend uses smaller learning rate
|
||||
mini_batches: int = 4 # JAX supports large batches, fewer mini-batches
|
||||
learning_epochs: int = 4 # Training rounds
|
||||
num_envs: int = 2048 # More parallel environments
|
||||
- **Network Architecture**: `hiddens=[32, 32]` - CartPole is a simple task, a small network is sufficient (default: `[256, 128, 64]`)
|
||||
- **Training Epochs**: `learning_epochs=5` - Higher than default value 2, ensuring thorough learning
|
||||
- **Mini-batches**: `mini_batches=4` - Fewer than default 32, suitable for simple tasks
|
||||
- **Training Duration**: `timesteps=5000` - Sufficient for CartPole (default: 10000)
|
||||
- **All Parameters**: All parameters from parent classes are explicitly specified, no hidden defaults
|
||||
|
||||
# Torch backend specific configuration
|
||||
@registry.rlcfg("my-task", backend="torch")
|
||||
@dataclass
|
||||
class MyTaskTorchCfg(PPOCfg):
|
||||
# Torch optimized configuration
|
||||
learning_rate: float = 1e-4 # Torch backend uses even smaller learning rate
|
||||
mini_batches: int = 32 # Torch needs more mini-batches
|
||||
learning_epochs: int = 2 # Fewer training rounds
|
||||
num_envs: int = 1024 # Fewer parallel environments
|
||||
For complete source code, see: [`motrix_rl/src/motrix_rl/tasks/cartpole.py`](https://github.com/Motphys/motrix-lab/blob/main/motrix_rl/src/motrix_rl/tasks/cartpole.py)
|
||||
|
||||
### RSLRL Configuration (RslrlCfg)
|
||||
|
||||
RSLRL is another high-performance reinforcement learning library, specifically designed for complex control tasks like quadruped robots.
|
||||
|
||||
#### Complete Configuration Example
|
||||
|
||||
The following is the actual `CartPoleRslrlPpo` configuration, demonstrating complete explicit parameter filling. This configuration uses a smaller network `[32, 32]` suitable for simple tasks like CartPole.
|
||||
|
||||
```{literalinclude} ../../../../motrix_rl/src/motrix_rl/tasks/cartpole.py
|
||||
:language: python
|
||||
:start-after: docs-start: cartpole-rslrl-config
|
||||
:end-before: docs-end: cartpole-rslrl-config
|
||||
```
|
||||
|
||||
### Complete Configuration Example
|
||||
**Key Configuration Notes:**
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class CompletePPOConfig(PPOCfg):
|
||||
"""
|
||||
Complete reinforcement learning training configuration example
|
||||
Contains all configuration parameters from basic to advanced
|
||||
"""
|
||||
- **Network Architecture**: `hidden_dims=[32, 32]` - CartPole is a simple task, a small network is sufficient (default: `[256, 128, 64]`)
|
||||
- **Training Iterations**: `max_iterations=300` - Total of 300 training iterations
|
||||
- **Steps per Environment**: `num_steps_per_env=16` - Number of steps to collect per environment
|
||||
- **Learning Rate**: `learning_rate=5.0e-4` - Learning rate setting
|
||||
- **Entropy Coefficient**: `entropy_coef=5e-3` - Entropy coefficient for exploration
|
||||
- **All Parameters**: All parameters from parent classes are explicitly specified, no hidden defaults
|
||||
|
||||
# ===== Basic Training Parameters =====
|
||||
seed: Optional[int] = None # Random seed
|
||||
num_envs: int = 2048 # Number of parallel environments during training
|
||||
play_num_envs: int = 16 # Number of parallel environments during evaluation
|
||||
max_env_steps: int = 2_048_000 # Maximum training steps
|
||||
check_point_interval: int = 1000 # Checkpoint save interval
|
||||
For detailed RSLRL configuration options and default values, refer to:
|
||||
|
||||
# ===== PPO Algorithm Core Parameters =====
|
||||
learning_rate: float = 3e-4 # Learning rate
|
||||
rollouts: int = 32 # Experience replay rounds
|
||||
learning_epochs: int = 2 # Number of training rounds per update
|
||||
mini_batches: int = 32 # Number of mini-batches
|
||||
discount_factor: float = 0.99 # Discount factor
|
||||
lambda_param: float = 0.95 # GAE parameter
|
||||
grad_norm_clip: float = 1.0 # Gradient clipping
|
||||
|
||||
# ===== PPO Clipping Parameters =====
|
||||
ratio_clip: float = 0.2 # PPO clipping ratio
|
||||
value_clip: float = 0.2 # Value clipping
|
||||
clip_predicted_values: bool = True # Clip predicted values
|
||||
|
||||
# ===== Loss Function Parameters =====
|
||||
entropy_loss_scale: float = 0.0 # Entropy loss coefficient
|
||||
value_loss_scale: float = 2.0 # Value loss coefficient
|
||||
kl_threshold: float = 0 # KL divergence threshold
|
||||
|
||||
# ===== Learning Rate Scheduler =====
|
||||
learning_rate_scheduler_kl_threshold: float = 0.008 # Adaptive learning rate KL threshold
|
||||
|
||||
# ===== Network Architecture Configuration =====
|
||||
# Small network (suitable for simple tasks like CartPole)
|
||||
# policy_hidden_layer_sizes: tuple[int, ...] = (128, 64)
|
||||
# value_hidden_layer_sizes: tuple[int, ...] = (128, 64)
|
||||
|
||||
# Medium network (default configuration, suitable for most tasks)
|
||||
policy_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64)
|
||||
value_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64)
|
||||
|
||||
# Large network (suitable for complex tasks like robot control)
|
||||
# policy_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
|
||||
# value_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
|
||||
|
||||
# ===== Network Sharing Configuration =====
|
||||
share_policy_value_features: bool = True # Policy and value networks share feature extraction layers
|
||||
|
||||
# ===== Training Control Parameters =====
|
||||
random_timesteps: int = 0 # Random timesteps
|
||||
learning_starts: int = 0 # Timesteps to start learning
|
||||
time_limit_bootstrap: bool = True # Time limit bootstrap
|
||||
|
||||
# ===== Reward Shaping =====
|
||||
rewards_shaper_scale: float = 1.0 # Reward scaling factor
|
||||
```
|
||||
- `motrix_rl/rslrl/cfg.py`: Configuration class definitions
|
||||
- `motrix_rl/template/rslrl_config.yaml`: YAML reference template
|
||||
|
||||
## Configuration Usage Methods
|
||||
|
||||
### 1. Default Configuration Usage
|
||||
|
||||
```bash
|
||||
# Use configuration given in code
|
||||
# Use configuration given in code (default: SKRL framework)
|
||||
uv run scripts/train.py --env my-task
|
||||
|
||||
# Specify training backend, system will automatically select corresponding backend configuration
|
||||
uv run scripts/train.py --env my-task --train-backend jax
|
||||
uv run scripts/train.py --env my-task --train-backend torch
|
||||
# Specify RL framework
|
||||
uv run scripts/train.py --env my-task --rllib skrl
|
||||
uv run scripts/train.py --env my-task --rllib rslrl
|
||||
|
||||
# Specify training backend for SKRL, system will automatically select corresponding backend configuration
|
||||
uv run scripts/train.py --env my-task --rllib skrl --train-backend jax
|
||||
uv run scripts/train.py --env my-task --rllib skrl --train-backend torch
|
||||
```
|
||||
|
||||
### 2. Command Line Parameter Override
|
||||
@@ -144,88 +83,10 @@ uv run scripts/train.py --env my-task --train-backend torch
|
||||
```bash
|
||||
# Override supported command line parameters
|
||||
uv run scripts/train.py --env my-task \
|
||||
--rllib skrl \
|
||||
--num-envs 1024 \
|
||||
--train-backend jax \
|
||||
--sim-backend np
|
||||
|
||||
# System will automatically select JAX backend configuration
|
||||
```
|
||||
|
||||
### 3. Configuration Priority
|
||||
|
||||
System selects configuration in the following priority:
|
||||
|
||||
1. **Backend-specific configuration**: If there is a configuration decorated with `@rlcfg(env_name, backend="jax/torch")`
|
||||
2. **Universal configuration**: If there is a configuration decorated with `@rlcfg(env_name)` (no backend parameter)
|
||||
3. **Configuration override**: Command line `cfg_override` parameter
|
||||
|
||||
For example:
|
||||
|
||||
```python
|
||||
# Highest priority - backend specific configuration
|
||||
@rlcfg("my-task", backend="jax")
|
||||
@dataclass
|
||||
class MyTaskJAXCfg(PPOCfg):
|
||||
mini_batches: int = 4
|
||||
|
||||
# Second priority - universal configuration
|
||||
@rlcfg("my-task")
|
||||
@dataclass
|
||||
class MyTaskRLCfg(PPOCfg):
|
||||
mini_batches: int = 32
|
||||
|
||||
# When using --train-backend jax, system will select MyTaskJAXCfg
|
||||
# When using --train-backend torch, system will select MyTaskRLCfg
|
||||
```
|
||||
|
||||
#### User Configurable Parameters
|
||||
|
||||
| MotrixLab Configuration Class | SKRL Framework Parameter | Description |
|
||||
| -------------------------------------- | --------------------------------------------- | ----------------------------------- |
|
||||
| `learning_rate` | `learning_rate` | Learning rate |
|
||||
| `rollouts` | `rollouts` | Experience replay rounds |
|
||||
| `learning_epochs` | `learning_epochs` | Training rounds |
|
||||
| `mini_batches` | `mini_batches` | Number of mini-batches |
|
||||
| `discount_factor` | `discount_factor` | Discount factor |
|
||||
| `grad_norm_clip` | `grad_norm_clip` | Gradient clipping |
|
||||
| `lambda_param` | `lambda` | GAE parameter |
|
||||
| `ratio_clip` | `ratio_clip` | PPO clipping ratio |
|
||||
| `value_clip` | `value_clip` | Value clipping |
|
||||
| `clip_predicted_values` | `clip_predicted_values` | Clip predicted values |
|
||||
| `entropy_loss_scale` | `entropy_loss_scale` | Entropy loss coefficient |
|
||||
| `value_loss_scale` | `value_loss_scale` | Value loss coefficient |
|
||||
| `kl_threshold` | `kl_threshold` | KL divergence threshold |
|
||||
| `random_timesteps` | `random_timesteps` | Random timesteps |
|
||||
| `learning_starts` | `learning_starts` | Learning start timesteps |
|
||||
| `time_limit_bootstrap` | `time_limit_bootstrap` | Time limit bootstrap |
|
||||
| `learning_rate_scheduler_kl_threshold` | `learning_rate_scheduler_kwargs.kl_threshold` | Adaptive learning rate KL threshold |
|
||||
| `check_point_interval` | `experiment.write_interval` | Log write interval |
|
||||
| `check_point_interval` | `experiment.checkpoint_interval` | Checkpoint save interval |
|
||||
| `rewards_shaper_scale` | `rewards_shaper` | Reward scaling function |
|
||||
|
||||
#### Preprocessor Parameters
|
||||
|
||||
| SKRL Framework Parameter | Type | Description |
|
||||
| ------------------------ | --------------------- | ------------------- |
|
||||
| `state_preprocessor` | RunningStandardScaler | State normalization |
|
||||
| `value_preprocessor` | RunningStandardScaler | Value normalization |
|
||||
|
||||
### Configuration Hierarchy Summary
|
||||
|
||||
```
|
||||
User Configuration Class (PPOCfg)
|
||||
↓ Backend specific selection
|
||||
Backend Configuration (JAX/Torch)
|
||||
↓ Parameter mapping
|
||||
SKRL Framework Configuration Dictionary
|
||||
↓ Pass to
|
||||
PPO Agent
|
||||
↓ Execute
|
||||
Reinforcement Learning Training
|
||||
```
|
||||
|
||||
This design allows users to:
|
||||
|
||||
1. Control complex training parameters through simple configuration classes
|
||||
2. Configure different parameters for different training backends to achieve optimal performance
|
||||
3. Maintain full compatibility with the SKRL framework
|
||||
|
||||
@@ -7,8 +7,8 @@ MotrixLab 是一个为机器人训练设计的机器学习通用架构。它基
|
||||
|
||||
:::{grid-item-card}
|
||||
|
||||
```{video} _static/videos/cartpole.mp4
|
||||
:poster: _static/images/poster/cartpole.jpg
|
||||
```{video} _static/videos/shadow_hand_repose.mp4
|
||||
:poster: _static/images/poster/shadow_hand_repose.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
@@ -147,8 +147,8 @@ MotrixLab 是一个为机器人训练设计的机器学习通用架构。它基
|
||||
|
||||
:::{grid-item-card}
|
||||
|
||||
```{video} _static/videos/dm_reacher.mp4
|
||||
:poster: _static/images/poster/dm_reacher.jpg
|
||||
```{video} _static/videos/cartpole.mp4
|
||||
:poster: _static/images/poster/cartpole.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
@@ -185,6 +185,32 @@ MotrixLab 是一个为机器人训练设计的机器学习通用架构。它基
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
:::
|
||||
:::{grid-item-card}
|
||||
|
||||
```{video} _static/videos/dm_quadruped_fetch.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_fetch.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
:::
|
||||
:::{grid-item-card}
|
||||
|
||||
```{video} _static/videos/dm_quadruped_run.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_run.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
|
||||
@@ -103,45 +103,18 @@ uv run scripts/play.py --env acrobot --policy runs/acrobot/nn/best_policy.pickle
|
||||
|
||||
### 环境配置
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class AcrobotEnvCfg(EnvCfg):
|
||||
model_file: str = ".../acrobot.xml" # MJCF 模型
|
||||
max_episode_seconds: float = 10.0
|
||||
sim_dt: float = 0.01
|
||||
ctrl_dt: float = 0.02
|
||||
reset_noise_scale: float = 0.1
|
||||
render_spacing: float = 2.0
|
||||
```{literalinclude} ../../../../motrix_envs/src/motrix_envs/basic/acrobot/cfg.py
|
||||
:language: python
|
||||
:start-after: '# -- docs-tag-start: acrobot-env-cfg --'
|
||||
:end-before: '# -- docs-tag-end: acrobot-env-cfg --'
|
||||
```
|
||||
|
||||
### 训练配置（PPO 示例）
|
||||
|
||||
```python
|
||||
@rlcfg("acrobot", backend="jax")
|
||||
@dataclass
|
||||
class AcrobotPPO(PPOCfg):
|
||||
max_env_steps: int = 60_000_000
|
||||
check_point_interval: int = 500
|
||||
|
||||
# 覆盖 PPO 配置
|
||||
policy_hidden_layer_sizes: tuple[int, ...] = (32, 32)
|
||||
value_hidden_layer_sizes: tuple[int, ...] = (32, 32)
|
||||
rollouts: int = 64
|
||||
learning_epochs: int = 5
|
||||
mini_batches: int = 8
|
||||
learning_rate: float = 3e-4
|
||||
grad_norm_clip: float = 0.1
|
||||
clip_predicted_values: bool = False
|
||||
value_clip: float = 10.0
|
||||
entropy_loss_scale: float = 0.1
|
||||
learning_rate_scheduler_kl_threshold: float = 0.02
|
||||
discount_factor: float = 0.995
|
||||
lambda_param: float = 0.97
|
||||
ratio_clip: float = 0.2
|
||||
value_loss_scale: float = 0.5
|
||||
random_timesteps: int = 0
|
||||
learning_starts: int = 0
|
||||
kl_threshold: float = 0.03
|
||||
```{literalinclude} ../../../../motrix_rl/src/motrix_rl/tasks/acrobot.py
|
||||
:language: python
|
||||
:start-after: '# -- docs-tag-start: acrobot-train-cfg --'
|
||||
:end-before: '# -- docs-tag-end: acrobot-train-cfg --'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
199
docs/source/zh_CN/user_guide/demo/dm_lqr.md
Normal file
199
docs/source/zh_CN/user_guide/demo/dm_lqr.md
Normal file
@@ -0,0 +1,199 @@
|
||||
# 线性二次调节器
|
||||
|
||||
LQR（Linear Quadratic Regulator，线性二次调节器）是一个典型的连续控制与稳定化任务。本仓库当前提供两个变体：
|
||||
|
||||
- `dm-lqr-2-1`：两个小球通过绳索相连，只有最后一个小球受控。
|
||||
- `dm-lqr-6-2`：六个小球串联成链，只有最后两个小球受控。
|
||||
|
||||
智能体需要在尽量小的控制代价下，将整条系统拉回中心附近并稳定停下。
|
||||
|
||||
```{video} /_static/videos/dm_lqr_2_1.mp4
|
||||
:poster: _static/images/poster/dm_lqr_2_1.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
```{video} /_static/videos/dm_lqr_6_2.mp4
|
||||
:poster: _static/images/poster/dm_lqr_6_2.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 任务描述
|
||||
|
||||
这两个任务都可以看作一维弹簧-阻尼链的平衡控制问题。每个小球只有一个沿 x 轴的滑动自由度，相邻小球之间通过“绳”形成耦合，系统同时受到：
|
||||
|
||||
- 球体自身阻尼
|
||||
- 相邻球体之间的弹簧力和相对速度阻尼
|
||||
- 指向原点的中心回复力
|
||||
- 控制输入作用在末端受控自由度上
|
||||
|
||||
其中：
|
||||
|
||||
- `dm-lqr-2-1` 更简单，适合先验证是否能学会稳定驻点。
|
||||
- `dm-lqr-6-2` 需要把控制作用沿链条传递到更远的非受控球，难度更高。
|
||||
|
||||
---
|
||||
|
||||
## 动作空间（Action Space）
|
||||
|
||||
### dm-lqr-2-1
|
||||
|
||||
| 项目 | 详细信息 |
|
||||
| -------- | ------------------------------- |
|
||||
| **类型** | `Box(-1.0, 1.0, (1,), float32)` |
|
||||
| **维度** | 1 |
|
||||
|
||||
| 序号 | 动作含义 | 最小值 | 最大值 | 对应 XML 关节 |
|
||||
| ---: | ------------------------ | :----: | :----: | :-----------: |
|
||||
| 0 | 作用在末端球上的控制输入 | -1.0 | 1.0 | `q1` |
|
||||
|
||||
### dm-lqr-6-2
|
||||
|
||||
| 项目 | 详细信息 |
|
||||
| -------- | ------------------------------- |
|
||||
| **类型** | `Box(-1.0, 1.0, (2,), float32)` |
|
||||
| **维度** | 2 |
|
||||
|
||||
| 序号 | 动作含义 | 最小值 | 最大值 | 对应 XML 关节 |
|
||||
| ---: | ---------------------- | :----: | :----: | :-----------: |
|
||||
| 0 | 倒数第二个球的控制输入 | -1.0 | 1.0 | `q4` |
|
||||
| 1 | 最后一个球的控制输入 | -1.0 | 1.0 | `q5` |
|
||||
|
||||
---
|
||||
|
||||
## 观察空间
|
||||
|
||||
观察由所有位置 `qpos` 和速度 `qvel` 直接拼接而成。
|
||||
|
||||
### dm-lqr-2-1
|
||||
|
||||
| 项目 | 详细信息 |
|
||||
| -------- | ------------------------------- |
|
||||
| **类型** | `Box(-inf, inf, (4,), float32)` |
|
||||
| **维度** | 4 |
|
||||
|
||||
| 序号 | 观察量 | 含义 |
|
||||
| ---: | ------ | --------------- |
|
||||
| 0 | `q0` | 第 1 个球的位置 |
|
||||
| 1 | `q1` | 第 2 个球的位置 |
|
||||
| 2 | `dq0` | 第 1 个球的速度 |
|
||||
| 3 | `dq1` | 第 2 个球的速度 |
|
||||
|
||||
### dm-lqr-6-2
|
||||
|
||||
| 项目 | 详细信息 |
|
||||
| -------- | -------------------------------- |
|
||||
| **类型** | `Box(-inf, inf, (12,), float32)` |
|
||||
| **维度** | 12 |
|
||||
|
||||
前 6 维为 `q0 ~ q5`，后 6 维为 `dq0 ~ dq5`。
|
||||
|
||||
---
|
||||
|
||||
## 奖励函数设计
|
||||
|
||||
当前实现的奖励由状态代价、速度代价、控制代价、成功奖励和越界惩罚组成：
|
||||
|
||||
```python
|
||||
state_cost = 0.5 * sum(qpos ** 2)
|
||||
velocity_cost = 0.5 * velocity_cost_coef * sum(qvel ** 2)
|
||||
control_cost = 0.5 * control_cost_coef * sum(action ** 2)
|
||||
|
||||
reward = 1.0 - (state_cost + velocity_cost + control_cost)
|
||||
reward += success_bonus
|
||||
reward -= out_of_bounds_penalty
|
||||
```
|
||||
|
||||
直观上：
|
||||
|
||||
- 离中心越远，奖励越低
|
||||
- 速度越大，奖励越低
|
||||
- 控制越激进，奖励越低
|
||||
- 当系统进入“接近原点且速度足够小”的稳定区域时，会得到成功奖励
|
||||
- 越界时会受到额外惩罚
|
||||
|
||||
---
|
||||
|
||||
## 初始状态
|
||||
|
||||
每次重置时：
|
||||
|
||||
- 位置向量从随机方向采样，再归一化到固定范数
|
||||
- 速度初始化为 0
|
||||
|
||||
当前配置中：
|
||||
|
||||
- `dm-lqr-2-1` 的初始位置范数约为 `0.8`
|
||||
- `dm-lqr-6-2` 的初始位置范数约为 `1.0`
|
||||
|
||||
---
|
||||
|
||||
## Episode 终止条件
|
||||
|
||||
满足以下任一条件时，当前 episode 结束并重置：
|
||||
|
||||
- 达到成功条件：
|
||||
位置范数小于成功距离阈值，且速度范数小于成功速度阈值
|
||||
- 超出边界：
|
||||
任意位置超过边界范围，或任意速度超过速度边界
|
||||
- 状态非常接近零点平衡态
|
||||
- 观察或动作中出现 `NaN`
|
||||
|
||||
---
|
||||
|
||||
## 使用指南
|
||||
|
||||
### 1. 环境预览
|
||||
|
||||
```bash
|
||||
uv run scripts/view.py --env dm-lqr-2-1
|
||||
uv run scripts/view.py --env dm-lqr-6-2
|
||||
```
|
||||
|
||||
### 2. 开始训练
|
||||
|
||||
```bash
|
||||
uv run scripts/train.py --env dm-lqr-2-1
|
||||
uv run scripts/train.py --env dm-lqr-6-2
|
||||
```
|
||||
|
||||
### 3. 查看训练曲线
|
||||
|
||||
```bash
|
||||
uv run tensorboard --logdir runs/dm-lqr-2-1
|
||||
uv run tensorboard --logdir runs/dm-lqr-6-2
|
||||
```
|
||||
|
||||
### 4. 测试训练结果
|
||||
|
||||
```bash
|
||||
uv run scripts/play.py --env dm-lqr-2-1
|
||||
uv run scripts/play.py --env dm-lqr-6-2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 预期训练结果
|
||||
|
||||
### dm-lqr-2-1
|
||||
|
||||
1. 受控球能够带动未受控球一起回到中心附近。
|
||||
2. 两个球的位置和速度最终都收敛到较小范围内。
|
||||
3. 策略不会长期停在偏离中心的静止点。
|
||||
|
||||
### dm-lqr-6-2
|
||||
|
||||
1. 最后两个受控球能够把整条链逐步拉回中心。
|
||||
2. 链条传播过程中不会出现明显发散或持续振荡。
|
||||
3. 成功率随训练推进逐步上升，越界率逐步下降。
|
||||
159
docs/source/zh_CN/user_guide/demo/dm_point_mass.md
Normal file
159
docs/source/zh_CN/user_guide/demo/dm_point_mass.md
Normal file
@@ -0,0 +1,159 @@
|
||||
# 质点环境
|
||||
|
||||
质点（Point Mass）环境是一个简单但基础的 2D 导航任务，智能体通过控制一个质点来到达目标位置。这个环境是强化学习概念和连续动作空间的优秀入门案例。
|
||||
|
||||
```{video} /_static/videos/point_mass.mp4
|
||||
:poster: _static/images/poster/point_mass.jpg
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
## 任务描述
|
||||
|
||||
质点环境是一个 2D 导航任务。智能体需要通过施加力来控制一个质点，使其移动到随机生成的目标位置。该任务要求智能体学习高效的导航策略，以最小的控制成本到达目标。
|
||||
|
||||
---
|
||||
|
||||
## 动作空间（Action Space）
|
||||
|
||||
| 项目 | 详细信息 |
|
||||
| -------- | ------------------------------- |
|
||||
| **类型** | `Box(-1.0, 1.0, (2,), float32)` |
|
||||
| **维度** | 2 |
|
||||
|
||||
动作对应如下：
|
||||
|
||||
| 序号 | 动作含义（施加的力） | 最小值 | 最大值 | 对应 XML 中名称 |
|
||||
| ---: | -------------------- | :----: | :----: | :-------------: |
|
||||
| 0 | x 方向力 | -1 | 1 | `x_force` |
|
||||
| 1 | y 方向力 | -1 | 1 | `y_force` |
|
||||
|
||||
---
|
||||
|
||||
## 观察空间
|
||||
|
||||
| 项目 | 详细信息 |
|
||||
| -------- | ------------------------------- |
|
||||
| **类型** | `Box(-inf, inf, (9,), float32)` |
|
||||
| **维度** | 9 |
|
||||
|
||||
质点环境的观测空间由以下部分组成（按顺序）：
|
||||
|
||||
| 部分 | 内容说明 | 维度 | 备注 |
|
||||
| -------- | -------------------------- | ---- | ---- |
|
||||
| **位置** | 质点的 x、y 坐标 | 2 | |
|
||||
| **速度** | 质点的 x、y 方向速度 | 2 | |
|
||||
| **目标** | 目标的 x、y 坐标 | 2 | |
|
||||
| **距离** | 到目标的 x、y 方向距离向量 | 2 | |
|
||||
| **距离** | 到目标的欧几里得距离 | 1 | |
|
||||
|
||||
---
|
||||
|
||||
## 奖励函数设计
|
||||
|
||||
质点环境的奖励函数由以下几个部分组成：
|
||||
|
||||
### 距离奖励
|
||||
|
||||
```python
|
||||
# 指数距离奖励 - 离目标越近奖励越强
|
||||
distance_reward = np.exp(-10 * dist_to_target)
|
||||
```
|
||||
|
||||
### 目标到达和停留奖励
|
||||
|
||||
```python
|
||||
# 到达目标的大额奖励
|
||||
target_bonus = 100.0 * in_target
|
||||
|
||||
# 在目标内持续停留的奖励
|
||||
continuous_reward = 30.0 * in_target
|
||||
```
|
||||
|
||||
### 控制和路径优化
|
||||
|
||||
```python
|
||||
# 在目标内时，距离目标中心越远的惩罚
|
||||
center_penalty = np.where(in_target, 10.0 * dist_to_target, 0.0)
|
||||
|
||||
# 控制惩罚，鼓励平滑移动
|
||||
control_penalty = 0.1 * vel_magnitude
|
||||
|
||||
# 路径优化奖励，鼓励直线移动
|
||||
path_reward = 0.5 * direction_alignment
|
||||
```
|
||||
|
||||
### 总奖励计算
|
||||
|
||||
```python
|
||||
# 组合所有奖励组件
|
||||
rwd = distance_reward + target_bonus + continuous_reward + path_reward - center_penalty - control_penalty
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 初始状态
|
||||
|
||||
- 质点位置在[-1.0, 1.0]范围内随机初始化
|
||||
- 目标位置在[-1.5, 1.5]范围内随机初始化
|
||||
- 质点速度初始化为 0
|
||||
|
||||
## Episode 终止条件
|
||||
|
||||
- 质点到达目标并在目标内停留 0.5 秒
|
||||
- 模拟时间达到 10 秒
|
||||
- 观测值出现异常数值（NaN）
|
||||
|
||||
---
|
||||
|
||||
## 使用指南
|
||||
|
||||
### 1. 环境预览
|
||||
|
||||
```bash
|
||||
uv run scripts/view.py --env point_mass
|
||||
```
|
||||
|
||||
### 2. 开始训练
|
||||
|
||||
```bash
|
||||
uv run scripts/train.py --env point_mass
|
||||
```
|
||||
|
||||
### 3. 查看训练进度
|
||||
|
||||
```bash
|
||||
uv run tensorboard --logdir runs/point_mass
|
||||
```
|
||||
|
||||
### 4. 测试训练结果
|
||||
|
||||
```bash
|
||||
uv run scripts/play.py --env point_mass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 预期训练结果
|
||||
|
||||
### 导航性能
|
||||
|
||||
1. 智能体学会直接向目标移动
|
||||
2. 移动平滑，控制 effort 最小
|
||||
3. 在 episode 持续时间内一致地到达目标
|
||||
|
||||
### 学习进度
|
||||
|
||||
1. 初始学习阶段迅速，智能体发现基本导航策略
|
||||
2. 控制策略逐渐精细化
|
||||
3. 在不同目标位置上表现稳定
|
||||
|
||||
### 行为特征
|
||||
|
||||
1. 朝向目标的高效路径规划
|
||||
2. 平滑接近目标中心
|
||||
3. 最小化过冲或振荡行为
|
||||
290
docs/source/zh_CN/user_guide/demo/dm_quadruped.md
Normal file
290
docs/source/zh_CN/user_guide/demo/dm_quadruped.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# 四足机器人
|
||||
|
||||
四足机器人（Quadruped）是 DeepMind Control Suite 中的经典连续控制任务。在 MotrixLab 中，`motrix_envs/src/motrix_envs/basic/quadruped` 目录当前注册了四个可直接训练的任务：平地行走 `dm-quadruped-walk`、平地奔跑 `dm-quadruped-run`、复杂地形逃离 `dm-quadruped-escape`，以及平地推球到目标区 `dm-quadruped-fetch`。
|
||||
|
||||
## 任务预览
|
||||
|
||||
### Walk
|
||||
|
||||
```{video} /_static/videos/dm_quadruped_walk.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_walk.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
### Run
|
||||
|
||||
```{video} /_static/videos/dm_quadruped_run.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_run.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
### Escape
|
||||
|
||||
```{video} /_static/videos/dm_quadruped_escape.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_escape.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
### Fetch
|
||||
|
||||
```{video} /_static/videos/dm_quadruped_fetch.mp4
|
||||
:poster: _static/images/poster/dm_quadruped_fetch.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
## 任务概览
|
||||
|
||||
| 环境 ID | 任务目标 | 模型文件 | 目标速度 | 观察维度 |
|
||||
| --------------------- | -------------------------------- | ---------------------- | -------- | -------- |
|
||||
| `dm-quadruped-walk` | 在平地上稳定向前行走并保持朝向 | `quadruped_walk.xml` | 0.5 m/s | 54 |
|
||||
| `dm-quadruped-run` | 在平地上高速奔跑并保持稳定姿态 | `quadruped_walk.xml` | 5.0 m/s | 54 |
|
||||
| `dm-quadruped-escape` | 在起伏地形上尽快向外逃离原点区域 | `quadruped_escape.xml` | 3.0 m/s | 57 |
|
||||
| `dm-quadruped-fetch` | 在平地上将球体推动到目标区域 | `quadruped_fetch.xml` | 2.0 m/s | 66 |
|
||||
|
||||
## 任务描述
|
||||
|
||||
Quadruped 是一个三维四足机器人任务。机器人主体由一个躯干和四条腿组成，每条腿具有偏航、抬升和伸展相关的控制能力。底层 XML 中定义了髋部、膝部和踝部的关节结构，而动作层采用每条腿 3 个执行器的耦合设计：
|
||||
|
||||
- `yaw`：控制腿部偏航
|
||||
- `lift`：通过 tendon 耦合控制抬腿动作
|
||||
- `extend`：通过 tendon 耦合控制腿部伸展/收缩
|
||||
|
||||
`walk` 与 `run` 使用相同的平地模型，区别主要在目标速度；`escape` 使用带高度场的 `quadruped_escape.xml`，要求机器人在崎岖地形中快速远离世界原点，同时保持躯干直立和运动稳定；`fetch` 使用 `quadruped_fetch.xml`，在场景中额外引入自由球体与目标区域，要求机器人先调整到合适站位，再将球向目标方向推进。
|
||||
|
||||
---
|
||||
|
||||
## 动作空间（Action Space）
|
||||
|
||||
| 项目 | 详细信息 |
|
||||
| -------- | -------------------------------- |
|
||||
| **类型** | `Box(low, high, (12,), float32)` |
|
||||
| **维度** | 12 |
|
||||
|
||||
动作按照四条腿依次排列，每条腿包含 `yaw / lift / extend` 三个执行器：
|
||||
|
||||
| 序号 | 动作含义 | 最小值 | 最大值 | 对应执行器 |
|
||||
| ---: | -------------- | :----: | :----: | -------------------- |
|
||||
| 0 | 左前腿偏航控制 | -1.0 | 1.0 | `yaw_front_left` |
|
||||
| 1 | 左前腿抬升控制 | -1.0 | 1.1 | `lift_front_left` |
|
||||
| 2 | 左前腿伸展控制 | -0.8 | 0.8 | `extend_front_left` |
|
||||
| 3 | 右前腿偏航控制 | -1.0 | 1.0 | `yaw_front_right` |
|
||||
| 4 | 右前腿抬升控制 | -1.0 | 1.1 | `lift_front_right` |
|
||||
| 5 | 右前腿伸展控制 | -0.8 | 0.8 | `extend_front_right` |
|
||||
| 6 | 右后腿偏航控制 | -1.0 | 1.0 | `yaw_back_right` |
|
||||
| 7 | 右后腿抬升控制 | -1.0 | 1.1 | `lift_back_right` |
|
||||
| 8 | 右后腿伸展控制 | -0.8 | 0.8 | `extend_back_right` |
|
||||
| 9 | 左后腿偏航控制 | -1.0 | 1.0 | `yaw_back_left` |
|
||||
| 10 | 左后腿抬升控制 | -1.0 | 1.1 | `lift_back_left` |
|
||||
| 11 | 左后腿伸展控制 | -0.8 | 0.8 | `extend_back_left` |
|
||||
|
||||
---
|
||||
|
||||
## 观察空间
|
||||
|
||||
| 环境 | 详细信息 |
|
||||
| --------------------- | -------------------------------- |
|
||||
| `dm-quadruped-walk` | `Box(-inf, inf, (54,), float32)` |
|
||||
| `dm-quadruped-run` | `Box(-inf, inf, (54,), float32)` |
|
||||
| `dm-quadruped-escape` | `Box(-inf, inf, (57,), float32)` |
|
||||
| `dm-quadruped-fetch` | `Box(-inf, inf, (66,), float32)` |
|
||||
|
||||
四种任务共享绝大部分本体观测，`escape` 额外增加了 3 维与原点相关的任务信息，`fetch` 额外增加了球体状态与目标位置信息：
|
||||
|
||||
| 部分 | 内容说明 | 维度 | `walk/run` | `escape` | `fetch` |
|
||||
| ---------------------- | ------------------------------------ | ---- | ---------- | -------- | ------- |
|
||||
| **egocentric dof pos** | 身体广义位置状态 | 16 | 是 | 是 | 是 |
|
||||
| **egocentric dof vel** | 身体广义速度状态 | 16 | 是 | 是 | 是 |
|
||||
| **actuator ctrl** | 当前 12 维执行器控制量 | 12 | 是 | 是 | 是 |
|
||||
| **torso velocity** | 躯干线速度传感器 `velocimeter` | 3 | 是 | 是 | 是 |
|
||||
| **torso upright** | 躯干朝上程度标量 | 1 | 是 | 是 | 是 |
|
||||
| **imu** | IMU 加速度与角速度 | 6 | 是 | 是 | 是 |
|
||||
| **origin** | 世界原点在本体坐标系中的位置 | 3 | 否 | 是 | 否 |
|
||||
| **ball state** | 球相对本体的位置、相对线速度与角速度 | 9 | 否 | 否 | 是 |
|
||||
| **target** | 目标区域在本体坐标系中的相对位置 | 3 | 否 | 否 | 是 |
|
||||
|
||||
环境还在 XML 中定义了足端力/力矩传感器与质心传感器，但当前实现的默认观测并未直接拼接这些量。
|
||||
|
||||
---
|
||||
|
||||
## 奖励函数设计
|
||||
|
||||
四种任务都以“保持躯干直立”为核心约束。实现中首先根据 `torso_upright` 计算 `upright_reward`，要求机器人主体保持接近直立的姿态。
|
||||
|
||||
### Walk / Run
|
||||
|
||||
`dm-quadruped-walk` 与 `dm-quadruped-run` 使用相同的奖励结构，只是目标速度不同：
|
||||
|
||||
- `walk` 追踪 `0.5 m/s`
|
||||
- `run` 追踪 `5.0 m/s`
|
||||
|
||||
总奖励由以下部分组成：
|
||||
|
||||
```python
|
||||
# 速度奖励：沿前向达到目标速度
|
||||
# 姿态奖励：保持躯干直立
|
||||
# 辅助奖励：高度、横向稳定性、朝向一致性、动作平滑性
|
||||
# 惩罚项：后退、竖直速度过大、横滚/俯仰角速度过大、偏离默认姿态
|
||||
total_reward = upright_reward * move_reward + shaping_terms - penalty_terms
|
||||
```
|
||||
|
||||
其中主要 shaping/penalty 项包括：
|
||||
|
||||
- `height_reward`：鼓励躯干保持在站立高度附近
|
||||
- `lateral_reward`：抑制过大的横向速度
|
||||
- `heading_reward`：鼓励机器人持续朝 +X 方向前进
|
||||
- `smooth_reward`：惩罚相邻时刻动作变化过大
|
||||
- `backward_penalty`：抑制向后运动
|
||||
- `lin_vel_z_penalty` 与 `ang_vel_xy_penalty`：抑制上下颠簸和躯干横滚/俯仰过大
|
||||
- `similar_to_default_penalty`：鼓励关节姿态不要偏离默认站姿太远
|
||||
|
||||
### Escape
|
||||
|
||||
`dm-quadruped-escape` 在 locomotion 奖励基础上，增加了“尽快逃离原点区域”的任务项。该任务使用 `quadruped_escape.xml` 中的高度场地形：
|
||||
|
||||
```python
|
||||
# 基础 locomotion 奖励
|
||||
# + 远离原点奖励
|
||||
# + 径向外逃速度奖励
|
||||
total_reward = locomotion_reward + upright_reward * escape_reward + radial_speed_reward
|
||||
```
|
||||
|
||||
额外任务项包括：
|
||||
|
||||
- `escape_reward`：根据机器人离原点区域的距离给出奖励
|
||||
- `radial_speed_reward`：鼓励沿着“远离原点”的方向加速前进
|
||||
|
||||
这使得 `escape` 不仅要求机器人跑得快，还要求它在复杂地形上沿正确方向脱离中心区域。
|
||||
|
||||
### Fetch
|
||||
|
||||
`dm-quadruped-fetch` 使用一套面向“站位 + 推球”的专用 shaping 结构。当前实现中，奖励主要围绕机器人相对球和目标的几何关系展开：
|
||||
|
||||
```python
|
||||
# 站位阶段：鼓励机器人先移动到球后方或侧后方
|
||||
# 就绪阶段：鼓励机器人面向球、贴近球，并与球-目标连线对齐
|
||||
# 推球阶段：鼓励球向目标方向滚动，并最终进入目标区域
|
||||
# 惩罚项：反向移动、把球推离目标方向、腿部与球体过近
|
||||
total_reward = stage_terms + ready_terms + push_terms - penalty_terms
|
||||
```
|
||||
|
||||
其中主要项包括：
|
||||
|
||||
- `stage_move`：鼓励机器人朝当前阶段目标点移动
|
||||
- `stage_reach`：鼓励机器人先到达球后方或侧后方的站位点
|
||||
- `behind_align`：鼓励机器人站到球和目标连线的后方
|
||||
- `face_ball`：鼓励机体朝向球体
|
||||
- `near_ball`：鼓励机器人接近球体
|
||||
- `ready` 与 `ready_gate`：综合站位、朝向和距离关系，决定是否进入更积极的推球阶段
|
||||
- `fetch`：鼓励球体靠近目标区域
|
||||
- `push`：鼓励球沿目标方向滚动
|
||||
- `backward`：惩罚朝阶段目标的反向运动
|
||||
- `away`：惩罚把球推向远离目标的方向
|
||||
- `leg_ball`：惩罚腿部几何体与球体过近，减少“缠球”或挤球现象
|
||||
|
||||
此外，`fetch` 还通过 `stability` 门控同时约束躯干直立程度和机体高度，避免机器人通过明显跌倒或趴地的方式获取任务奖励。
|
||||
|
||||
---
|
||||
|
||||
## 初始状态
|
||||
|
||||
- `walk`、`run` 和 `escape` 都从 XML 中的默认四足站姿开始重置
|
||||
- 这三个任务的根关节朝向固定为初始朝向，不随机旋转
|
||||
- `fetch` 会在平面上随机初始化机器人位置与偏航角，同时随机初始化球体在地面上的位置
|
||||
- 所有任务的关节速度与球体速度都初始化为 0
|
||||
- 重置时会自动抬高机体，直到机器人与地面不发生初始穿透/碰撞
|
||||
|
||||
## Episode 终止条件
|
||||
|
||||
- 最大时长为 20 秒
|
||||
- 当观测中出现 `NaN` 时，episode 终止
|
||||
- `walk`、`run` 与 `escape` 当前实现没有单独设置“跌倒即终止”的条件
|
||||
- `fetch` 在机体明显跌倒时会提前终止，条件包括躯干直立程度过低或躯干高度过低
|
||||
- 当前实现尚未单独设置“球进入目标区域即成功终止”的条件
|
||||
|
||||
---
|
||||
|
||||
## 使用指南
|
||||
|
||||
### 1. 环境预览
|
||||
|
||||
```bash
|
||||
uv run scripts/view.py --env dm-quadruped-walk
|
||||
uv run scripts/view.py --env dm-quadruped-run
|
||||
uv run scripts/view.py --env dm-quadruped-escape
|
||||
uv run scripts/view.py --env dm-quadruped-fetch
|
||||
```
|
||||
|
||||
### 2. 开始训练
|
||||
|
||||
```bash
|
||||
uv run scripts/train.py --env dm-quadruped-walk
|
||||
uv run scripts/train.py --env dm-quadruped-run
|
||||
uv run scripts/train.py --env dm-quadruped-escape
|
||||
uv run scripts/train.py --env dm-quadruped-fetch
|
||||
```
|
||||
|
||||
### 3. 查看训练进度
|
||||
|
||||
```bash
|
||||
uv run tensorboard --logdir runs/dm-quadruped-walk
|
||||
uv run tensorboard --logdir runs/dm-quadruped-run
|
||||
uv run tensorboard --logdir runs/dm-quadruped-escape
|
||||
uv run tensorboard --logdir runs/dm-quadruped-fetch
|
||||
```
|
||||
|
||||
### 4. 测试训练结果
|
||||
|
||||
```bash
|
||||
uv run scripts/play.py --env dm-quadruped-walk
|
||||
uv run scripts/play.py --env dm-quadruped-run
|
||||
uv run scripts/play.py --env dm-quadruped-escape
|
||||
uv run scripts/play.py --env dm-quadruped-fetch
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 预期训练结果
|
||||
|
||||
### 行走任务（`dm-quadruped-walk`）
|
||||
|
||||
1. 稳定维持接近 `0.5 m/s` 的平地前进速度
|
||||
2. 身体姿态平稳，横向摆动较小
|
||||
3. 能够持续保持朝 +X 方向行走
|
||||
|
||||
### 奔跑任务（`dm-quadruped-run`）
|
||||
|
||||
1. 速度提升到接近或超过 `5.0 m/s`
|
||||
2. 步幅明显增大，动作具有更强爆发性
|
||||
3. 高速运动下仍能保持较好的躯干稳定性
|
||||
|
||||
### 逃离任务（`dm-quadruped-escape`）
|
||||
|
||||
1. 能够快速离开原点附近区域
|
||||
2. 在高度场地形上保持稳定落脚，不易侧翻
|
||||
3. 运动方向以向外逃离为主，而不是原地打转
|
||||
|
||||
### 推球任务（`dm-quadruped-fetch`）
|
||||
|
||||
1. 能够先调整到球和目标连线的合理站位，而不是直接从侧面乱撞球
|
||||
2. 能够稳定地将球向目标区域推进，而不是把球踢飞或持续推离目标
|
||||
3. 在推球过程中保持较好的机体稳定性，减少趴地、翻倒和腿部缠球现象
|
||||
@@ -61,25 +61,14 @@ GO1 环境的观测空间由以下部分组成（按顺序）：
|
||||
|
||||
GO1 的奖励函数是一个复杂的复合函数，包含多个组件：
|
||||
|
||||
```python
|
||||
# 主要奖励组件
|
||||
reward_config.scales = {
|
||||
"tracking_lin_vel": 1.0, # 线速度跟踪奖励
|
||||
"tracking_ang_vel": 0.5, # 角速度跟踪奖励
|
||||
"feet_air_time": 1.0, # 足部空中时间奖励
|
||||
"lin_vel_z": -2.0, # Z轴线速度惩罚
|
||||
"ang_vel_xy": -0.05, # XY轴角速度惩罚
|
||||
"orientation": -0.0, # 姿态偏离惩罚
|
||||
"torques": -0.00001, # 力矩消耗惩罚
|
||||
"dof_acc": -2.5e-7, # 关节加速度惩罚
|
||||
"action_rate": -0.001, # 动作变化率惩罚
|
||||
"hip_pos": -1, # 髋关节位置惩罚
|
||||
"calf_pos": -0.3, # 腿关节位置惩罚
|
||||
}
|
||||
|
||||
# 总奖励 = 加权组合以上所有项
|
||||
```{literalinclude} ../../../../motrix_envs/src/motrix_envs/locomotion/go1/cfg.py
|
||||
:language: python
|
||||
:start-after: '# -- docs-tag-start: go1-reward-config --'
|
||||
:end-before: '# -- docs-tag-end: go1-reward-config --'
|
||||
```
|
||||
|
||||
_总奖励 = 加权组合以上所有项_
|
||||
|
||||
---
|
||||
|
||||
## 初始状态
|
||||
|
||||
@@ -86,25 +86,14 @@ Unitree GO1 复杂地形行走环境是一个四足机器人强化学习任务
|
||||
|
||||
GO1 复杂地形的奖励函数采用多目标加权设计：
|
||||
|
||||
```python
|
||||
# 核心奖励组件
|
||||
reward_config.scales = {
|
||||
"tracking_lin_vel": 1.0, # 线速度跟踪精度
|
||||
"tracking_ang_vel": 0.5, # 角速度跟踪精度
|
||||
"orientation": -0.0, # 身体姿态稳定性惩罚
|
||||
"torques": -0.00001, # 关节力矩惩罚（能量效率）
|
||||
"dof_acc": -2.5e-7, # 关节加速度惩罚
|
||||
"action_rate": -0.001, # 动作平滑性惩罚
|
||||
"feet_air_time": 1.0, # 足部空中时间奖励（鼓励大步幅）
|
||||
"stand_still": 0.0, # 静止指令的关节位置保持
|
||||
"hip_pos": -1, # 髋关节位置偏好
|
||||
"calf_pos": -0.3, # 小腿关节位置偏好
|
||||
"feet_stumble": -0.5, # 当足部横向触碰障碍物时施加惩罚
|
||||
}
|
||||
|
||||
# 总奖励 = 加权组合以上所有项
|
||||
```{literalinclude} ../../../../motrix_envs/src/motrix_envs/locomotion/go1/cfg.py
|
||||
:language: python
|
||||
:start-after: '# -- docs-tag-start: go1-reward-config --'
|
||||
:end-before: '# -- docs-tag-end: go1-reward-config --'
|
||||
```
|
||||
|
||||
_总奖励 = 加权组合以上所有项_
|
||||
|
||||
---
|
||||
|
||||
## 初始状态
|
||||
|
||||
@@ -104,30 +104,18 @@ uv run scripts/play.py --env pendulum --policy runs/pendulum/nn/best_policy.pick
|
||||
|
||||
### 环境配置（示例）
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class PendulumEnvCfg(EnvCfg):
|
||||
model_file: str = ".../pendulum.xml" # MJCF 模型，默认 gear=5
|
||||
max_episode_seconds: float = 20.0
|
||||
sim_dt: float = 0.0125
|
||||
ctrl_dt: float = 0.025
|
||||
```{literalinclude} ../../../../motrix_envs/src/motrix_envs/basic/pendulum/cfg.py
|
||||
:language: python
|
||||
:start-after: '# -- docs-tag-start: pendulum-env-cfg --'
|
||||
:end-before: '# -- docs-tag-end: pendulum-env-cfg --'
|
||||
```
|
||||
|
||||
### 训练配置（示例 PPO）
|
||||
|
||||
```python
|
||||
@rlcfg("pendulum")
|
||||
@dataclass
|
||||
class PendulumPPO(PPOCfg):
|
||||
seed: int = 42
|
||||
max_env_steps: int = 10_000_000
|
||||
num_envs: int = 1024
|
||||
learning_rate: float = 3e-4
|
||||
rollouts: int = 32
|
||||
learning_epochs: int = 5
|
||||
mini_batches: int = 4
|
||||
policy_hidden_layer_sizes: tuple[int, ...] = (64, 64)
|
||||
value_hidden_layer_sizes: tuple[int, ...] = (64, 64)
|
||||
```{literalinclude} ../../../../motrix_rl/src/motrix_rl/tasks/pendulum.py
|
||||
:language: python
|
||||
:start-after: '# -- docs-tag-start: pendulum-train-cfg --'
|
||||
:end-before: '# -- docs-tag-end: pendulum-train-cfg --'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
258
docs/source/zh_CN/user_guide/demo/rm65_open_cabinet.md
Normal file
258
docs/source/zh_CN/user_guide/demo/rm65_open_cabinet.md
Normal file
@@ -0,0 +1,258 @@
|
||||
# RM65 打开抽屉
|
||||
|
||||
## 概述
|
||||
|
||||
本文档描述 `rm65-open-cabinet` 操作任务环境。该环境使用 RM65 六自由度机械臂与并联夹爪，目标是接近柜体底部抽屉把手、完成抓取并将抽屉拉开。
|
||||
|
||||
```{video} /_static/videos/rm65_open_cabinet.mp4
|
||||
:poster: _static/images/poster/rm65_open_cabinet.png
|
||||
:nocontrols:
|
||||
:autoplay:
|
||||
:playsinline:
|
||||
:muted:
|
||||
:loop:
|
||||
:width: 100%
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 环境描述
|
||||
|
||||
该任务基于 RM65 机械臂和柜体抽屉场景构建。虽然环境注册名为 `rm65-open-cabinet`，但当前实际操作对象是柜体底部抽屉，代码中对应目标为 `drawer_bottom_handle` 与 `drawer_bottom_joint`。
|
||||
|
||||
### 机器人结构
|
||||
|
||||
RM65 机器人在该环境中包含以下主要部分：
|
||||
|
||||
- **基座（base_link）**：固定在工作台前方
|
||||
- **6 个机械臂关节**：`joint_1` ~ `joint_6`
|
||||
- **并联夹爪**：主驱动关节为 `gripper_Left_1_Joint`，其余 5 个夹爪关节通过联动关系同步
|
||||
- **末端执行器（TCP）**：`gripper` 站点，用于计算末端与把手的相对位姿
|
||||
- **手指接触点**：`left_finger_pad`、`right_finger_pad`，用于判定是否对准把手以及是否发生穿透
|
||||
|
||||
### 场景物体
|
||||
|
||||
- **柜体**：包含多个门和抽屉
|
||||
- **目标把手**：`drawer_bottom_handle`
|
||||
- **目标关节**：`drawer_bottom_joint`，滑动范围为 `0.0 ~ 0.4 m`
|
||||
|
||||
### 任务目标
|
||||
|
||||
机器人需要完成以下阶段：
|
||||
|
||||
1. **靠近把手**：将末端执行器移动到抽屉把手附近
|
||||
2. **姿态对齐**：让夹爪姿态与把手姿态匹配
|
||||
3. **稳定抓取**：闭合夹爪并连续保持抓取
|
||||
4. **打开抽屉**：沿抽屉滑动方向将底部抽屉拉开
|
||||
|
||||
---
|
||||
|
||||
## 动作空间
|
||||
|
||||
动作空间为 `Box(-inf, inf, (7,), float32)`。
|
||||
|
||||
其中前 6 维控制机械臂关节目标，最后 1 维控制夹爪开合。
|
||||
|
||||
### 控制模式
|
||||
|
||||
- **机械臂**：默认使用 `joint_target` 模式，并启用归一化目标控制
|
||||
机械臂动作会先裁剪到 `[-1, 1]`，再线性映射到各关节控制范围
|
||||
- **夹爪**：默认使用 `binary` 模式
|
||||
原始动作经过 Sigmoid 映射为闭合概率，再通过带迟滞的二值开关决定夹爪开闭
|
||||
|
||||
### 动作维度详细说明
|
||||
|
||||
| 编号 | 动作说明 | 原始输入范围 | 实际控制对象 |
|
||||
| ---- | ------------ | ------------- | ---------------------- |
|
||||
| 0 | 关节 1 目标 | `(-inf, inf)` | `joint_1` |
|
||||
| 1 | 关节 2 目标 | `(-inf, inf)` | `joint_2` |
|
||||
| 2 | 关节 3 目标 | `(-inf, inf)` | `joint_3` |
|
||||
| 3 | 关节 4 目标 | `(-inf, inf)` | `joint_4` |
|
||||
| 4 | 关节 5 目标 | `(-inf, inf)` | `joint_5` |
|
||||
| 5 | 关节 6 目标 | `(-inf, inf)` | `joint_6` |
|
||||
| 6 | 夹爪开合命令 | `(-inf, inf)` | `gripper_Left_1_Joint` |
|
||||
|
||||
### 控制约束
|
||||
|
||||
- 控制周期 `ctrl_dt = 0.025s`，对应 40Hz 控制频率
|
||||
- 机械臂默认启用速度限制、加速度限制、动作延迟与执行器一阶滞后
|
||||
- 训练时会在每个 episode 内随机化机械臂延迟、滞后、速度上限和加速度上限，以增强 sim2real 鲁棒性
|
||||
- 夹爪使用迟滞阈值控制：
|
||||
- 开启到闭合阈值：`0.78`
|
||||
- 闭合到开启阈值：`0.62`
|
||||
- 最小切换间隔：`0.25s`
|
||||
|
||||
---
|
||||
|
||||
## 观测空间
|
||||
|
||||
观测空间为 `Box(-inf, inf, (84,), float32)`，最终观测会裁剪到 `[-5, 5]`。
|
||||
|
||||
### 观测组成
|
||||
|
||||
观测由以下 4 部分拼接而成：
|
||||
|
||||
1. **关节位置（7 维）**
|
||||
- 6 个机械臂关节
|
||||
- 1 个夹爪主驱动关节
|
||||
- 统一归一化到 `[-1, 1]`
|
||||
2. **关节速度（7 维）**
|
||||
- 使用相邻两帧关节位置差分估计
|
||||
- 再除以 `2` 做缩放
|
||||
3. **目标相对位姿（7 维）**
|
||||
- 末端到把手的相对位置 `Δx, Δy, Δz`
|
||||
- 把手相对末端的四元数姿态差
|
||||
4. **动作历史（63 维）**
|
||||
- 最近 `9` 步原始动作历史
|
||||
- 每步 `7` 维，共 `9 × 7 = 63` 维
|
||||
|
||||
### 观测维度详细说明
|
||||
|
||||
| 编号范围 | 内容说明 | 维度 |
|
||||
| -------- | -------------------------- | ---- |
|
||||
| 0-6 | 归一化关节位置 | 7 |
|
||||
| 7-13 | 关节速度 | 7 |
|
||||
| 14-16 | 末端到把手的相对位置 | 3 |
|
||||
| 17-20 | 末端到把手的相对姿态四元数 | 4 |
|
||||
| 21-83 | 最近 9 步动作历史 | 63 |
|
||||
|
||||
### 观测噪声
|
||||
|
||||
该环境默认开启 sim2real 观测扰动，主要包括：
|
||||
|
||||
- 关节位置与速度噪声
|
||||
- 把手位置与姿态噪声
|
||||
- 把手观测偏置
|
||||
- 把手观测随机丢帧，并在丢帧时保持上一帧结果
|
||||
|
||||
---
|
||||
|
||||
## 奖励函数
|
||||
|
||||
奖励函数采用分阶段复合设计，既鼓励接近与对齐，也鼓励稳定抓取和持续拉开抽屉。
|
||||
|
||||
### 主要奖励项
|
||||
|
||||
1. **距离奖励**
|
||||
|
||||
```python
|
||||
dist_reward = 15.0 * (1 - tanh(distance / 0.4))
|
||||
```
|
||||
|
||||
鼓励末端执行器持续靠近把手。
|
||||
|
||||
2. **姿态奖励**
|
||||
|
||||
基于末端姿态与把手姿态的四元数相似度计算，并仅在末端距离把手较近时生效。
|
||||
|
||||
3. **夹爪闭合奖励**
|
||||
|
||||
当末端距离小于 `0.035m` 且左右手指在把手上下方对齐时，闭合夹爪可获得正奖励；
|
||||
否则闭合会受到惩罚。该项还会随夹爪闭合程度缩放。
|
||||
|
||||
4. **抽屉打开奖励**
|
||||
|
||||
```python
|
||||
open_reward = (exp(open_dist) - 1.0) * 420.0
|
||||
```
|
||||
|
||||
只有在满足“已经抓住或进入抓取阶段”且末端仍靠近把手时才生效。
|
||||
|
||||
5. **抽屉增量奖励**
|
||||
|
||||
对本步新增的打开距离给予额外奖励，鼓励稳定、连续地拉动抽屉。
|
||||
|
||||
6. **稳定抓取奖励**
|
||||
|
||||
当末端距离小于 `0.03m`、夹爪闭合比例大于 `0.7`，并持续保持 `6` 步后，环境认为已经抓住把手，并提供持续奖励。
|
||||
|
||||
7. **里程碑奖励**
|
||||
|
||||
- 抽屉打开超过 `0.15m` 时奖励 `35`
|
||||
- 抽屉打开超过 `0.22m` 时额外奖励 `70`
|
||||
|
||||
### 惩罚项
|
||||
|
||||
1. **滑脱惩罚**
|
||||
|
||||
如果已经进入抓取阶段但后来失去抓取，同时抽屉已经被拉开，则施加额外惩罚。
|
||||
|
||||
2. **手指穿透惩罚**
|
||||
|
||||
当手指接触点穿过把手上下边界时施加惩罚，用于抑制不合理接触。
|
||||
|
||||
3. **夹爪频繁切换惩罚**
|
||||
|
||||
当末端靠近把手时，频繁在开合之间切换会触发惩罚。
|
||||
|
||||
4. **动作变化惩罚**
|
||||
|
||||
对相邻两步动作差的平方和施加惩罚。
|
||||
|
||||
5. **关节速度惩罚**
|
||||
|
||||
对关节速度平方和施加惩罚，后期训练权重更高。
|
||||
|
||||
6. **终止惩罚**
|
||||
|
||||
当触发终止条件时，额外施加 `-10.0` 惩罚。
|
||||
|
||||
---
|
||||
|
||||
## 初始状态
|
||||
|
||||
### 机器人初始化
|
||||
|
||||
- 机械臂默认关节初值为零位姿
|
||||
- 夹爪默认处于张开状态
|
||||
- 复位时默认不添加关节角随机噪声
|
||||
- 所有关节速度初始化为零
|
||||
|
||||
### 场景初始化
|
||||
|
||||
- 目标抽屉初始为完全关闭状态
|
||||
- 抽屉与柜体其余部分固定在场景默认位置
|
||||
|
||||
### 随机化内容
|
||||
|
||||
每次 reset 会重新采样部分 sim2real 参数，包括：
|
||||
|
||||
- 机械臂动作延迟
|
||||
- 机械臂执行器滞后
|
||||
- 机械臂速度/加速度上限
|
||||
- 把手观测偏置
|
||||
|
||||
---
|
||||
|
||||
## Episode 终止条件
|
||||
|
||||
以下任一条件满足时，episode 会提前终止：
|
||||
|
||||
1. **TCP 落到把手后方过多**
|
||||
- 终止阈值：`tcp_x - handle_x < -0.02`
|
||||
2. **关节速度过大**
|
||||
- 任意机器人关节速度绝对值超过 `3.93 rad/s`
|
||||
|
||||
此外，环境最大时长为 `30s`。
|
||||
|
||||
---
|
||||
|
||||
## 使用方法
|
||||
|
||||
### 训练
|
||||
|
||||
```bash
|
||||
uv run scripts/train.py --env rm65-open-cabinet --train-backend torch
|
||||
```
|
||||
|
||||
### 策略评估
|
||||
|
||||
```bash
|
||||
uv run scripts/play.py --env rm65-open-cabinet
|
||||
```
|
||||
|
||||
### TensorBoard
|
||||
|
||||
```bash
|
||||
uv run tensorboard --logdir runs/rm65_open_cabinet
|
||||
```
|
||||
@@ -89,4 +89,7 @@ uv sync --all-packages --extra skrl-jax
|
||||
|
||||
# 安装 SKRL PyTorch
|
||||
uv sync --all-packages --extra skrl-torch
|
||||
|
||||
# 安装 RSLRL（仅支持 PyTorch）
|
||||
uv sync --all-packages --extra rslrl
|
||||
```
|
||||
|
||||
@@ -34,14 +34,18 @@ demo/dm_walker
|
||||
demo/dm_cheetah
|
||||
demo/dm_hopper
|
||||
demo/dm_reacher
|
||||
demo/dm_lqr
|
||||
demo/dm_finger
|
||||
demo/dm_humanoid
|
||||
demo/dm_quadruped
|
||||
demo/bring_ball
|
||||
demo/dm_point_mass
|
||||
demo/locomotion_unitree_go1
|
||||
demo/locomotion_unitree_go1_rough_terrain
|
||||
demo/anymal_c
|
||||
demo/franka_lift_cube
|
||||
demo/franka_open_cabinet
|
||||
demo/rm65_open_cabinet
|
||||
demo/shadow_hand_repose
|
||||
|
||||
```
|
||||
|
||||
@@ -16,6 +16,7 @@ MotrixLab/
|
||||
│ └── registry.py # 环境注册系统
|
||||
├── motrix_rl/ # 训练层：RL算法和配置
|
||||
│ ├── skrl/ # SKRL框架集成（JAX/PyTorch）
|
||||
│ ├── rslrl/ # RSLRL框架集成（PyTorch）
|
||||
│ ├── base.py # RL配置基类
|
||||
│ └── registry.py # RL配置注册系统
|
||||
└── scripts
|
||||
@@ -34,7 +35,7 @@ MotrixLab/
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 训练算法层 (SKRL) │
|
||||
│ 训练算法层 (SKRL / RSLRL) │
|
||||
│ PPO训练器 │ 网络架构 │ 优化器 │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
@@ -128,13 +129,15 @@ train.py --env cartpole
|
||||
- **奖励配置**影响学习信号（奖励权重、计算方式等）
|
||||
- **训练配置**控制算法行为（网络结构、学习率、批次大小等）
|
||||
|
||||
## 多后端支持
|
||||
## 多框架支持
|
||||
|
||||
MotrixLab 的分层设计天然支持多种后端：
|
||||
MotrixLab 的分层设计天然支持多种 RL 框架：
|
||||
|
||||
- **仿真后端**：MotrixSim
|
||||
- **训练后端**：JAX 和 PyTorch，支持 GPU 加速
|
||||
- **算法框架**：主要集成 SKRL，易于扩展其他算法
|
||||
- **仿真后端**：MotrixSim（CPU）
|
||||
- **训练框架**：
|
||||
- **SKRL**：支持 JAX 和 PyTorch 后端，支持 GPU 加速
|
||||
- **RSLRL**：支持 PyTorch 后端，支持 GPU 加速
|
||||
- **框架选择**：使用 `--rllib` 参数在 `skrl`（默认）和 `rslrl` 之间选择
|
||||
|
||||
## 设计优势
|
||||
|
||||
|
||||
@@ -7,41 +7,62 @@
|
||||
### 基本训练命令
|
||||
|
||||
```bash
|
||||
# 使用默认参数训练
|
||||
# 使用默认参数训练（SKRL 框架）
|
||||
uv run scripts/train.py --env cartpole
|
||||
|
||||
# 指定 RL 框架
|
||||
uv run scripts/train.py --env cartpole --rllib skrl
|
||||
uv run scripts/train.py --env cartpole --rllib rslrl
|
||||
|
||||
# 指定仿真后端
|
||||
uv run scripts/train.py --env cartpole --sim-backend np
|
||||
|
||||
# 指定训练后端
|
||||
uv run scripts/train.py --env cartpole --train-backend jax
|
||||
uv run scripts/train.py --env cartpole --train-backend torch
|
||||
# 指定训练后端（仅 SKRL）
|
||||
uv run scripts/train.py --env cartpole --rllib skrl --train-backend jax
|
||||
uv run scripts/train.py --env cartpole --rllib skrl --train-backend torch
|
||||
```
|
||||
|
||||
### 高级训练配置
|
||||
|
||||
```bash
|
||||
# 自定义训练参数
|
||||
# 使用 SKRL 自定义训练参数
|
||||
uv run scripts/train.py --env cartpole \
|
||||
--rllib skrl \
|
||||
--num-envs 1024 \
|
||||
--train-backend jax \
|
||||
--sim-backend np
|
||||
|
||||
# 使用 RSLRL 自定义训练参数
|
||||
uv run scripts/train.py --env cartpole \
|
||||
--rllib rslrl \
|
||||
--num-envs 1024 \
|
||||
--sim-backend np
|
||||
|
||||
# 注意：学习率等参数需要通过配置文件或代码覆盖设置
|
||||
|
||||
# 启用渲染监控训练过程
|
||||
uv run scripts/train.py --env cartpole --render
|
||||
```
|
||||
|
||||
### 不同框架配置
|
||||
|
||||
系统支持不同的 RL 框架，具有不同的配置系统：
|
||||
|
||||
- **SKRL 框架**：支持 JAX 和 PyTorch 训练后端，可通过 Python 数据类为每个后端配置参数
|
||||
- **RSLRL 框架**：支持 PyTorch 后端，通过 Python 数据类（RslrlCfg）进行配置
|
||||
|
||||
对于 SKRL，系统支持为不同训练后端（JAX/Torch）配置不同的强化学习参数。例如：
|
||||
|
||||
### 支持的命令行参数
|
||||
|
||||
| 参数 | 说明 | 默认值 |
|
||||
| ----------------- | -------------------- | ---------- |
|
||||
| `--env` | 环境名称 | `cartpole` |
|
||||
| `--sim-backend` | 仿真后端 (np) | 自动选择 |
|
||||
| `--train-backend` | 训练后端 (jax/torch) | 自动选择 |
|
||||
| `--num-envs` | 并行环境数量 | 2048 |
|
||||
| `--render` | 启用渲染 | False |
|
||||
| 参数 | 说明 | 默认值 |
|
||||
| ----------------- | ----------------------------- | ---------- |
|
||||
| `--env` | 环境名称 | `cartpole` |
|
||||
| `--rllib` | RL 框架 (skrl/rslrl) | `skrl` |
|
||||
| `--sim-backend` | 仿真后端 (np) | 自动选择 |
|
||||
| `--train-backend` | 训练后端 (jax/torch，仅 SKRL) | 自动选择 |
|
||||
| `--num-envs` | 并行环境数量 | 2048 |
|
||||
| `--render` | 启用渲染 | False |
|
||||
|
||||
> **注意**: 其他参数如学习率、网络结构等需要通过单独文件设置。
|
||||
|
||||
|
||||
@@ -2,85 +2,80 @@
|
||||
|
||||
MotrixLab 提供了灵活的配置系统，允许用户自定义强化学习训练参数。本节介绍如何配置训练环境和强化学习算法参数。
|
||||
|
||||
## RL 训练配置 (PPOCfg)
|
||||
## RL 训练配置
|
||||
|
||||
MotrixLab 支持多个 RL 框架，具有不同的配置系统：
|
||||
|
||||
- **SKRL 框架**：使用 Python 数据类配置（SkrlCfg）
|
||||
- **RSLRL 框架**：使用 Python 数据类配置（RslrlCfg）
|
||||
|
||||
### SKRL 配置 (SkrlCfg)
|
||||
|
||||
训练配置定义了基于 PPO 算法的强化学习算法的参数。MotrixLab 现在支持为不同训练后端配置不同的参数。
|
||||
|
||||
### 完整配置示例
|
||||
#### 完整配置示例
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class CompletePPOConfig(PPOCfg):
|
||||
"""
|
||||
完整的强化学习训练配置示例
|
||||
包含了从基础到高级的所有配置参数
|
||||
"""
|
||||
以下是 `CartPoleSkrlPpo` 的实际配置，展示了完整的显式参数填充方法。该配置使用较小的网络 `[32, 32]`，适合 CartPole 这样的简单任务。
|
||||
|
||||
# ===== 基础训练参数 =====
|
||||
seed: Optional[int] = None # 随机种子
|
||||
num_envs: int = 2048 # 训练时并行环境数量
|
||||
play_num_envs: int = 16 # 评估时并行环境数量
|
||||
max_env_steps: int = 2_048_000 # 最大训练步数
|
||||
check_point_interval: int = 1000 # 检查点保存间隔
|
||||
|
||||
# ===== PPO算法核心参数 =====
|
||||
learning_rate: float = 3e-4 # 学习率
|
||||
rollouts: int = 32 # 经验回放轮数
|
||||
learning_epochs: int = 2 # 每次更新的训练轮数
|
||||
mini_batches: int = 32 # 小批量数量
|
||||
discount_factor: float = 0.99 # 折扣因子
|
||||
lambda_param: float = 0.95 # GAE参数
|
||||
grad_norm_clip: float = 1.0 # 梯度裁剪
|
||||
|
||||
# ===== PPO裁剪参数 =====
|
||||
ratio_clip: float = 0.2 # PPO裁剪比率
|
||||
value_clip: float = 0.2 # 价值裁剪
|
||||
clip_predicted_values: bool = True # 裁剪预测值
|
||||
|
||||
# ===== 损失函数参数 =====
|
||||
entropy_loss_scale: float = 0.0 # 熵损失系数
|
||||
value_loss_scale: float = 2.0 # 价值损失系数
|
||||
kl_threshold: float = 0 # KL散度阈值
|
||||
|
||||
# ===== 学习率调度器 =====
|
||||
learning_rate_scheduler_kl_threshold: float = 0.008 # 自适应学习率KL阈值
|
||||
|
||||
# ===== 网络架构配置 =====
|
||||
# 小型网络（适合简单任务如 CartPole）
|
||||
# policy_hidden_layer_sizes: tuple[int, ...] = (128, 64)
|
||||
# value_hidden_layer_sizes: tuple[int, ...] = (128, 64)
|
||||
|
||||
# 中型网络（默认配置，适合大部分任务）
|
||||
policy_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64)
|
||||
value_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64)
|
||||
|
||||
# 大型网络（适合复杂任务如机器人控制）
|
||||
# policy_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
|
||||
# value_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
|
||||
|
||||
# ===== 网络共享配置 =====
|
||||
share_policy_value_features: bool = True # 策略和价值网络共享特征提取层
|
||||
|
||||
# ===== 训练控制参数 =====
|
||||
random_timesteps: int = 0 # 随机步数
|
||||
learning_starts: int = 0 # 开始学习的步数
|
||||
time_limit_bootstrap: bool = True # 时间限制引导
|
||||
|
||||
# ===== 奖励整形 =====
|
||||
rewards_shaper_scale: float = 1.0 # 奖励缩放因子
|
||||
```{literalinclude} ../../../../motrix_rl/src/motrix_rl/tasks/cartpole.py
|
||||
:language: python
|
||||
:start-after: docs-start: cartpole-skrl-config
|
||||
:end-before: docs-end: cartpole-skrl-config
|
||||
```
|
||||
|
||||
**关键配置说明：**
|
||||
|
||||
- **网络架构**: `hiddens=[32, 32]` - CartPole 是简单任务，使用小网络即可（默认: `[256, 128, 64]`）
|
||||
- **训练轮数**: `learning_epochs=5` - 比默认值 2 更高，确保充分学习
|
||||
- **小批量数量**: `mini_batches=4` - 比默认值 32 更少，适合简单任务
|
||||
- **训练时长**: `timesteps=5000` - 对于 CartPole 来说已经足够（默认: 10000）
|
||||
- **所有参数**: 从父类继承的所有参数都被显式指定，无隐藏默认值
|
||||
|
||||
完整的源代码请参考: [`motrix_rl/src/motrix_rl/tasks/cartpole.py`](https://github.com/Motphys/motrix-lab/blob/main/motrix_rl/src/motrix_rl/tasks/cartpole.py)
|
||||
|
||||
### RSLRL 配置 (RslrlCfg)
|
||||
|
||||
RSLRL 是另一个高性能强化学习库，专门用于四足机器人等复杂控制任务。
|
||||
|
||||
#### 完整配置示例
|
||||
|
||||
以下是 `CartPoleRslrlPpo` 的实际配置，展示了完整的显式参数填充方法。该配置使用较小的网络 `[32, 32]`，适合 CartPole 这样的简单任务。
|
||||
|
||||
```{literalinclude} ../../../../motrix_rl/src/motrix_rl/tasks/cartpole.py
|
||||
:language: python
|
||||
:start-after: docs-start: cartpole-rslrl-config
|
||||
:end-before: docs-end: cartpole-rslrl-config
|
||||
```
|
||||
|
||||
**关键配置说明：**
|
||||
|
||||
- **网络架构**: `hidden_dims=[32, 32]` - CartPole 是简单任务，使用小网络即可（默认: `[256, 128, 64]`）
|
||||
- **训练迭代**: `max_iterations=300` - 总共训练 300 次迭代
|
||||
- **每轮步数**: `num_steps_per_env=16` - 每个环境收集 16 步
|
||||
- **学习率**: `learning_rate=5.0e-4` - 学习率设置
|
||||
- **熵系数**: `entropy_coef=5e-3` - 熵系数，用于探索
|
||||
- **所有参数**: 从父类继承的所有参数都被显式指定，无隐藏默认值
|
||||
|
||||
详细的 RSLRL 配置选项和默认值，请参考：
|
||||
|
||||
- `motrix_rl/rslrl/cfg.py`：配置类定义
|
||||
- `motrix_rl/template/rslrl_config.yaml`：YAML 参考模板
|
||||
|
||||
## 配置使用方法
|
||||
|
||||
### 1. 默认配置使用
|
||||
|
||||
```bash
|
||||
# 使用代码中给定的配置
|
||||
# 使用代码中给定的配置（默认：SKRL 框架）
|
||||
uv run scripts/train.py --env my-task
|
||||
|
||||
# 指定训练后端，系统会自动选择对应的后端配置
|
||||
uv run scripts/train.py --env my-task --train-backend jax
|
||||
uv run scripts/train.py --env my-task --train-backend torch
|
||||
# 指定 RL 框架
|
||||
uv run scripts/train.py --env my-task --rllib skrl
|
||||
uv run scripts/train.py --env my-task --rllib rslrl
|
||||
|
||||
# 指定 SKRL 的训练后端，系统会自动选择对应的后端配置
|
||||
uv run scripts/train.py --env my-task --rllib skrl --train-backend jax
|
||||
uv run scripts/train.py --env my-task --rllib skrl --train-backend torch
|
||||
```
|
||||
|
||||
### 2. 命令行参数覆盖
|
||||
@@ -88,91 +83,10 @@ uv run scripts/train.py --env my-task --train-backend torch
|
||||
```bash
|
||||
# 覆盖支持的命令行参数
|
||||
uv run scripts/train.py --env my-task \
|
||||
--rllib skrl \
|
||||
--num-envs 1024 \
|
||||
--train-backend jax \
|
||||
--sim-backend np
|
||||
|
||||
# 系统会自动选择JAX后端对应的配置
|
||||
```
|
||||
|
||||
### 3. 配置优先级
|
||||
|
||||
系统按以下优先级选择配置：
|
||||
|
||||
1. **后端特定配置**: 如果存在 `@rlcfg(env_name, backend="jax/torch")` 装饰的配置
|
||||
2. **通用配置**: 如果存在 `@rlcfg(env_name)` 装饰的配置（无 backend 参数）
|
||||
|
||||
例如：
|
||||
|
||||
```python
|
||||
# 最高优先级 - 后端特定配置
|
||||
@rlcfg("my-task", backend="jax")
|
||||
@dataclass
|
||||
class MyTaskJAXCfg(PPOCfg):
|
||||
mini_batches: int = 4
|
||||
|
||||
# 次优先级 - 通用配置
|
||||
@rlcfg("my-task")
|
||||
@dataclass
|
||||
class MyTaskRLCfg(PPOCfg):
|
||||
mini_batches: int = 32
|
||||
|
||||
# 当使用 --train-backend jax 时，系统会选择 MyTaskJAXCfg
|
||||
# 当使用 --train-backend torch 时，系统会选择 MyTaskRLCfg
|
||||
```
|
||||
|
||||
## SKRL 框架配置映射
|
||||
|
||||
在 MotrixLab 中，用户通过 `PPOCfg` 配置类设置参数，这些参数会被映射到 SKRL 框架的配置字典中。
|
||||
|
||||
### 用户可配置参数
|
||||
|
||||
| MotrixLab 配置类 | SKRL 框架参数 | 说明 |
|
||||
| -------------------------------------- | --------------------------------------------- | -------------------- |
|
||||
| `learning_rate` | `learning_rate` | 学习率 |
|
||||
| `rollouts` | `rollouts` | 经验回放轮数 |
|
||||
| `learning_epochs` | `learning_epochs` | 训练轮数 |
|
||||
| `mini_batches` | `mini_batches` | 小批量数量 |
|
||||
| `discount_factor` | `discount_factor` | 折扣因子 |
|
||||
| `grad_norm_clip` | `grad_norm_clip` | 梯度裁剪 |
|
||||
| `lambda_param` | `lambda` | GAE 参数 |
|
||||
| `ratio_clip` | `ratio_clip` | PPO 裁剪比率 |
|
||||
| `value_clip` | `value_clip` | 价值裁剪 |
|
||||
| `clip_predicted_values` | `clip_predicted_values` | 裁剪预测值 |
|
||||
| `entropy_loss_scale` | `entropy_loss_scale` | 熵损失系数 |
|
||||
| `value_loss_scale` | `value_loss_scale` | 价值损失系数 |
|
||||
| `kl_threshold` | `kl_threshold` | KL 散度阈值 |
|
||||
| `random_timesteps` | `random_timesteps` | 随机步数 |
|
||||
| `learning_starts` | `learning_starts` | 开始学习的步数 |
|
||||
| `time_limit_bootstrap` | `time_limit_bootstrap` | 时间限制引导 |
|
||||
| `learning_rate_scheduler_kl_threshold` | `learning_rate_scheduler_kwargs.kl_threshold` | 自适应学习率 KL 阈值 |
|
||||
| `check_point_interval` | `experiment.write_interval` | 日志写入间隔 |
|
||||
| `check_point_interval` | `experiment.checkpoint_interval` | 检查点保存间隔 |
|
||||
| `rewards_shaper_scale` | `rewards_shaper` | 奖励缩放函数 |
|
||||
|
||||
### 预处理器参数
|
||||
|
||||
| SKRL 框架参数 | 类型 | 说明 |
|
||||
| -------------------- | --------------------- | ---------- |
|
||||
| `state_preprocessor` | RunningStandardScaler | 状态标准化 |
|
||||
| `value_preprocessor` | RunningStandardScaler | 价值标准化 |
|
||||
|
||||
### 配置层次总结
|
||||
|
||||
```
|
||||
用户配置类 (PPOCfg)
|
||||
↓ 后端特定选择
|
||||
后端配置 (JAX/Torch)
|
||||
↓ 参数映射
|
||||
SKRL 框架配置字典
|
||||
↓ 传递给
|
||||
PPO Agent
|
||||
↓ 执行
|
||||
强化学习训练
|
||||
```
|
||||
|
||||
这种设计允许用户：
|
||||
|
||||
1. 通过简单的配置类来控制复杂的训练参数
|
||||
2. 为不同训练后端配置不同的参数以获得最佳性能
|
||||
3. 保持与 SKRL 框架的完全兼容性
|
||||
|
||||
Reference in New Issue
Block a user