cyy/Motrixlab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: release v0.3.0

Browse Source
This commit is contained in:
motphys-developers
2026-04-02 03:45:10 +00:00
parent c84d382b8c
commit e1421d1055
232 changed files with 20258 additions and 2004 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
docs/source/en/user_guide/tutorial/basic_frame.md
View File

@@ -16,6 +16,7 @@ MotrixLab/
│ └── registry.py # Environment registry system
├── motrix_rl/ # Training layer: RL algorithms and configuration
│ ├── skrl/ # SKRL framework integration (JAX/PyTorch)
│ ├── rslrl/ # RSLRL framework integration (PyTorch)
│ ├── base.py # RL configuration base class
│ └── registry.py # RL configuration registry system
└── scripts
@@ -34,7 +35,7 @@ MotrixLab/
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Training Algorithm Layer (SKRL) │
│ Training Algorithm Layer (SKRL / RSLRL) │
│ PPO Trainer │ Network Architecture │ Optimizer │
└─────────────────────────────────────────────────────────────────┘
│
@@ -128,13 +129,15 @@ Configuration parameters play a key connecting role throughout the process:
- **Reward Configuration** affects learning signals (reward weights, calculation methods, etc.)
- **Training Configuration** controls algorithm behavior (network structure, learning rate, batch size, etc.)
## Multi-Backend Support
## Multi-Framework Support
MotrixLab's layered design naturally supports multiple backends:
MotrixLab's layered design naturally supports multiple RL frameworks:
- **Simulation Backends**: MotrixSim (CPU)
- **Training Backends**: JAX and PyTorch, supporting GPU acceleration
- **Algorithm Framework**: Mainly integrates SKRL, easy to extend to other algorithms
- **Training Frameworks**:
- **SKRL**: Supports JAX and PyTorch backends with GPU acceleration
- **RSLRL**: Supports PyTorch backend with GPU acceleration
- **Framework Selection**: Use `--rllib` parameter to choose between `skrl` (default) and `rslrl`
## Design Advantages

58
docs/source/en/user_guide/tutorial/training_and_result.md
View File

@@ -7,56 +7,64 @@ This section introduces how to execute reinforcement learning training and how t
### Basic Training Commands
```bash
# Train with default parameters
# Train with default parameters (SKRL framework)
uv run scripts/train.py --env cartpole
# Specify RL framework
uv run scripts/train.py --env cartpole --rllib skrl
uv run scripts/train.py --env cartpole --rllib rslrl
# Specify simulation backend
uv run scripts/train.py --env cartpole --sim-backend np
# Specify training backend
uv run scripts/train.py --env cartpole --train-backend jax
uv run scripts/train.py --env cartpole --train-backend torch
# Specify training backend (SKRL only)
uv run scripts/train.py --env cartpole --rllib skrl --train-backend jax
uv run scripts/train.py --env cartpole --rllib skrl --train-backend torch
```
### Advanced Training Configuration
```bash
# Customize training parameters
# Customize training parameters with SKRL
uv run scripts/train.py --env cartpole \
--rllib skrl \
--num-envs 1024 \
--train-backend jax \
--sim-backend np
# Customize training parameters with RSLRL
uv run scripts/train.py --env cartpole \
--rllib rslrl \
--num-envs 1024 \
--sim-backend np
# Note: Parameters like learning rate need to be set through configuration files or code override
# Enable rendering to monitor training process
uv run scripts/train.py --env cartpole --render
```
### Different Backend Configuration Differences
### Different Framework Configuration
The system supports configuring different reinforcement learning parameters for different training backends (JAX/Torch). For example:
The system supports different RL frameworks with different configuration systems:
- **dm-walker environment**:
- **SKRL Framework**: Supports JAX and PyTorch training backends with configurable parameters per backend (via Python dataclasses)
- **RSLRL Framework**: Supports PyTorch backend with configuration via Python dataclasses (RslrlCfg)
- JAX backend: `mini_batches: 4`
- Torch backend: `mini_batches: 32`
- **dm-runner environment**:
- JAX backend: `learning_epochs: 4`
- Torch backend: `learning_epochs: 2`
These differences are implemented through the `@rlcfg(env_name, backend="jax/torch")` decorator in configuration classes. The system automatically applies the corresponding configuration based on the selected training backend.
For SKRL, the system supports configuring different reinforcement learning parameters for different training backends (JAX/Torch).
### Supported Command Line Parameters
| Parameter | Description | Default Value |
| ----------------- | ------------------------------- | ------------- |
| `--env` | Environment name | `cartpole` |
| `--sim-backend` | Simulation backend (np) | Auto select |
| `--train-backend` | Training backend (jax/torch) | Auto select |
| `--num-envs` | Number of parallel environments | 2048 |
| `--render` | Enable rendering | False |
| Parameter | Description | Default Value |
| ----------------- | --------------------------------------- | ------------- |
| `--env` | Environment name | `cartpole` |
| `--rllib` | RL framework (skrl/rslrl) | `skrl` |
| `--sim-backend` | Simulation backend (np) | Auto select |
| `--train-backend` | Training backend (jax/torch, SKRL only) | Auto select |
| `--num-envs` | Number of parallel environments | 2048 |
| `--render` | Enable rendering | False |
> **Note**: Other parameters such as learning rate, network structure, etc., can be set in configuration files. Some environments support configuring different parameters for different training backends.
> **Note**: Other parameters such as learning rate, network structure, etc., need to be set through separate configuration files.
## Training Process Monitoring
@@ -83,7 +91,7 @@ uv run tensorboard --logdir runs/cartpole
uv run scripts/play.py --env cartpole
# Manually specify policy file for testing
uv run scripts/play.py --env cartpole --policy runs/cartpole/nn/best_policy.pickle
uv run scripts/play.py --env cartpole --policy runs/cartpole/nn/best_agent.pickle
# Specify number of test environments
uv run scripts/play.py --env cartpole --num-envs 100

241
docs/source/en/user_guide/tutorial/training_environment_config.md
View File

@@ -2,141 +2,80 @@
MotrixLab provides a flexible configuration system that allows users to customize reinforcement learning training parameters. This section introduces how to configure training environments and reinforcement learning algorithm parameters.
## RL Training Configuration (PPOCfg)
## RL Training Configuration
Training configuration defines parameters for reinforcement learning algorithms. MotrixLab now supports configuring different parameters for different training backends.
MotrixLab supports multiple RL frameworks with different configuration systems:
### Basic Training Configuration
- **SKRL Framework**: Uses Python dataclass configuration (SkrlCfg)
- **RSLRL Framework**: Uses Python dataclass configuration (RslrlCfg)
```python
from dataclasses import dataclass
from motrix_rl.skrl.cfg import PPOCfg
from motrix_rl import registry
### SKRL Configuration (SkrlCfg)
# Universal configuration (applies to all backends)
@registry.rlcfg("my-task")
@dataclass
class MyTaskRLCfg(PPOCfg):
# Environment parameters
num_envs: int = 2048 # Number of parallel environments during training
play_num_envs: int = 16 # Number of parallel environments during evaluation
Training configuration defines parameters for reinforcement learning algorithms based on PPO. MotrixLab now supports configuring different parameters for different training backends.
# PPO algorithm parameters
learning_rate: float = 3e-4 # Learning rate
rollouts: int = 32 # Experience replay rounds
learning_epochs: int = 10 # Number of epochs per update
mini_batches: int = 32 # Number of mini-batches
discount_factor: float = 0.99 # Discount factor
grad_norm_clip: float = 1.0 # Gradient clipping
#### Complete Configuration Example
# Network structure parameters
policy_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64) # Policy network hidden layers
value_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64) # Value network hidden layers
The following is the actual `CartPoleSkrlPpo` configuration, demonstrating complete explicit parameter filling. This configuration uses a smaller network `[32, 32]` suitable for simple tasks like CartPole.
# Training control parameters
max_env_steps: int = 1_000_000 # Maximum environment steps
check_point_interval: int = 10_000 # Checkpoint interval
```{literalinclude} ../../../../motrix_rl/src/motrix_rl/tasks/cartpole.py
:language: python
:start-after: docs-start: cartpole-skrl-config
:end-before: docs-end: cartpole-skrl-config
```
### Different Backend Configurations
**Key Configuration Notes:**
```python
# JAX backend specific configuration
@registry.rlcfg("my-task", backend="jax")
@dataclass
class MyTaskJAXCfg(PPOCfg):
# JAX optimized configuration
learning_rate: float = 2e-4 # JAX backend uses smaller learning rate
mini_batches: int = 4 # JAX supports large batches, fewer mini-batches
learning_epochs: int = 4 # Training rounds
num_envs: int = 2048 # More parallel environments
- **Network Architecture**: `hiddens=[32, 32]` - CartPole is a simple task, a small network is sufficient (default: `[256, 128, 64]`)
- **Training Epochs**: `learning_epochs=5` - Higher than default value 2, ensuring thorough learning
- **Mini-batches**: `mini_batches=4` - Fewer than default 32, suitable for simple tasks
- **Training Duration**: `timesteps=5000` - Sufficient for CartPole (default: 10000)
- **All Parameters**: All parameters from parent classes are explicitly specified, no hidden defaults
# Torch backend specific configuration
@registry.rlcfg("my-task", backend="torch")
@dataclass
class MyTaskTorchCfg(PPOCfg):
# Torch optimized configuration
learning_rate: float = 1e-4 # Torch backend uses even smaller learning rate
mini_batches: int = 32 # Torch needs more mini-batches
learning_epochs: int = 2 # Fewer training rounds
num_envs: int = 1024 # Fewer parallel environments
For complete source code, see: [`motrix_rl/src/motrix_rl/tasks/cartpole.py`](https://github.com/Motphys/motrix-lab/blob/main/motrix_rl/src/motrix_rl/tasks/cartpole.py)
### RSLRL Configuration (RslrlCfg)
RSLRL is another high-performance reinforcement learning library, specifically designed for complex control tasks like quadruped robots.
#### Complete Configuration Example
The following is the actual `CartPoleRslrlPpo` configuration, demonstrating complete explicit parameter filling. This configuration uses a smaller network `[32, 32]` suitable for simple tasks like CartPole.
```{literalinclude} ../../../../motrix_rl/src/motrix_rl/tasks/cartpole.py
:language: python
:start-after: docs-start: cartpole-rslrl-config
:end-before: docs-end: cartpole-rslrl-config
```
### Complete Configuration Example
**Key Configuration Notes:**
```python
@dataclass
class CompletePPOConfig(PPOCfg):
"""
Complete reinforcement learning training configuration example
Contains all configuration parameters from basic to advanced
"""
- **Network Architecture**: `hidden_dims=[32, 32]` - CartPole is a simple task, a small network is sufficient (default: `[256, 128, 64]`)
- **Training Iterations**: `max_iterations=300` - Total of 300 training iterations
- **Steps per Environment**: `num_steps_per_env=16` - Number of steps to collect per environment
- **Learning Rate**: `learning_rate=5.0e-4` - Learning rate setting
- **Entropy Coefficient**: `entropy_coef=5e-3` - Entropy coefficient for exploration
- **All Parameters**: All parameters from parent classes are explicitly specified, no hidden defaults
# ===== Basic Training Parameters =====
seed: Optional[int] = None # Random seed
num_envs: int = 2048 # Number of parallel environments during training
play_num_envs: int = 16 # Number of parallel environments during evaluation
max_env_steps: int = 2_048_000 # Maximum training steps
check_point_interval: int = 1000 # Checkpoint save interval
For detailed RSLRL configuration options and default values, refer to:
# ===== PPO Algorithm Core Parameters =====
learning_rate: float = 3e-4 # Learning rate
rollouts: int = 32 # Experience replay rounds
learning_epochs: int = 2 # Number of training rounds per update
mini_batches: int = 32 # Number of mini-batches
discount_factor: float = 0.99 # Discount factor
lambda_param: float = 0.95 # GAE parameter
grad_norm_clip: float = 1.0 # Gradient clipping
# ===== PPO Clipping Parameters =====
ratio_clip: float = 0.2 # PPO clipping ratio
value_clip: float = 0.2 # Value clipping
clip_predicted_values: bool = True # Clip predicted values
# ===== Loss Function Parameters =====
entropy_loss_scale: float = 0.0 # Entropy loss coefficient
value_loss_scale: float = 2.0 # Value loss coefficient
kl_threshold: float = 0 # KL divergence threshold
# ===== Learning Rate Scheduler =====
learning_rate_scheduler_kl_threshold: float = 0.008 # Adaptive learning rate KL threshold
# ===== Network Architecture Configuration =====
# Small network (suitable for simple tasks like CartPole)
# policy_hidden_layer_sizes: tuple[int, ...] = (128, 64)
# value_hidden_layer_sizes: tuple[int, ...] = (128, 64)
# Medium network (default configuration, suitable for most tasks)
policy_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64)
value_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64)
# Large network (suitable for complex tasks like robot control)
# policy_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
# value_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
# ===== Network Sharing Configuration =====
share_policy_value_features: bool = True # Policy and value networks share feature extraction layers
# ===== Training Control Parameters =====
random_timesteps: int = 0 # Random timesteps
learning_starts: int = 0 # Timesteps to start learning
time_limit_bootstrap: bool = True # Time limit bootstrap
# ===== Reward Shaping =====
rewards_shaper_scale: float = 1.0 # Reward scaling factor
```
- `motrix_rl/rslrl/cfg.py`: Configuration class definitions
- `motrix_rl/template/rslrl_config.yaml`: YAML reference template
## Configuration Usage Methods
### 1. Default Configuration Usage
```bash
# Use configuration given in code
# Use configuration given in code (default: SKRL framework)
uv run scripts/train.py --env my-task
# Specify training backend, system will automatically select corresponding backend configuration
uv run scripts/train.py --env my-task --train-backend jax
uv run scripts/train.py --env my-task --train-backend torch
# Specify RL framework
uv run scripts/train.py --env my-task --rllib skrl
uv run scripts/train.py --env my-task --rllib rslrl
# Specify training backend for SKRL, system will automatically select corresponding backend configuration
uv run scripts/train.py --env my-task --rllib skrl --train-backend jax
uv run scripts/train.py --env my-task --rllib skrl --train-backend torch
```
### 2. Command Line Parameter Override
@@ -144,88 +83,10 @@ uv run scripts/train.py --env my-task --train-backend torch
```bash
# Override supported command line parameters
uv run scripts/train.py --env my-task \
--rllib skrl \
--num-envs 1024 \
--train-backend jax \
--sim-backend np
# System will automatically select JAX backend configuration
```
### 3. Configuration Priority
System selects configuration in the following priority:
1. **Backend-specific configuration**: If there is a configuration decorated with `@rlcfg(env_name, backend="jax/torch")`
2. **Universal configuration**: If there is a configuration decorated with `@rlcfg(env_name)` (no backend parameter)
3. **Configuration override**: Command line `cfg_override` parameter
For example:
```python
# Highest priority - backend specific configuration
@rlcfg("my-task", backend="jax")
@dataclass
class MyTaskJAXCfg(PPOCfg):
mini_batches: int = 4
# Second priority - universal configuration
@rlcfg("my-task")
@dataclass
class MyTaskRLCfg(PPOCfg):
mini_batches: int = 32
# When using --train-backend jax, system will select MyTaskJAXCfg
# When using --train-backend torch, system will select MyTaskRLCfg
```
#### User Configurable Parameters
| MotrixLab Configuration Class | SKRL Framework Parameter | Description |
| -------------------------------------- | --------------------------------------------- | ----------------------------------- |
| `learning_rate` | `learning_rate` | Learning rate |
| `rollouts` | `rollouts` | Experience replay rounds |
| `learning_epochs` | `learning_epochs` | Training rounds |
| `mini_batches` | `mini_batches` | Number of mini-batches |
| `discount_factor` | `discount_factor` | Discount factor |
| `grad_norm_clip` | `grad_norm_clip` | Gradient clipping |
| `lambda_param` | `lambda` | GAE parameter |
| `ratio_clip` | `ratio_clip` | PPO clipping ratio |
| `value_clip` | `value_clip` | Value clipping |
| `clip_predicted_values` | `clip_predicted_values` | Clip predicted values |
| `entropy_loss_scale` | `entropy_loss_scale` | Entropy loss coefficient |
| `value_loss_scale` | `value_loss_scale` | Value loss coefficient |
| `kl_threshold` | `kl_threshold` | KL divergence threshold |
| `random_timesteps` | `random_timesteps` | Random timesteps |
| `learning_starts` | `learning_starts` | Learning start timesteps |
| `time_limit_bootstrap` | `time_limit_bootstrap` | Time limit bootstrap |
| `learning_rate_scheduler_kl_threshold` | `learning_rate_scheduler_kwargs.kl_threshold` | Adaptive learning rate KL threshold |
| `check_point_interval` | `experiment.write_interval` | Log write interval |
| `check_point_interval` | `experiment.checkpoint_interval` | Checkpoint save interval |
| `rewards_shaper_scale` | `rewards_shaper` | Reward scaling function |
#### Preprocessor Parameters
| SKRL Framework Parameter | Type | Description |
| ------------------------ | --------------------- | ------------------- |
| `state_preprocessor` | RunningStandardScaler | State normalization |
| `value_preprocessor` | RunningStandardScaler | Value normalization |
### Configuration Hierarchy Summary
```
User Configuration Class (PPOCfg)
↓ Backend specific selection
Backend Configuration (JAX/Torch)
↓ Parameter mapping
SKRL Framework Configuration Dictionary
↓ Pass to
PPO Agent
↓ Execute
Reinforcement Learning Training
```
This design allows users to:
1. Control complex training parameters through simple configuration classes
2. Configure different parameters for different training backends to achieve optimal performance
3. Maintain full compatibility with the SKRL framework