cyy/Motrixlab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: release v0.0.1

Browse Source
This commit is contained in:
motphys-developers
2025-11-20 08:57:48 +00:00
commit 5133830b5a
105 changed files with 8789 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

149
docs/source/en/user_guide/tutorial/basic_frame.md Normal file
View File

@@ -0,0 +1,149 @@
# Basic Framework
MotrixLab is a robot reinforcement learning platform. This section introduces MotrixLab's framework design and the relationships between various components. If you are already familiar with reinforcement learning content, you can skip directly to the next section to learn how to develop your own training environments.
## MotrixLab's Framework Design
MotrixLab adopts a layered architecture design, clearly separating training environments from training logic:
```
MotrixLab/
├── motrix_envs/ # Environment layer: Physics simulation and task definition
│ ├── basic/ # Basic environments (cartpole, walker, etc.)
│ ├── locomotion/ # Locomotion environments (GO1 robot, etc.)
│ ├── np/ # NumPy simulation backend framework
│ ├── base.py # Environment base class
│ └── registry.py # Environment registry system
├── motrix_rl/ # Training layer: RL algorithms and configuration
│ ├── skrl/ # SKRL framework integration (JAX/PyTorch)
│ ├── base.py # RL configuration base class
│ └── registry.py # RL configuration registry system
└── scripts
├── train.py # Training entry script
├── play.py # Testing entry script
└── view.py # Visualization script
```
## Core Component Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ User Interface Layer │
│ train.py │ play.py │ view.py │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Training Algorithm Layer (SKRL) │
│ PPO Trainer │ Network Architecture │ Optimizer │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Environment Implementation Layer │
│ Environment Config(EnvCfg) │ Environment Impl(Env) │ Reward │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Physics Simulation Layer (MotrixSim) │
│ MJCF Model │ Physics Engine │ Collision │
└─────────────────────────────────────────────────────────────────┘
```
## Detailed Core Components
### 1. Training Environment
**Location**: Environment Implementation Layer
The training environment is the core component of MotrixLab, containing three key parts:
- **Environment Configuration (EnvCfg)**: Defines physics simulation parameters (model files, time steps, episode length, etc.) and task-specific parameters
- **Environment Implementation (Env)**: Inherits from base environment class, implements specific task logic, physics simulation interaction, and termination condition checking
- **Reward Function (Reward)**: Implemented in the environment's step method, calculates reward values based on current state and actions
Environments are registered to the system through decorators.
### 2. Reward Function
**Location**: Configuration Management Layer + Environment Implementation Layer
The reward function in MotrixLab adopts a dual-structure design:
- **Configuration Level**: Define reward weights, reward component types, and scaling parameters in configuration classes
- **Implementation Level**: Calculate specific reward values in the environment's `_compute_reward` method based on configuration parameters
This design allows reward functions to be flexibly adjusted through configuration files while implementing complex computational logic in code.
### 3. Configuration Parameters
**Location**: Configuration Management Layer
Configuration parameters adopt a hierarchical management structure:
- **Environment Configuration (EnvCfg)**: Controls physics simulation and task behavior, including simulation parameters, reset noise, time limits, etc.
- **Training Configuration (RLCfg)**: Controls reinforcement learning algorithms, including network structure, learning rate, batch size, training steps, etc.
Configuration classes support inheritance, parameter validation, and runtime overriding, ensuring parameter reasonableness and flexibility.
### 4. Registry System
**Location**: Hub connecting various components
The registry system implements automatic component registration through the decorator pattern:
- Environment configuration classes are registered through `@registry.envcfg()`
- Environment implementation classes are registered through `@registry.env()`, supporting multiple backends
- RL configuration classes are registered through `@registry.rlcfg()`
The registry system achieves component decoupling, making it simple and fast to add new environments or modify configurations.
## Data Flow and Workflow
### Training Process Overview
```
User Command → Configuration Parsing → Environment Creation → Training Loop → Model Save
↓
train.py --env cartpole
↓
Find Configuration Classes → Create Environment → Start PPO Training → Save Model
```
### Core Workflow
1. **Environment Definition**: Create environment configuration classes and implementation classes in `src/motrix_envs/`
2. **Automatic Registration**: Register components to the system through decorators
3. **Configuration Loading**: When starting from command line, the system automatically finds and loads corresponding configurations
4. **Environment Creation**: Factory pattern creates environment instances, supporting parameter override
5. **Training Execution**: PPO algorithm interacts with the environment, collects data and updates policy
6. **Result Saving**: Periodically save checkpoints and final models
### Role of Configuration Parameters
Configuration parameters play a key connecting role throughout the process:
- **Environment Configuration** determines physics simulation behavior (time steps, model files, noise, etc.)
- **Reward Configuration** affects learning signals (reward weights, calculation methods, etc.)
- **Training Configuration** controls algorithm behavior (network structure, learning rate, batch size, etc.)
## Multi-Backend Support
MotrixLab's layered design naturally supports multiple backends:
- **Simulation Backends**: MotrixSim (CPU)
- **Training Backends**: JAX and PyTorch, supporting GPU acceleration
- **Algorithm Framework**: Mainly integrates SKRL, easy to extend to other algorithms
## Design Advantages
This architecture design brings the following core advantages:
1. **Module Decoupling**: Environment development and training logic are completely separated
2. **Flexible Configuration**: Supports hierarchical configuration and runtime parameter override
3. **Strong Extensibility**: Easily add new components through the registry system
4. **Multi-Backend Compatibility**: Same environment can use different simulation and training backends
5. **Experiment-Friendly**: Configurations can be saved and compared, ensuring experimental reproducibility
Through this framework design, MotrixLab provides a clear, flexible, and easy-to-use development platform for robot reinforcement learning.

66
docs/source/en/user_guide/tutorial/physics_environment.md Normal file
View File

@@ -0,0 +1,66 @@
# Physics Environment Configuration
Physics environment configuration defines simulation parameters and model file settings in reinforcement learning training.
MotrixLab uses [MotrixSim](https://motrixsim.readthedocs.io/en/latest/user_guide/index.html) as the physics simulation backend.
## Supported File Formats
- [**MJCF**](https://mujoco.readthedocs.io/en/stable/XMLreference.html) (MuJoCo XML format) - Provides rich physics features and simulation configuration
## Model File Configuration
You need to specify model file paths in environment configuration classes:
```python
@registry.envcfg("my-task")
@dataclass
class MyTaskEnvCfg(EnvCfg):
# Model file path (required)
model_file: str = "my_model.xml"
# Simulation time parameters
sim_dt: float = 0.002 # Simulation time step
ctrl_dt: float = 0.02 # Control update frequency
# Episode parameters
max_episode_seconds: float = 20.0
reset_noise_scale: float = 0.01
```
### Recommended Directory Structure
```
motrix_envs/my_task/
├── __init__.py # Module initialization
├── cfg.py # Environment configuration
├── my_model.xml # Physics model file
└── my_env.py # Environment implementation
```
For complex models with many referenced files, it's recommended to use folder management.
## Common Configuration Issues
### File Path Issues
- When using relative paths, ensure paths are relative to the configuration file location
- Avoid using hardcoded absolute paths
- Check file permissions and accessibility
- Ensure all referenced sub-files exist
### Time Step Settings
- `ctrl_dt` should be an integer multiple of `sim_dt`
- `sim_dt` that is too small will affect simulation performance
- `ctrl_dt` that is too large will affect control precision
- Recommend `sim_dt` between 0.001-0.02 seconds
### Simulation Stability
- Avoid excessively large time steps
- Set contact parameters reasonably to avoid penetration
- Mass and inertia distribution should be reasonable
- Joint limits should match actual conditions
Through proper physics environment configuration, you can create accurate and efficient simulation environments for reinforcement learning training.

50
docs/source/en/user_guide/tutorial/rewards.md Normal file
View File

@@ -0,0 +1,50 @@
# Reward Function Design
The reward function tells the agent what behaviors are desired and is a core part of reinforcement learning environment design.
## Position of Reward Function in Training Loop
In MotrixLab's NpEnv, reward calculation occurs in the `update_state` phase of the `step` function:
```python
# Execution flow of NpEnv.step()
def step(self, actions: np.ndarray) -> NpEnvState:
# 1. Preparation phase: Clear rewards and state
self._prev_physics_step() # reward = 0.0, terminated = False, truncated = False
# 2. Apply actions
self._state = self.apply_action(actions, self._state)
# 3. Physics simulation
self.physics_step() # Execute physics simulation
# 4. Update state ← Reward function is calculated here
self._state = self.update_state(self._state) # Calculate rewards and observations
# 5. Post-processing
self._update_truncate() # Check time truncation
self._reset_done_envs() # Reset completed environments
return self._state
```
You need to implement reward calculation logic in the `update_state` method of subclasses. For specific reward function design ideas, please refer to the training examples.
### Reward Component Design Principles
1. **Separation of Concerns**: Each reward function should handle a specific goal
2. **Weight Configuration**: Manage weights of different components through configuration files
3. **Normalization**: Keep reward values within reasonable ranges
4. **Smoothness**: Avoid hard thresholds, use exponential functions for smooth transitions
This approach makes reward functions modular, facilitating debugging and adjustment of individual component weights.
## Design Principles
1. **Clear Goal Orientation**: Reward functions should directly reflect task goals
2. **Reasonable Reward Range**: Avoid overly large or small reward values to maintain training stability
3. **Balance Exploration and Exploitation**: Appropriately reward behaviors close to goals, avoiding sparse rewards
4. **Avoid Reward Hacking**: Check if agents can obtain high rewards through unintended means
5. **Debug-Friendly**: Output reward decomposition information during development for optimization
By correctly implementing reward calculation in the `update_state` method, you can design effective learning signals for various robot tasks.

92
docs/source/en/user_guide/tutorial/training_and_result.md Normal file
View File

@@ -0,0 +1,92 @@
# Training Execution and Result Analysis
This section introduces how to execute reinforcement learning training and how to analyze and use training results.
## Start Training
### Basic Training Commands
```bash
# Train with default parameters
uv run scripts/train.py --env cartpole
# Specify simulation backend
uv run scripts/train.py --env cartpole --sim-backend np
# Specify training backend
uv run scripts/train.py --env cartpole --train-backend jax
uv run scripts/train.py --env cartpole --train-backend torch
```
### Advanced Training Configuration
```bash
# Customize training parameters
uv run scripts/train.py --env cartpole \
--num-envs 1024 \
--train-backend jax \
--sim-backend np
# Enable rendering to monitor training process
uv run scripts/train.py --env cartpole --render
```
### Different Backend Configuration Differences
The system supports configuring different reinforcement learning parameters for different training backends (JAX/Torch). For example:
- **dm-walker environment**:
- JAX backend: `mini_batches: 4`
- Torch backend: `mini_batches: 32`
- **dm-runner environment**:
- JAX backend: `learning_epochs: 4`
- Torch backend: `learning_epochs: 2`
These differences are implemented through the `@rlcfg(env_name, backend="jax/torch")` decorator in configuration classes. The system automatically applies the corresponding configuration based on the selected training backend.
### Supported Command Line Parameters
| Parameter | Description | Default Value |
| ----------------- | ------------------------------- | ------------- |
| `--env` | Environment name | `cartpole` |
| `--sim-backend` | Simulation backend (np) | Auto select |
| `--train-backend` | Training backend (jax/torch) | Auto select |
| `--num-envs` | Number of parallel environments | 2048 |
| `--render` | Enable rendering | False |
> **Note**: Other parameters such as learning rate, network structure, etc., can be set in configuration files. Some environments support configuring different parameters for different training backends.
## Training Process Monitoring
### TensorBoard Monitoring
Start TensorBoard to view training progress:
```bash
uv run tensorboard --logdir runs/{env-name}
```
For example:
```bash
uv run tensorboard --logdir runs/cartpole
```
## Model Evaluation and Testing
### Using Trained Policies
```bash
# Automatically find best policy for testing (recommended)
uv run scripts/play.py --env cartpole
# Manually specify policy file for testing
uv run scripts/play.py --env cartpole --policy runs/cartpole/nn/best_policy.pickle
# Specify number of test environments
uv run scripts/play.py --env cartpole --num-envs 100
```
> **Note**: The system will automatically find the latest and best policy files in the `runs/cartpole/` directory for testing.

231
docs/source/en/user_guide/tutorial/training_environment_config.md Normal file
View File

@@ -0,0 +1,231 @@
# Training Environment Configuration
MotrixLab provides a flexible configuration system that allows users to customize reinforcement learning training parameters. This section introduces how to configure training environments and reinforcement learning algorithm parameters.
## RL Training Configuration (PPOCfg)
Training configuration defines parameters for reinforcement learning algorithms. MotrixLab now supports configuring different parameters for different training backends.
### Basic Training Configuration
```python
from dataclasses import dataclass
from motrix_rl.skrl.cfg import PPOCfg
from motrix_rl import registry
# Universal configuration (applies to all backends)
@registry.rlcfg("my-task")
@dataclass
class MyTaskRLCfg(PPOCfg):
# Environment parameters
num_envs: int = 2048 # Number of parallel environments during training
play_num_envs: int = 16 # Number of parallel environments during evaluation
# PPO algorithm parameters
learning_rate: float = 3e-4 # Learning rate
rollouts: int = 32 # Experience replay rounds
learning_epochs: int = 10 # Number of epochs per update
mini_batches: int = 32 # Number of mini-batches
discount_factor: float = 0.99 # Discount factor
grad_norm_clip: float = 1.0 # Gradient clipping
# Network structure parameters
policy_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64) # Policy network hidden layers
value_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64) # Value network hidden layers
# Training control parameters
max_env_steps: int = 1_000_000 # Maximum environment steps
check_point_interval: int = 10_000 # Checkpoint interval
```
### Different Backend Configurations
```python
# JAX backend specific configuration
@registry.rlcfg("my-task", backend="jax")
@dataclass
class MyTaskJAXCfg(PPOCfg):
# JAX optimized configuration
learning_rate: float = 2e-4 # JAX backend uses smaller learning rate
mini_batches: int = 4 # JAX supports large batches, fewer mini-batches
learning_epochs: int = 4 # Training rounds
num_envs: int = 2048 # More parallel environments
# Torch backend specific configuration
@registry.rlcfg("my-task", backend="torch")
@dataclass
class MyTaskTorchCfg(PPOCfg):
# Torch optimized configuration
learning_rate: float = 1e-4 # Torch backend uses even smaller learning rate
mini_batches: int = 32 # Torch needs more mini-batches
learning_epochs: int = 2 # Fewer training rounds
num_envs: int = 1024 # Fewer parallel environments
```
### Complete Configuration Example
```python
@dataclass
class CompletePPOConfig(PPOCfg):
"""
Complete reinforcement learning training configuration example
Contains all configuration parameters from basic to advanced
"""
# ===== Basic Training Parameters =====
seed: Optional[int] = None # Random seed
num_envs: int = 2048 # Number of parallel environments during training
play_num_envs: int = 16 # Number of parallel environments during evaluation
max_env_steps: int = 2_048_000 # Maximum training steps
check_point_interval: int = 1000 # Checkpoint save interval
# ===== PPO Algorithm Core Parameters =====
learning_rate: float = 3e-4 # Learning rate
rollouts: int = 32 # Experience replay rounds
learning_epochs: int = 2 # Number of training rounds per update
mini_batches: int = 32 # Number of mini-batches
discount_factor: float = 0.99 # Discount factor
lambda_param: float = 0.95 # GAE parameter
grad_norm_clip: float = 1.0 # Gradient clipping
# ===== PPO Clipping Parameters =====
ratio_clip: float = 0.2 # PPO clipping ratio
value_clip: float = 0.2 # Value clipping
clip_predicted_values: bool = True # Clip predicted values
# ===== Loss Function Parameters =====
entropy_loss_scale: float = 0.0 # Entropy loss coefficient
value_loss_scale: float = 2.0 # Value loss coefficient
kl_threshold: float = 0 # KL divergence threshold
# ===== Learning Rate Scheduler =====
learning_rate_scheduler_kl_threshold: float = 0.008 # Adaptive learning rate KL threshold
# ===== Network Architecture Configuration =====
# Small network (suitable for simple tasks like CartPole)
# policy_hidden_layer_sizes: tuple[int, ...] = (128, 64)
# value_hidden_layer_sizes: tuple[int, ...] = (128, 64)
# Medium network (default configuration, suitable for most tasks)
policy_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64)
value_hidden_layer_sizes: tuple[int, ...] = (256, 128, 64)
# Large network (suitable for complex tasks like robot control)
# policy_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
# value_hidden_layer_sizes: tuple[int, ...] = (512, 256, 128)
# ===== Network Sharing Configuration =====
share_policy_value_features: bool = True # Policy and value networks share feature extraction layers
# ===== Training Control Parameters =====
random_timesteps: int = 0 # Random timesteps
learning_starts: int = 0 # Timesteps to start learning
time_limit_bootstrap: bool = True # Time limit bootstrap
# ===== Reward Shaping =====
rewards_shaper_scale: float = 1.0 # Reward scaling factor
```
## Configuration Usage Methods
### 1. Default Configuration Usage
```bash
# Use configuration given in code
uv run scripts/train.py --env my-task
# Specify training backend, system will automatically select corresponding backend configuration
uv run scripts/train.py --env my-task --train-backend jax
uv run scripts/train.py --env my-task --train-backend torch
```
### 2. Command Line Parameter Override
```bash
# Override supported command line parameters
uv run scripts/train.py --env my-task \
--num-envs 1024 \
--train-backend jax \
--sim-backend np
# System will automatically select JAX backend configuration
```
### 3. Configuration Priority
System selects configuration in the following priority:
1. **Backend-specific configuration**: If there is a configuration decorated with `@rlcfg(env_name, backend="jax/torch")`
2. **Universal configuration**: If there is a configuration decorated with `@rlcfg(env_name)` (no backend parameter)
3. **Configuration override**: Command line `cfg_override` parameter
For example:
```python
# Highest priority - backend specific configuration
@rlcfg("my-task", backend="jax")
@dataclass
class MyTaskJAXCfg(PPOCfg):
mini_batches: int = 4
# Second priority - universal configuration
@rlcfg("my-task")
@dataclass
class MyTaskRLCfg(PPOCfg):
mini_batches: int = 32
# When using --train-backend jax, system will select MyTaskJAXCfg
# When using --train-backend torch, system will select MyTaskRLCfg
```
#### User Configurable Parameters
| MotrixLab Configuration Class | SKRL Framework Parameter | Description |
| -------------------------------------- | --------------------------------------------- | ----------------------------------- |
| `learning_rate` | `learning_rate` | Learning rate |
| `rollouts` | `rollouts` | Experience replay rounds |
| `learning_epochs` | `learning_epochs` | Training rounds |
| `mini_batches` | `mini_batches` | Number of mini-batches |
| `discount_factor` | `discount_factor` | Discount factor |
| `grad_norm_clip` | `grad_norm_clip` | Gradient clipping |
| `lambda_param` | `lambda` | GAE parameter |
| `ratio_clip` | `ratio_clip` | PPO clipping ratio |
| `value_clip` | `value_clip` | Value clipping |
| `clip_predicted_values` | `clip_predicted_values` | Clip predicted values |
| `entropy_loss_scale` | `entropy_loss_scale` | Entropy loss coefficient |
| `value_loss_scale` | `value_loss_scale` | Value loss coefficient |
| `kl_threshold` | `kl_threshold` | KL divergence threshold |
| `random_timesteps` | `random_timesteps` | Random timesteps |
| `learning_starts` | `learning_starts` | Learning start timesteps |
| `time_limit_bootstrap` | `time_limit_bootstrap` | Time limit bootstrap |
| `learning_rate_scheduler_kl_threshold` | `learning_rate_scheduler_kwargs.kl_threshold` | Adaptive learning rate KL threshold |
| `check_point_interval` | `experiment.write_interval` | Log write interval |
| `check_point_interval` | `experiment.checkpoint_interval` | Checkpoint save interval |
| `rewards_shaper_scale` | `rewards_shaper` | Reward scaling function |
#### Preprocessor Parameters
| SKRL Framework Parameter | Type | Description |
| ------------------------ | --------------------- | ------------------- |
| `state_preprocessor` | RunningStandardScaler | State normalization |
| `value_preprocessor` | RunningStandardScaler | Value normalization |
### Configuration Hierarchy Summary
```
User Configuration Class (PPOCfg)
↓ Backend specific selection
Backend Configuration (JAX/Torch)
↓ Parameter mapping
SKRL Framework Configuration Dictionary
↓ Pass to
PPO Agent
↓ Execute
Reinforcement Learning Training
```
This design allows users to:
1. Control complex training parameters through simple configuration classes
2. Configure different parameters for different training backends to achieve optimal performance
3. Maintain full compatibility with the SKRL framework