137 lines
3.4 KiB
Markdown
137 lines
3.4 KiB
Markdown
# Pendulum
|
||
|
||
Pendulum is a single-joint swing-up and balance task. The goal is to swing the pole up and keep it inverted using one motor torque.
|
||
|
||
```{video} /_static/videos/pendulum.mp4
|
||
:poster: _static/images/poster/pendulum.jpg
|
||
:nocontrols:
|
||
:autoplay:
|
||
:playsinline:
|
||
:muted:
|
||
:loop:
|
||
:width: 100%
|
||
```
|
||
|
||
---
|
||
|
||
## Task Description
|
||
|
||
A single-link pendulum with one hinge joint is driven by a single motor (configurable gear). The motor’s torque rotates the rod in a plane, enabling swing-up from arbitrary initial angles, inverted balance, and maintenance. Torque is limited by the actuator ctrlrange; by modulating its magnitude and direction, the policy must accumulate energy to swing up and stabilize near the inverted position while damping angular-velocity-induced oscillations.
|
||
|
||
## Action Space
|
||
|
||
| Item | Details |
|
||
| ------------- | ------------------------------- |
|
||
| **Type** | `Box(-1.0, 1.0, (1,), float32)` |
|
||
| **Dimension** | 1 |
|
||
|
||
---
|
||
|
||
## Observation Space
|
||
|
||
| Item | Details |
|
||
| ------------- | ------------------------------- |
|
||
| **Type** | `Box(-inf, inf, (3,), float32)` |
|
||
| **Dimension** | 3 |
|
||
|
||
Order: `cos(theta), sin(theta), angular velocity`.
|
||
|
||
---
|
||
|
||
## Reward Function Design
|
||
|
||
- Upright reward: encourages angle near π (inverted)
|
||
- Energy shaping: target energy near inverted position
|
||
- Penalties: `ang_vel^2`, `ctrl^2`, `(ctrl - prev_ctrl)^2` to reduce oscillation and aggressive actuation
|
||
|
||
---
|
||
|
||
## Initial State
|
||
|
||
- Angle randomized in `[-pi, pi]`
|
||
- Angular velocity small random noise (if configured)
|
||
- Control history (`prev_ctrl`) reset to zero
|
||
|
||
## Episode Termination Conditions
|
||
|
||
- No fall/angle termination; only NaN check
|
||
- Episode length limited by `max_episode_seconds`
|
||
|
||
---
|
||
|
||
### 1. Environment Preview
|
||
|
||
```bash
|
||
uv run scripts/view.py --env pendulum
|
||
```
|
||
|
||
### 2. Start Training
|
||
|
||
```bash
|
||
# Train with default parameters
|
||
uv run scripts/train.py --env pendulum
|
||
|
||
# Customize parallel environments
|
||
uv run scripts/train.py --env pendulum --num-envs 1024
|
||
|
||
# Enable rendering during training
|
||
uv run scripts/train.py --env pendulum --render
|
||
```
|
||
|
||
### 3. View Training Progress
|
||
|
||
```bash
|
||
uv run tensorboard --logdir runs/pendulum
|
||
```
|
||
|
||
### 4. Test Training Results
|
||
|
||
```bash
|
||
# Auto-discover best policy (recommended)
|
||
uv run scripts/play.py --env pendulum
|
||
|
||
# Manually specify a policy file
|
||
uv run scripts/play.py --env pendulum --policy runs/pendulum/nn/best_policy.pickle
|
||
```
|
||
|
||
> **Tip**: Policies are auto-selected from `runs/pendulum/`. You can override with `--policy`.
|
||
|
||
---
|
||
|
||
## Configuration Parameters
|
||
|
||
### Environment Configuration
|
||
|
||
```python
|
||
@dataclass
|
||
class PendulumEnvCfg(EnvCfg):
|
||
model_file: str = ".../pendulum.xml" # MJCF model (gear=5)
|
||
max_episode_seconds: float = 20.0
|
||
sim_dt: float = 0.0125
|
||
ctrl_dt: float = 0.025
|
||
```
|
||
|
||
### Training Configuration (PPO example)
|
||
|
||
```python
|
||
@rlcfg("pendulum")
|
||
@dataclass
|
||
class PendulumPPO(PPOCfg):
|
||
seed: int = 42
|
||
max_env_steps: int = 10_000_000
|
||
num_envs: int = 1024
|
||
learning_rate: float = 3e-4
|
||
rollouts: int = 32
|
||
learning_epochs: int = 5
|
||
mini_batches: int = 4
|
||
policy_hidden_layer_sizes: tuple[int, ...] = (64, 64)
|
||
value_hidden_layer_sizes: tuple[int, ...] = (64, 64)
|
||
```
|
||
|
||
---
|
||
|
||
## Expected Training Results
|
||
|
||
1. Pendulum can swing up and stay near inverted
|
||
2. Oscillation around upright is reduced by angular-velocity and control-change penalties
|