chore: release v0.0.1

This commit is contained in:
motphys-developers
2025-11-20 08:57:48 +00:00
commit 5133830b5a
105 changed files with 8789 additions and 0 deletions

View File

@@ -0,0 +1,50 @@
# Reward Function Design
The reward function tells the agent what behaviors are desired and is a core part of reinforcement learning environment design.
## Position of Reward Function in Training Loop
In MotrixLab's NpEnv, reward calculation occurs in the `update_state` phase of the `step` function:
```python
# Execution flow of NpEnv.step()
def step(self, actions: np.ndarray) -> NpEnvState:
# 1. Preparation phase: Clear rewards and state
self._prev_physics_step() # reward = 0.0, terminated = False, truncated = False
# 2. Apply actions
self._state = self.apply_action(actions, self._state)
# 3. Physics simulation
self.physics_step() # Execute physics simulation
# 4. Update state ← Reward function is calculated here
self._state = self.update_state(self._state) # Calculate rewards and observations
# 5. Post-processing
self._update_truncate() # Check time truncation
self._reset_done_envs() # Reset completed environments
return self._state
```
You need to implement reward calculation logic in the `update_state` method of subclasses. For specific reward function design ideas, please refer to the training examples.
### Reward Component Design Principles
1. **Separation of Concerns**: Each reward function should handle a specific goal
2. **Weight Configuration**: Manage weights of different components through configuration files
3. **Normalization**: Keep reward values within reasonable ranges
4. **Smoothness**: Avoid hard thresholds, use exponential functions for smooth transitions
This approach makes reward functions modular, facilitating debugging and adjustment of individual component weights.
## Design Principles
1. **Clear Goal Orientation**: Reward functions should directly reflect task goals
2. **Reasonable Reward Range**: Avoid overly large or small reward values to maintain training stability
3. **Balance Exploration and Exploitation**: Appropriately reward behaviors close to goals, avoiding sparse rewards
4. **Avoid Reward Hacking**: Check if agents can obtain high rewards through unintended means
5. **Debug-Friendly**: Output reward decomposition information during development for optimization
By correctly implementing reward calculation in the `update_state` method, you can design effective learning signals for various robot tasks.