All projects
Robustness of Implicit Q-Learning Under Distribution Shift
How much does an offline-RL policy degrade when the world changes after training, and does a bigger critic ensemble help?
2026 · SJSU research paper4-person team · I built the shift wrappers and evaluation pipeline, coordinated the project and reviewed the manuscript
JAX / Flax
IQL
MuJoCo
D4RL
Gym wrappers
SJSU HPC
1,536shift-level evaluations (3 envs × 8 configs × 4 shifts × 4 levels × 4 seeds)
14%lower gravity-shift degradation on Hopper with a triple-critic ensemble
~70%performance lost by every IQL variant at extreme gravity shifts
The problem
Offline RL trains policies from a fixed dataset, with no interaction. Deployments rarely match the data: a robot may face a wet floor, or a controller may get noisy sensors. Offline-RL algorithms are almost always benchmarked with identical train and test environments.
We asked how badly Implicit Q-Learning degrades under test-time-only shifts, and whether a simple architectural change (three Q-networks instead of two) makes it more robust.
How it works
- 1TrainReproduce IQL on D4RL medium-v2 (Hopper, HalfCheetah, Walker2d) with the double critic (2Q) and a triple critic (3Q), sweeping expectile τ ∈ {0.5, 0.7, 0.8, 0.9}.
- 2ShiftFreeze each policy and wrap the environment: gravity and friction scaling, Gaussian observation noise, and reward perturbation as a control, each at 4 severities.
- 3MeasureArea Under the Degradation Curve (AUDC): the mean relative return drop across severity levels, over 4 seeds × 10 episodes. Lower is more robust.
What I built
- Designed and implemented the four evaluation-time Gym wrappers (gravity, friction, observation noise, reward) that shift MuJoCo physics without retraining.
- Built the evaluation pipeline behind the 1,536-run grid on the SJSU HPC GPU partition.
- Coordinated the project across the team and reviewed the manuscript.
Results
| Shift | 2Q (baseline) | 3Q (ensemble) |
|---|---|---|
| Gravity | 0.616 ± 0.027 | 0.529 ± 0.069 |
| Friction | 0.713 ± 0.026 | 0.692 ± 0.011 |
| Obs. noise | 0.134 ± 0.020 | 0.126 ± 0.017 |
| Reward (control) | 0.002 ± 0.001 | 0.001 ± 0.000 |
Baselines reproduce published IQL scores (normalized ≈43 on HalfCheetah, ≈74 on Walker2d), confirming a correct implementation.
What I learned
- The ensemble helps, but it depends on the environment. 3Q improved robustness on every Hopper shift, showed no significant difference on HalfCheetah, and Walker2d often favored 2Q.
- Lower τ (more pessimistic value estimates) generally traded peak return for robustness, while higher τ inflated variance across seeds.
- Morphology explains the pattern. Hopper, a tall and unstable biped, is sensitive to any change in effective joint impedance, while HalfCheetah's planar gait is nearly immune to friction.
- The reward-perturbation control stayed near zero everywhere, as expected for a frozen offline policy, which validated the harness.
What I'd do next
- Larger ensembles and uncertainty-weighted critics (EDAC-style).
- Environment-aware selection of τ and critic count.
- Combining test-time shift evaluation with training-time domain randomization.