SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
2026-07-16 · Verified
New to SEED? Start here
Understand the mechanism without a GPU first, then run a reduced method check. Move to strict reproduction only when the model, data, environments, hyperparameters, and 8×A800 80GB setup are aligned.
Complete the mechanism track first. A method smoke test is still useful without multi-GPU resources, but its score is not directly comparable with the paper's main tables.
Understand the mechanism
Explain the supervision gap SEED addresses, how its two stages connect, and why it is not an external-memory method.
Read the problem before the equations
Long-horizon agents often receive sparse reward only after a trajectory ends, so the reward does not identify which intermediate observations, actions, or tool calls deserve credit.
Done when: You can distinguish a trajectory-level outcome from a token-level learning signal.
Abstract and introduction ↗Sketch Stage 1
A normal policy collects completed trajectories. An external analyzer writes reusable or failure-avoidance hindsight skills, and the trajectory–skill pairs train the policy with SFT.
Done when: Your diagram includes trajectories, the external analyzer, skill annotations, and the SFT checkpoint.
Paper §3.2 ↗Sketch Stage 2
The latest frozen policy performs rollouts and trajectory analysis. The same sampled actions are re-scored in ordinary and skill-augmented contexts; their probability shift forms the OPD signal optimized jointly with GRPO.
Done when: You can identify the actor, analyzer, old policy, trainable policy, and OPD loss.
Paper §3.3 ↗Confirm the inference boundary
Hindsight skills, the analyzer, and skill prompts are training-time tools. Deployment runs only the policy that internalized the behavioral change.
Done when: You would not describe SEED as retrieving a skill bank at inference time.
Official code overview ↗
Method smoke test
Validate the trajectory → hindsight skill → token-level OPD → policy-update chain without claiming the paper's main-table scores.
Choose one environment
Start with ALFWorld: the official repository provides direct setup commands and the paper reports detailed training dynamics. Pin the split, reward, maximum interaction steps, and metric.
Done when: Your manifest contains one environment and an explicit evaluation protocol.
Official setup guide ↗Run the environment and vanilla policy first
Install veRL and ALFWorld; pin the repository commit, Python/CUDA, and dependencies. Save one vanilla rollout and inspect the observation, action, reward, and terminal outcome fields.
Done when: One trajectory can be rerun and produces a structurally complete record.
Official code installation ↗Validate the Stage 1 data chain
Generate trajectories and hindsight skills from a small task subset. Check that successful trajectories yield reusable workflows and failed trajectories yield avoidance guidance. Reduced data validates mechanism only.
Done when: You can trace one SFT sample from its raw trajectory to the skill target.
Stage 1 scripts ↗Check Stage 2 against GRPO
Log environment return, OPD loss, episode length, and valid-token masks for both vanilla GRPO and SEED. Confirm that OPD produces a non-zero, stable signal before discussing performance.
Done when: You can show the score shift for the same sampled actions under ordinary and skill contexts.
Paper §3.3 and training scripts ↗
Strict reproduction
Align the paper version, three backbones, data scale, baselines, hyperparameters, and 8×A800 80GB compute to recheck the main tables, training dynamics, and generalization claims.
Freeze paper and code revisions
Record arXiv v1, the code commit, every Qwen checkpoint revision, tokenizer, chat template, CUDA version, and dependency lock state.
Done when: Another researcher can reconstruct the same software and model revisions from your manifest.
Paper v1 and official code ↗Align data and rollout counts
For each benchmark, SFT uses 180 tasks with 8 rollouts per task. RL uses 2,400 ALFWorld, 2,400 WebShop, and 19,200 Search instances.
Done when: The data manifest's splits, counts, and preprocessing outputs match Table 3.
Paper Table 3 ↗Lock hyperparameters and compute
Use 150 policy updates, rollout group 8, learning rate 1e-6, OPD coefficient 0.01, and KL coefficient 0.01; also record the paper's batches, prompt/response lengths, and environment-specific interaction limits.
Done when: A config diff against Table 5 has no unexplained differences.
Paper Table 5 ↗Recheck scores and training trajectories
Reproduce Vanilla/GRPO before SEED. Compare not only final scores but also training curves, episode length, sample efficiency, unseen splits, and ablations.
Done when: Your report separates paper-reported results, independent run results, and conditions that could not be aligned.
Paper §4.2–4.6 ↗
Paper info
Start research from this paper
Choose a research mode to carry this paper into the workspace context.
Reproducibility status
No structured reproducibility record has been added yet.
Model-role topology
This view only shows recorded model-role relations; it does not invent workflow edges.
- artifactCompleted on-policy trajectoryanalyze after completion
- moduleHindsight-skill analyzerextract reusable guidance
- artifactNatural-language hindsight skillre-score sampled tokens
- moduleOn-policy distillation + GRPOjoint dense OPD + outcome RL
- moduleUpdated shared policy
Nodes and edges come from the paper record; unrecorded relations are not filled in.
Model roles
Qwen2.5-3B-Instruct
Actor · weights updated
Qwen2.5-7B-Instruct
Actor · weights updated
Qwen3-1.7B
Actor · weights updated
GLM-5.2
Analyzer · weights not updated