QWEN3-8B · KLEAR-AGENTFORGE-8B-SFT · PPO

面向代码智能体的
PPO训练

PPO Training for Code Agents

基于 Qwen3-8B 的 Klear-AgentForge-8B-SFT 模型，使用改造的 PPO 算法进行 RL 训练，在 SWE-bench 验证指标从 39.8% 提高到 42.8%，提升 3.0 个百分点。

Starting from the Qwen3-8B-based Klear-AgentForge-8B-SFT, our adapted PPO training raises the SWE-bench result from 39.8% to 42.8%, a gain of 3.0 percentage points.

获取样本与模型 / Get assets查看评测 / Results

SWE-BENCH VERIFIED · 500 题 / TASKS

PPO 训练后的代码任务效果
Code-task results after PPO training

39.8%复现 SFT 模型 / Reproduced SFT model: 199/500
42.8%最终 RL 模型 / Final RL model: 214/500

整体架构 / ARCHITECTURE

训练、价值模型与任务基础设施
Training, value model, and task infrastructure

01 / RL 训练 / TRAINING

PPO 训练模块 / PPO training

生成包含过程奖励与终局奖励的样本 → 两轮 Critic 更新 → 重算 GAE 并更新 Actor

Generate samples with process and terminal rewards → update the Critic twice → recompute GAE and update the Actor.

冻结训练池为 529 题，主要选取 pass@4 为 1/4、2/4、3/4 的任务。每步抽取 32 题生成当前策略轨迹；只有终局结果、过程标签、轨迹与 token 对齐均有效的样本参与更新。

The frozen 529-task pool mostly contains tasks with pass@4 of 1/4, 2/4, or 3/4. Each step samples 32 tasks for fresh trajectories. Only samples with valid terminal outcomes, process labels, trajectories, and token alignment enter training.

02 / 价值模型 / VALUE MODEL

Critic Pretrain 模块 / Critic pretraining

多种子离线 rollout 与奖励标注 → 多个 epoch 更新 Critic → 用隔离样本选定 checkpoint

Generate and label multi-seed offline rollouts → train the Critic for multiple epochs → select a checkpoint using held-out samples.

过程奖励和终局奖励共同构成训练目标；仅保留标签完整且满足上下文限制的轨迹。train/val/test 约 5,400/300/300 条。最终 Critic 的 selection MSE 约 0.23、EV 约 0.16；在线 PPO 的 Critic EV 约 0.15。

Process and terminal rewards form the training target. Complete labels and context-length compliance are required. Train/validation/test contain about 5,400/300/300 trajectories. The selected Critic has selection MSE ≈ 0.23 and EV ≈ 0.16; its online PPO EV is about 0.15.

03 / 任务基础设施 / INFRASTRUCTURE

沙箱与数据链路 / Sandbox and data

SWE-Smith 原始任务 → 多级验证与仓库隔离 → Harbor 独立 Pod 执行

SWE-Smith source tasks → validation and repository separation → isolated Harbor Pod per task.

从上万道 SWE-Smith 任务中，经结构、补丁、强模型、NOP/Oracle、题面与测试对齐等检查，筛选出 1,800 余道候选任务。Harbor 为每题隔离执行环境；BuildKit 与 TCR 提供任务镜像，Gateway 连接模型服务，并检查网络、依赖和产物下载。

More than 1,800 candidates were selected from over ten thousand SWE-Smith tasks through structure, patch, strong-model, NOP/Oracle, and problem/test-alignment checks. Harbor isolates task execution; BuildKit and TCR provide images, Gateway connects model serving, and network, dependencies, and artifact downloads are checked.

HUGGING FACE 发布产物 / RELEASES

样本 · RL模型 · Critic模型
Tasks · RL model · Critic model

DATASET

训练样本 / Task dataset

1,761 道 Critic 任务及 529 道 PPO 训练任务清单。
1,761 Critic tasks and a 529-task PPO subset.

 MODEL

RL 模型 / RL model

最终选定的完整推理权重与 tokenizer。
Selected merged inference weights and tokenizer.

 MODEL

Critic 模型 / Critic model

用于 PPO 初始化的价值模型权重。
Value-model weights for PPO initialization.

支持该项目 / SUPPORT THIS PROJECT

支持该项目
Support This Project

数据、开发、GPU和CPU等训练的费用均由我自行承担。如果您觉得这个模型对您有用，并希望资助更大的数据集和未来的开源版本，可以给我点赞，留言。

I personally cover the costs of data, development, GPU and CPU training. If you find this model useful and would like to support larger datasets and future open-source releases, please give it a like and leave a comment.