This post turns a small open-weight model, Qwen3.5-4B, into an agent for ALFWorld, a text-based household benchmark where an agent types commands like go to fridge 1 or heat apple 1 with microwave 1 to finish tasks such as "put a hot apple in the fridge". The goal was to learn post-training end to end (imitation, RL, quantization, evaluation) and to see how a 4B model compares with a hosted frontier model when cost and latency matter.

TL;DR

• Imitation does most of the work. Fine-tuning on 33 expert games lifts the model from 3.7% to 88% success on unseen games; 340 games reach 97%.
• RL on top helps most when demos are scarce. GRPO from the 33-demo model reaches 95.3% and solves games in a third fewer steps.
• GRPO was the best RL method despite being unstable. Averaged over three training runs each, GRPO reached 95.3% vs. 92.3% for RLOO and 91.5% for filtered behavior cloning, even though every GRPO run had a sudden instability.
• 4-bit quantization is free: same success, ~1.5x faster per step. The agent also runs on my M1 laptop.
• It matches or beats the cheapest useful Gemini (3.5 Flash-Lite) at ~45-80x lower cost per episode, with no measurable loss of general ability (IFEval, GSM8K, MMLU-Pro).

Setup in brief

• Task: ALFWorld's 6 household task types, 50-step limit, reward only for finishing. Results are success on the 134 unseen-room games (valid_unseen), 3 sampling seeds, mean ± standard deviation.
• Harder than most papers: the agent types free-text commands and is never shown the list of valid actions. Most published ALFWorld RL results include that list, which turns each step into multiple choice, so my numbers aren't directly comparable to theirs.
• Model and compute: Qwen3.5-4B (thinking off), LoRA fine-tuning, vLLM serving, rented A40 GPUs at $0.59/h.

Baselines

policy	success on unseen games
random valid action	0.2%
Qwen3.5-4B, zero-shot	3.7%
Qwen3.5-4B, 2 example games in the prompt	57.7%
Reference: Gemini 3.5 Flash-Lite, zero-shot	80.8%
Reference: Gemini 3.5 Flash-Lite, 2 example games in the prompt	94.8%

Zero-shot, the 4B model doesn't know ALFWorld's conventions: it tries to take objects from across the room and never figures out from "Nothing happens." that it must go to a place first. Even Gemini fails most heat tasks zero-shot for the same kind of reason. Two example games fix much of it for both. This was the theme of the whole project: what's missing is knowledge of this environment's conventions, not reasoning ability.

Results

1. Imitation (SFT). LoRA fine-tuning on the shortest successful game per training task, recorded from ALFWorld's scripted expert:

expert demos	success	steps per solved game
33 (1%)	88.1 ± 2.0	18.7
340 (10%)	97.3 ± 0.4	17.8
3,405 (100%)	98.0 ± 0.4	17.1

Ten times more demos after 340 buys only 0.7 points.

2. RL from the 33-demo model. Three methods with the same budget, each run three times with different training seeds. Reward is 1 for a win minus 0.01 per step:

• GRPO: sample 8 attempts per game; each attempt's advantage is its reward minus the group mean, divided by the group's standard deviation.
• RLOO: the same, but the advantage is the reward minus the mean of the other 7 attempts, with no division.
• Filtered behavior cloning (BC): imitate the attempts that won. The baseline to beat.

from SFT 1% (88.1%, 18.7 steps)	success, run 1 / 2 / 3	mean of 3 runs	steps per solved game (mean)
GRPO	93.5 / 95.3 / 97.0	95.3	12.4
RLOO	91.5 / 93.3 / 92.0	92.3	14.3
filtered BC	92.3 / 92.5 / 89.6	91.5	19.3

GRPO is best on both success and episode length, and every GRPO run beats its starting model. Filtered BC never shortens episodes, since it only copies wins and never sees the step cost. From the 340-demo model, GRPO reached 99.0% (one run) with 3 fewer steps per game.

3. Fast and cheap. 4-bit quantization (GPTQ, calibrated on ALFWorld games) keeps success and cuts the time per step for a single agent from 0.21 s to 0.14 s on an A40. Running 64 episodes in parallel, an A40 completes 5-6k episodes per hour, about $0.0001 per episode. On my 16 GB M1 laptop, the 4-bit model still scores 98.0% at ~0.9 s per step.

4. Against Gemini. The 33-demo GRPO model ties Gemini 3.5 Flash-Lite with two examples in its prompt (13 games each way) and clearly beats it zero-shot. The 340- and 3,405-demo models beat both. Gemini costs $5-8 per 1,000 episodes against ~$0.10 for the 4B agent. Gemini remains the far better generalist (83.8 vs. 68.8 on MMLU-Pro).

5. No forgetting. On IFEval, GSM8K and MMLU-Pro, compared question by question against a rerun of the base model, no fine-tuned checkpoint drops significantly. GSM8K even rises ~2 points.

What I learned

• GRPO was unstable in every run, but still the best. Each GRPO run had a sudden jump in KL divergence from the starting model (up to 4.5) or a drop in validation success (down to 80%) after ~15 iterations; RLOO never did. Keeping the best checkpoint, evaluated every 10 iterations, rescued it.

• Shorter episodes are the real cost lever. 4-bit makes one agent faster but doesn't raise batch throughput; RL's ~6 fewer steps per game does.
• Infrastructure bugs were bigger than modeling questions. vLLM silently ignored LoRA adapters for this model, the chat template broke the match between training and inference, and ALFWorld's own expert had a years-old bug. Details in the appendix.

Appendix

A. Setup details

• Environment: ALFWorld text version (TextWorld), 3,553 training games, 140 valid_seen games (for checkpoint selection), 134 valid_unseen games (for reported results). Six task types: pick and place, look at an object under a lamp, clean, heat, cool, and pick two objects.
• Model choice: Qwen3.5-4B over Qwen3-4B-2507, which scored 28% vs. Qwen3.5's 45% with two example games on 20 validation games.
• Sampling: temperature 0.7, top-p 0.8, top-k 20 for Qwen; Gemini used its own recommended settings (temperature 1.0, top-p 0.95).
• Comparisons: paired by game and seed. A game counts for whichever model won it on more of its 3 seeds, then an exact McNemar test on those games.

B. Imitation details

• Data: the scripted expert played every training game 3 times; I kept the shortest win per game. Nested subsets of 33, 340 and 3,405 games, stratified by task type.
• Training: LoRA rank 32 on all linear layers of the language model, learning rate 1e-4 with cosine decay, 3 epochs, loss only on the action tokens. The 100% run took 2.25 h on one A40.
• The student beats its teacher. The expert solves ~94% of games, but the 340-demo model reaches 97% because it imitates only the expert's shortest wins.
• Fixing the expert. ALFWorld's scripted expert solved only 29% of pick-two tasks. Its state tracker looks for the command put, but the game renamed it to move X to Y years ago, so after its first placement the expert thinks it's still holding the object. A one-line patch took pick-two from 29% to 88%.
• Fixing the prompt format. Qwen3.5's chat template writes an empty <think></think> block on the turn being generated but strips it from earlier turns, so a whole training game matches the inference prompt only at its last step. Training one step at a time would cost 13x more tokens. Instead I keep the empty block on every turn, during training and inference alike. Every inference prompt is then an exact prefix of the training sequence, which RL also needs: one forward pass scores a whole game exactly as it was played.

C. RL details

• Loop: each iteration plays 8 training games 8 times each with vLLM, takes one LoRA update with a small KL penalty (0.01) toward the SFT model, merges the weights and pushes them into the running vLLM server through CUDA IPC (~2 s). ~2.3 minutes per iteration; a 30-iteration run costs ~$0.70.
• Checkpoint selection: one pass over valid_seen every 10 iterations; the best is evaluated on valid_unseen.
• Per-run results, from SFT 1%:

method	run 1 / 2 / 3	steps per solved game, run 1 / 2 / 3
GRPO	93.5 / 95.3 / 97.0	12.3 / 13.2 / 11.8
RLOO	91.5 / 93.3 / 92.0	12.4 / 17.3 / 13.1
filtered BC	92.3 / 92.5 / 89.6	19.7 / 20.6 / 17.6

• Significance: every GRPO run beats SFT 1% game by game (p = 0.001 to 0.035); RLOO and filtered BC do so in one of three runs each. Across runs, GRPO vs. RLOO p = 0.08 and GRPO vs. filtered BC p = 0.05 (Welch t-test, 3 runs each), so the ranking is suggestive rather than settled.
• Methodology note: my first comparison used one training run per method (93.5 / 91.5 / 92.3) and concluded the methods were tied. Run-to-run variation (GRPO alone ranged from 93.5 to 97.0) is as large as the gap between methods, so one run per method was simply too few; the extra runs fixed that.
• Why GRPO is unstable here: once the policy is good, most groups of 8 attempts all win, so the only remaining reward difference is the tiny step cost. Dividing by the group's standard deviation inflates those 0.01-sized differences into full-size advantages. Dr. GRPO, a published GRPO variant that removes this division, is the obvious next experiment.
• From SFT 10%: GRPO 99.0 ± 0.4 and RLOO 97.8 ± 0.7 (one run each), both ~3 fewer steps per game. Their success gain over SFT 10% (97.3) isn't significant: near the ceiling only a few games are left to win.
• Rejected settings: learning rate 5e-4 collapsed the policy in one step; 1e-4 with smaller batches dropped success from 88% to 50% within 3 steps.

D. Speed and cost details

	success	seconds per step, 1 agent	episodes per A40-hour	$ per 1,000 episodes
SFT 100%, 16-bit, A40	98.0	0.21	5,310	0.11
SFT 100%, 4-bit, A40	98.3	0.14	5,020	0.12
SFT 1% + GRPO, 4-bit, A40	93.8	0.14	5,980	0.10
SFT 100%, 4-bit, M1 laptop	98.0	0.86	—	0
SFT 1% + GRPO, 4-bit, M1 laptop	91.3	0.88	—	0
Gemini 3.5 Flash-Lite, zero-shot	80.8	~0.5-1.2 (hosted)	—	5.31
Gemini 3.5 Flash-Lite, 2 examples	94.8	~0.5 (hosted)	—	8.08

• 4-bit: GPTQ with llm-compressor, calibrated on 256 training games, 8.5 GB → 3.6 GB; generation speed rises from 58 to 139 tokens/s. At 64 parallel episodes the GPU is busy processing prompts, which 4-bit doesn't speed up, so throughput is unchanged.
• Laptop: MLX 4-bit without calibration. SFT 100% loses nothing; the GRPO model loses ~2.5 points (not significant, but consistent with RL's small weight changes being rounded away).
• After 4-bit, the environment is the bottleneck: with one agent, ALFWorld's own step takes longer than the model's 0.14 s.

E. Gemini comparison details

• Model choice: Gemini 2.5 Flash-Lite is closed to new API users, and 3.1 Flash-Lite scored lower than 3.5 Flash-Lite in a 20-game test while using more tokens per game.
• Same context: same system prompt, example games, observations, history and rules as the 4B model, checked character by character on a logged game. Only the message format differs.
• Failure mode: zero-shot, Gemini solves only 14.5% of heat tasks. It tries use microwave 1, declares task complete, or heats from the wrong spot.
• Paired results: 33-demo GRPO vs. Gemini zero-shot 31 games to 12 (p = 0.005); vs. Gemini with examples 13 to 13. 340- and 3,405-demo SFT vs. Gemini with examples: 15 to 5 (p = 0.04) and 15 to 3 (p = 0.008).

F. General ability (forgetting check)

checkpoint	IFEval	GSM8K	MMLU-Pro
Qwen3.5-4B base (rerun)	81.3	82.2	68.8
SFT 1% / 10% / 100%	83.4 / 80.6 / 82.4	84.3 / 84.6 / 83.9	68.6 / 68.0 / 67.7
SFT 1% + GRPO	81.5	84.0	68.7
SFT 10% + RLOO	82.1	84.2	67.5
Gemini 3.5 Flash-Lite	87.2	85.0	83.8

Greedy decoding, MMLU-Pro limited to 150 questions per subject and 2,048 output tokens. That cap cuts off ~9% of the 4B model's MMLU-Pro answers, so vendor-reported scores (which use longer outputs and thinking mode) aren't comparable. None of the fine-tuned models ever answered an IFEval prompt with an ALFWorld command.

G. Gotchas

• vLLM's runtime LoRA silently does nothing for Qwen3.5, a hybrid of Gated DeltaNet and attention layers. Adapters load without error and outputs match the base model; there's an open upstream issue. Workaround: merge adapters into the weights, and in RL push the merged weights after every update.
• TextWorld leaks disk space (a temporary library copy per game) and memory (an unbounded global cache) over thousands of games.
• Prefix caching on a hybrid model reused only ~69% of each prompt, though every step just appends ~100 tokens.
• "OpenAI-compatible" servers disagree on details: Gemini rejects any seed and an empty stop list; mlx-lm's server rejects "logprobs": null. Send optional fields only when they're set.
• Checkpoint selection is noisy: twice, the checkpoint with the best valid_seen score wasn't the best on valid_unseen.
• Per-iteration RL metrics mostly reflect which games came up: spikes in invalid actions I first blamed on GRPO appeared in every method on the same hard games.
• A wait loop cost me two results: pgrep -f over ssh matched its own command line, the script waited forever, and the GPU pod's self-destruct timer deleted the pod with the results still on its disk. Now every job writes a done marker, and results are copied off the pod every 5 minutes.
• A pod with an older CUDA driver had no usable GPU for CUDA 13 builds of PyTorch.
• An evaluation library printed my API key: lm-eval's HTTP client logs failed requests with their full headers, including Authorization: Bearer .... I rotated the key and now run such tools behind a log filter that redacts secrets.

H. How I ran it, and what it cost

Claude Code did the hands-on work: it wrote the training and evaluation code, rented and terminated GPU pods, ran the experiments and drafted this write-up, while I made the decisions. Every pod ran a self-terminating watchdog so nothing could bill overnight, and API keys lived in files the agent's tools couldn't read. Total spend: about $15 of GPU time and $11 of Gemini API.