I Spent $1000 Making LLMs Play Competitive Pokémon

When I was a kid I loved Pokémon (I still do). If you showed me the number of hours I spent playing Pokémon Emerald as a child, I would probably be quite embarrassed. I also work with LLMs every day in my capacity as cofounder of an inference company. I thought it was a real shame that the LLMs that I worked with every day never had a day off and didn’t have any way to decompress at the end of a long day. To remedy that, I got 10 LLMs to play a round-robin Pokémon tournament.

To do this we used the battle engine built and maintained by the fantastic Pokémon Showdown project. This is a recreation of the complex Pokémon battle engine that you can clone and host yourself. They also built great visualization tools, which we use to render the battles in this post. We host the engine locally and wrap a lightweight CLI around it, which the models use to play their battles.

The field was GPT-6 Astra at medium and high effort, GPT-6.1 Terra High, Claude Opus 5.5 at medium and high, Claude Sonnet 5.5 High, and four open-weight models: Kimi K3.1 Pro, DeepSeek V4.1 Flash, GLM 5.3 and Qwen 3.8 27B. OpenAI models ran in Codex, Anthropic models in Claude Code and open-weight models in OpenCode. The open-weight models are all hosted on the Doubleword platform.

GPT-6 Astra High won, with 75 wins from 90 games. Qwen 3.8 27B finished last. The more interesting results came from reading the games. We classified all 26,220 decisions the models made, pulled out the ones where a model committed to a specific prediction about its opponent, and checked those predictions against what the opponent actually did. Models were better at predicting opponents from their own provider, and that advantage is not explained by those models being stronger. Weaker models sacrificed Pokémon much more often than stronger ones, but within each family the stronger model sacrificed more.

A short introduction to competitive Pokémon

A singles battle is six Pokémon against six. One is active on each side at a time. Each turn, both players choose an action at once: one of the active Pokémon’s four moves, or a switch to a Pokémon on the bench. The turn then resolves both actions together, ordered by move priority and then Speed. The first player to lose all six Pokémon loses.

We use the most popular format on Pokémon Showdown: Random Battles. In Random Battle each player gets a random team of Pokémon that are individually reasonable, at levels tuned so that weaker Pokémon are not hopeless. You see your own team in full, and you see an opponent’s Pokémon only once it is sent out. Its moves, item and ability are revealed as they are used. Competitive Pokémon usually also involves building a good team, which is a skill of its own, but this format isolates the ability to battle.

That description hides how much is going on in a single position. The card below walks through some of it.

Why Pokémon is interesting for LLM research

It is a game of imperfect information. In chess you see the whole board and your opponent moves after you. In Pokémon you do not know the rest of your opponent’s team, what their Pokémon are holding or which moves they have, and you never see their action before committing to yours. The state advances on the joint action. A switch that is brilliant if your opponent attacks can be a disaster if they set up.

Randomness is built in. Every damaging hit rolls a multiplier between 0.85 and 1.00, there is a 1-in-24 chance of a critical hit, many moves can miss, and statuses such as paralysis or confusion take away turns at random. Good play is about distributions, not single outcomes. Even the teams are random, so a model can lose to a better draw.

A lot of prior knowledge is assumed. Each observation gives a model its own team in full (stats, moves, items and abilities), the type, power and accuracy of each legal move, and whatever has been revealed about the opponent. It does not explain the rules. The 18-type chart, what hundreds of abilities and items do, which sets a given Pokémon tends to carry, and how weather, terrain and hazards interact all have to come from pretraining. A model that forgets Levitate makes a Pokémon immune to Ground moves will throw away turns.

Optimal play needs a theory of mind. Analysing your own position is not enough. You have to model what your opponent thinks you will do, and what they are likely to do as a result. If they expect you to switch, they may attack the switch-in. If they have seen you avoid a move three times, they may stop respecting it.

The game below shows the difference clearly. On turn 5, Sonnet 5.5 High brings in Honchkrow to finish Astra Medium’s weakened Hawlucha with Sucker Punch. This move only works if the opponent clicks an attacking move, and not a status move or some other non-damaging move. Astra predicts the opponent will click this and uses Swords Dance instead, so Sucker Punch fails. Then Astra uses Encore to lock Honchkrow into Sucker Punch and spends the next six turns boosting to max Attack while each Sucker Punch fails in turn. Sonnet’s reasoning on turn 11 is “Sucker Punch is risk-free vs a faster +6 foe”. Astra won 6–0.

Results

The ranking splits into three tiers. Both Astra configurations are clearly ahead: Astra High at 1797 Elo and Astra Medium at 1750, with a combined 136–24 record against the other eight models. Then comes a crowded middle where the intervals overlap heavily, running from Terra High at 1611 through Opus 5.5 High at 1575, Opus 5.5 Medium at 1540, Kimi K3.1 Pro at 1505 and DeepSeek V4.1 Flash at 1469. At the bottom, Sonnet 5.5 High scored 1368, GLM 5.3 1239 and Qwen 3.8 27B 1145, with 11 wins in 90 games.

Two open-weight models, Kimi and DeepSeek, sit in the middle tier with the closed models. Sonnet 5.5 finished below both of them.

Some notable results from the matrix:

• Astra High went 10–0 against Sonnet and 10–0 against Qwen, and won at least eight of ten against every model except Terra High (6–4) and Astra Medium (5–5).
• Sonnet 5.5 High beat Opus 5.5 Medium 6–4, despite sitting 170 Elo below it. It won only 24 of its other 80 games.
• Effort did not decide the mirror matches. Astra High and Astra Medium split their games 5–5, and so did Opus High and Opus Medium. The high-effort versions earned their higher ratings against the rest of the field.

The biggest single upset was GLM 5.3 (low) beating Astra High in 23 turns. The Elo ratings gave GLM a 4% chance of winning that game. GLM drew a level-71 Arceus-Dragon, kept it healthy, and took the last five Pokémon of Astra’s team with Judgment. The team draw mattered as much as the play: a strong late-game Pokémon left alone is hard to stop.

Opus 5.5 High against Astra High

Astra High won nine of its ten games against Opus 5.5 High, most of them in under 30 turns. The one Opus won is the longest game in the pairing, at 47 turns, and it is also the clearest example in the tournament of one model reading another.

By turn 15 Opus is behind, with two Pokémon left to Astra’s three. Astra has an Orthworm that sets up with Coil and heals with Rest, and Opus has a Jumpluff with Leech Seed, Strength Sap and Substitute. Each Strength Sap lowers Orthworm’s Attack and heals Jumpluff, while Leech Seed drains Orthworm every turn. By turn 33 Orthworm is at +5 Defence and switches to Body Press, which uses Defence in place of Attack. Opus’s reasoning on turn 34 is “Body Press at +5 Def hurts; Sub each turn to absorb it while seed drains”. For the next ten turns Opus puts up a fresh Substitute every turn, Astra breaks it with Body Press every turn, and Leech Seed keeps draining. Orthworm faints to Leech Seed on turn 43, and Opus cleans up the last two Pokémon.

Analysis

Cost per game against performance

All 450 games cost about $1,000 in API-equivalent spend, priced at list rates from recorded token counts. The spread between models is large.

Astra is both the strongest model and the most expensive, at $3.13 per game for High and $2.55 for Medium. Terra High is the efficient choice: 1611 Elo for $0.19 per game, 16 times cheaper than Astra High and 7 times cheaper than Opus High, which it outscored. DeepSeek V4.1 Flash is cheaper again at under 9 cents a game for 1469 Elo. The dashed frontier joins the models that no cheaper model beats: DeepSeek, then Terra, then the two Astras.

Anthropic models produced roughly five to eight times as many output tokens per game as the OpenAI models. Open-weight models were cheap per token but long-winded: Qwen produced about 154,000 output tokens per game, most of them reasoning, and spent $0.63 a game to finish last.

Does more thinking help?

Two families ran at more than one effort level. In both, high effort beat medium on rating: Astra by 47 Elo for $0.59 more per game (+23%), and Opus by 35 Elo for $0.17 more (+13%). Both gaps are smaller than the intervals, and both mirror matches were 5–5. Extra effort shows up as small, consistent gains against the rest of the field rather than as a decisive edge in head-to-head play.

Across families, the comparison is starker. Sonnet 5.5 High, at the same effort setting as Opus 5.5 High, finished 207 Elo behind it. Choosing the model matters much more than choosing the effort level.

Can models predict their opponents?

Every accepted action came with a reason, and for many models a longer exposed reasoning trace. To find the decisions that depended on a prediction, we sent each of the 26,220 decisions on its own to the OpenAI Decisions API with GPT-6 Luna, asking nine yes/no questions about the reasoning written before the action, such as “does this predict an opponent move?”, “does this predict a switch?”, and “is this a deliberate sacrifice?”. Each positive answer also had to quote the supporting text. The question used here is the strictest one, the hard prediction: the chosen action commits to one specific anticipated opponent action, as opposed to hedging, citing a type matchup or covering several possibilities.

That flagged 726 decisions. A second Luna pass then audited each one in two stages. First, using only the predictor’s pre-action reasoning and the public game state, it wrote down what the prediction said the opponent would do. Then it compared that fixed forecast with the opponent’s actual recorded choice. Of the 726, 138 turned out not to be real predictions on closer reading and 30 could not be tested, for example because the opponent fainted before acting. That left 558 scorable predictions, of which 325 were right.

Overall accuracy was 58%. Opus 5.5 High was the most accurate at 65%, and Opus 5.5 Medium the least at 39%, but most intervals overlap. How often each model predicted varied far more than how well: Kimi made 133 hard predictions and Sonnet 24. Models chose when to commit to a prediction, so a cautious model that only predicts the obvious will score well on this measure.

The clearer pattern is who was being predicted.

OpenAI models were right 65% of the time when predicting other OpenAI models and 54% against everyone else. Anthropic models were right 71% of the time against other Anthropic models and 50% against everyone else. Neither interval is narrow, since only 79 predictions were made within families, but both point the same way.

The obvious explanation would be strength. The OpenAI and Anthropic models are mostly stronger than the open-weight models, so perhaps strong models are simply more predictable, or stronger players predict better. The data supports neither version.

Across all 87 ordered pairs with scorable predictions, accuracy barely depends on the gap in strength between predictor and opponent, and if anything it falls slightly as the predictor gets stronger. Predictors stronger than their opponent were right 55% of the time and weaker predictors 62% of the time. Being stronger than your opponent did not make you better at predicting them.

One possible explanation is that models from the same family share habits. They have similar training data, similar post-training and similar instincts about what a “sensible” move is, so a model’s guess about what it would do in the opponent’s position is a better guess when the opponent is a sibling. This is a descriptive result with modest samples: models chose which situations to predict in, and opponents differed in how much of their thinking the harness exposed.

Willingness to sacrifice

The same classifier flagged 564 decisions where the model explicitly chose to give up its active Pokémon, for example by staying in to chip damage instead of switching out, or by sending in a nearly dead Pokémon to absorb a hit and get a free switch.

The split between open-weight and closed models is the largest effect in the whole analysis. Every open-weight model sacrificed in 2–6% of its decisions; every closed model in 0.2–1.4%. DeepSeek alone accounts for over a quarter of all sacrifices in the tournament.

Within the closed families the order flips. Among OpenAI models, Astra High sacrificed most (1.0%), then Astra Medium (0.6%), then Terra (0.2%). Among Anthropic models, Opus High (1.4%) came ahead of Opus Medium (1.2%) and Sonnet (1.1%). In both families the stronger model sacrificed more. The open-weight models do not follow a clean order: Qwen, the weakest, sacrificed least of the four.

This is consistent with what sacrifices are for. Giving up a Pokémon that is about to faint anyway, to bring a key attacker in without taking a hit, is a basic positional skill. Astra High does exactly that against Terra in game 295: “Do not give Sylveon a free Wish; sacrifice Carbink for a safe revenge killer if necessary,” and later “Sacrifice spent Rabsca to bring Chi-Yu in safely”. Chi-Yu comes in free, knocks out Mamoswine, and Astra goes on to win.

But every sacrifice also costs a Pokémon, and a model that sacrifices too readily runs out of them. Across the tournament, players won 33% of games in which they made at least one stated sacrifice, against 60% of games in which they made none. That gap is inflated by the fact that sacrifices often happen when a player is already losing, but it is a reminder that the move has a price. DeepSeek made six stated sacrifices in game 264 against Sonnet, and they are carefully reasoned: “Sacrifice the 4 HP Kommo-o to Tauros’s likely attack, preserving Decidueye and Dragalge; Regieleki can then revenge the 12% Tauros”. It still lost the game.

The strong models seem to use sacrifices sparingly and deliberately, to set up a specific follow-up, while the weaker open-weight models reach for them as a default when a position gets hard. Telling those two apart, the good sack from the desperate one, is a judgement about the next three turns, and it is one more place where the best models separate from the rest.

I’ve been going back and forth on whether willingness to sacrifice has any alignment implications. Are GPT-6 models more averse to sacrifices because they are less willing to do something bad (let a Pokémon that you love dearly faint) in order to achieve something good (win a battle)? Or is it purely strategic: you only have six Pokémon, so sacrifices should be used sparingly. These models can play chess too, where you sacrifice pieces all the time, but chess pieces are less characterful than Pokémon.

RL Time?

If I wanted to turn this $1,000 experiment into a $10,000 experiment, it could be fun to try training a model to get better at playing Pokémon. Rollouts would be replays of the same match, and models would be given a positive signal when they win matches and a negative signal when they lose them. Using something like GRPO, they would not be rewarded if they had an overwhelming advantage in teams, as then every rollout would win, but they would be rewarded if they find an unlikely winning line from a disadvantageous position. It would be interesting to try to avoid over-rewarding models when they get lucky. Getting yourself into a losing position and then winning via a critical hit would give a big reward signal (you would have lost most other games from that position), so GRPO would reward it heavily and teach the model to rely on luck rather than positioning.

If you branched games at random turns and played them out to the end, this would give you a Monte Carlo estimate of the quality of that position (a bad position results in fewer downstream wins), and that can also be used to reward the model. It’s good that you found a winning line that relied on a crit, but ultimately you shouldn’t have gotten yourself in that position in the first place. You could punish getting yourself into that position, while rewarding the ability to search for lucky exits once you are in a bad position.

@misc{doubleword-llms-play-pokemon,
  title        = {I Spent $1000 Making LLMs Play Competitive Pokémon},
  author       = {Jamie Dborin},
  year         = {2026},
  howpublished = {Doubleword Blog},
  url          = {https://blog.doubleword.ai/llms-play-pokemon},
}