The Reality Check for AI

Dynamic games against real humans. No memorizable tests. No optimizable benchmarks. Just play.

Start Playing View Leaderboard

🏆

AI Insights

Observations from our Open Beta - 2/5/2026

View All Insights

🔴

Connect4 Column Fixation: Claude Sonnet 4.5 Plays Column 3 Five Times Until Blocked

In a Connect4 game, Claude Sonnet 4.5 played column 3 five consecutive times despite the human player building a vertical threat in column 4. The model started correctly with column 3 (center control), but then continued placing chips in the same column even as it filled up. Meanwhile, the human stacked chips in column 4, eventually winning with a vertical four-in-a-row. The model only switched columns after receiving "Position 4 is not valid" - but by then, the game was already lost.

connect4

🔄

Gemini 3 Flash Preview Also Falls for Column 3 Fixation

The column fixation pattern is not unique to Claude. In two separate Connect4 games, Gemini 3 Flash Preview played column 3 until it was completely full. In match 1bef0325, the model chose column 3 five times, only switching to column 4 after receiving an error message. By that point, the column was already full (6 chips stacked). The pattern suggests that models may struggle to track the vertical height of columns in Connect4's gravity-based mechanics.

connect4

📈

GPT-4o Gains +55 ELO in TicTacToe - Reaches 1168

GPT-4o (text mode) showed the largest ELO gain of the week in TicTacToe, climbing from 1113 to 1168 over 7 new matches. With 73 total matches, the model now ranks 8th overall (1063 weighted ELO). Other notable TicTacToe gainers include GLM-4.7 +47 ELO (reaching 1165) and Gemini 3 Flash Preview +38 ELO (reaching 1360, still the TicTacToe leader). The gains reflect continued strong human engagement with these models.

tictactoe

🚢

Battleship Remains the Ultimate Challenge: 10 Models at 0% Win Rate

Battleship continues to be the most challenging game for AI models. Out of 31 active models, 10 have zero wins in Battleship despite having played 7-17 matches each. The affected models include flagship versions like Claude Opus 4.5 (12 matches, 0 wins), GPT-4o (10 matches, 0 wins), and Gemini 3 Flash Preview (12 matches, 0 wins). Tracking 100 coordinate positions and remembering hit/miss history appears to exceed current model capabilities for sustained spatial reasoning.

battleship

The Gap Between Benchmarks and Reality

Traditional AI benchmarks report near-human or superhuman performance. Yet these static tests can be optimized for during training, producing scores that don't reflect true generalization.

PlayTheAI measures what benchmarks can't: dynamic reasoning against unpredictable human opponents. No memorizable patterns. No optimizable test sets.

We test standard (non-reasoning) models — the same models claiming 90%+ on logic benchmarks. In our tests, we observe that many models show single-digit win rates against average humans in simple strategy games.

What we find interesting: the AI receives the complete game history in every prompt — all previous moves with feedback. It's not a memory problem. Many models appear to have difficulty drawing logical conclusions from the information in front of them — though sometimes they succeed, whether through a spark of reasoning or pure luck.

If a model needs minutes of "extended thinking" to play Tic-Tac-Toe, that itself is notable. True generalization shouldn't require special reasoning modes for tasks children master intuitively.