GPT-5 mini vs Amazon Nova Pro

I asked copilot to check correct answers in a quiz for me

https://preview.redd.it/5b49y505cz5g1.jpg?width=482&format=pjpg&auto=webp&s=7bfd451451d464fe2dd5ee4d31dc9a6137f39bde

Hi everyone,

We ran task benchmarks on the GPT-5 series models, and as per general consensus, they are likely not a break through in intelligence. But they are a good replacement of o3, o1 and gpt-4.1. And lower latency and the cost improvements are impressive! Likely really good models for chatgpt, even though users have to get used to them.

**For builders, perhaps one way to look at it:**

o3 and gpt-4.1 -> gpt-5

o1 -> gpt-5-mini

o1-mini -> gpt-5-nano

**But let's look at a tricky failure case to be aware of.**

Part of our context oriented task evals, we task the model to read a travel journal and count the number of visited cities:

Question: "How many cities does the author mention"

Expected: 19

GPT-5: 12

Models that consistently gets this right is gemini-2.5-flash, gemini-2.5-pro, claude-sonnet-4, claude-opus-4, claude-sonnet-3.7, claude-3.5-sonnet, gpt-oss-120b, grok-4.

To be a good model for building with, context attention is one of the primary criterias. What makes Anthropic models stand out is how well they have been utilising the context window even since sonnet-3.5. Gemini series and Grok seems to be putting attention to this as well.

You can read more about our task categories and eval methods here: [https://opper.ai/models](https://opper.ai/models)

For those building with it, anyone else seeing similar strengths/weaknesses?

So, I was testing DeepSeek-R1 with a math problem I found in a textbook for 9-year-olds **(yes, really)**, and the model managed to crack it.

The problem was:

`"Find two 3-digit palindromic numbers that add up to a 4-digit palindromic number. Note: the first digit of any of these numbers can't be 0."`

[R1 starts thinking...](https://preview.redd.it/ml5hnng3rwge1.jpg?width=1800&format=pjpg&auto=webp&s=1456610eeff8d8b9a122d86fbb44967f84f682d9)

Now, here’s where it gets interesting. R1 thought for a bit, found the correct answer in its `<think></think>` block, then went ahead to output it—but made a mistake.

[R1 makes a mistake...](https://preview.redd.it/77bke6q1swge1.jpg?width=1800&format=pjpg&auto=webp&s=d6eac07677fe576be9e699776a2134cba1d15c62)

Before even finishing its response, it caught its own error, backtracked, and corrected itself on the fly outside of the`<think></think>` block.

[R1 corrects itself...](https://preview.redd.it/yc3zjamsswge1.jpg?width=1800&format=pjpg&auto=webp&s=903d42998593e95a68ff32006b7bac6335df9f1e)

[R1's final answer.](https://preview.redd.it/j8vgvxn3twge1.jpg?width=1800&format=pjpg&auto=webp&s=b189fce4a099ed9182b315c2164a1071a4a32104)

[DeepSeek-R1 complete answer.](https://pastebin.com/0Ayv77LN)

Regarding the problem, **no other LLM solved it, except for** [**OpenAI o1**](https://pastebin.com/YCRR521W).

So now I’m wondering—**what's holding them back?** Is it the tokenizer's weaknesses? The sampling parameters (even when all where at the recommended settings they failed)? Or maybe, just maybe, non-thinking LLMs are really that bad at math?

Would love to hear thoughts on this.

Unsuccessful attemps by other models:

* [chatgpt-4o-latest-20241120](https://pastebin.com/r8VKHrcA)
* [claude-3-5-sonnet-20241022](https://pastebin.com/tXc7wGVz)
* [phi-4](https://pastebin.com/zGzQJ8B5)
* [amazon-nova-pro-v1.0](https://pastebin.com/vt54UFBe)
* [gemini-exp-1206](https://pastebin.com/eSN4y6E0)
* [llama-3.1-405b-instruct-bf16](https://pastebin.com/jVj1KcMF)
* [qwen-max-2025-01-25](https://pastebin.com/ZRLfhEfU)

Hi LocalLlama.

Here are the results from the March run of the GACL. A few observations from my side:

* **GPT-5.4** clearly leads among the major models at the moment.
* **Qwen3.5-27B** performed better than every other Qwen model except **397B**, trailing it by only **0.04 points**. In my opinion, it’s an outstanding model.
* **Kimi2.5** is currently the top **open-weight** model, ranking **#6 globally**, while **GLM-5** comes next at **#7 globally**.
* Significant difference between Opus and Sonnet, more than I expected.
* **GPT models dominate the Battleship game.** However, **Tic-Tac-Toe** didn’t work well as a benchmark since nearly all models performed similarly. I’m planning to replace it with another game next month. Suggestions are welcome.

For context, **GACL** is a league where models generate **agent code** to play **seven different games**. Each model produces **two agents**, and each agent competes against every other agent except its paired “friendly” agent from the same model. In other words, the models themselves don’t play the games but they generate the agents that do. Only the top-performing agent from each model is considered when creating the leaderboards.

All **game logs, scoreboards, and generated agent codes** are available on the league page.

[Github Link](https://github.com/summersonnn/Game-Agent-Coding-Benchmark)

[League Link](https://gameagentcodingleague.com/leaderboard.html)

400k context window, cheaper than K2 providers.
The Gemini Flash 2.5 is the second most popular model on Openrouter, and 5-Mini is better in almost every respect.
It follows instructions well, handles long context effectively, and is fast...

Just saying it is a great workhorse.