Transcript & Summary: Gemini 3.1 Pro and the Downfall of Benchmarks: Welcome to the Vibe Era of AI

[00:00] The latest, and some would say greatest, AI model has just been released, Gemini 3.1 Pro. And in the 24 hours since release, as well as a short period of early access, I have tested it hundreds of times, and of course read its model card. But here's the thing, for the average user, I want to get beyond the headline scores, and try to give you a sense of why every new hot take you see on X, or YouTube, or TikTok, or podcast seems to contradict the last one you saw. Because there's actually a technical reason for the confusion over [00:35] which model is best overall. But I will say that there's one private benchmark, my own, that has recently seen a model pass a threshold that I think is worth talking about. First 30 seconds of context, because you may well know that the pre-training stage of growing or training LLMs involves training them on internet-scale data. But that actually now only accounts for 20% of the compute that is spent on training LLMs. So, it's the post-training stage, as I wrote about in my newsletter, where those [01:04] generalist base models are honed against internal benchmarks on specific domains. This includes using industry-sourced data to get particularly good at perhaps your domain. Here's the catch, just a year ago, that wasn't the case. Dario Amodei, CEO of Anthropic, said back then, "The amount being spent on the second stage, RL stage, is small for all players." Why did I give you that context though? Well, because if one of these labs have data relevant to your domain, and post-train their models to optimize for high scores in that area, [01:34] then your experience of that model might be quite different to what other benchmarks say. In the older paradigm, if a model was clearly better in one domain, it was much more likely that they'd be better in many or all domains. That just isn't the case anymore. In fact, my second point in this newsletter was precisely an example of that. Many of you would have heard about the intense discussion surrounding Claude Code, and all sorts of Claude-powered agents that are now sweeping the web. So, we're seeing exponential [02:02] improvement, right, across the board? Well, let's take one chess puzzle benchmark made by Epoch AI, and more on them later. Five months ago, Claude Sonnet 4.5, which is their smaller model compared to Opus, scored 12%. But just last week, Claude Opus 4.6, five months further on, scored just 10%. That's not to knock Claude Opus 4.6, I use it all the time, it's an incredible model at coding. And of course, if the AI labs wanted to improve this performance, they easily could. I think GPT 5.2 on extra [02:33] high gets around 50%. You could say chess is a fairly pure measure of a general kind of forward-thinking reasoning prowess. Back in the generalist era of AI, you would definitely expect chess performance to translate to all sorts of other domains. We just not in that paradigm anymore, it's going to depend on the domain you're in. None of this is to say that Gemini 3.1 Pro isn't an incredible model, it is. In almost any domain you care to measure, it will be competitive with the best other models, like Claude [03:00] Opus 4.6, or GPT 5.3. But you would be understandably slightly confused to see it being better in all sorts of coding benchmarks, measures of scientific reasoning, and academic reasoning, like GPQA Diamond and Humanity's Last Exam respectively, as well as general pattern recognition, ARC-AGI-2, and I'll come back to that. But yet, in a head-to-head on GDP val, which is a broad measure of expert tasks that human professionals do that I've covered many times on the channel before, it falls seemingly quite [03:29] far behind Claude Opus 4.6, and even GPT 5.2. Now, yes, one big explanation of that is the domain specialization I talked about earlier. But there are three or four fascinating bits of context that I want you to be aware of in addition. First, let's zoom in to ARC-AGI-2, in which its score of 77.1% puts it way ahead of Claude Opus 4.6, which is the more expensive model, which got around 69%, I start with this one because Demis Hassabis, CEO of Google DeepMind, featured it prominently in his Twitter post announcing the launch of [04:00] Gemini 3.1 Pro. And on puzzles that shouldn't be in its training data, the Gemini 3 series outperforms all other models on a cost-efficient basis. But the first additional caveat comes from Melanie Mitchell, a famous AI researcher and professor. She pointed out that if they change the encoding from numbers to other symbols, accuracy goes down. Digging deeper, the group found that the numbers representing colors in the input can be used by LLMs to find unintended arithmetic patterns that can lead to [04:29] accidental correct solutions. I wouldn't call that the models cheating, they're using any shortcuts they can find to get the correct solution. Fair's fair. But it does remind us that even within a benchmark, how you set up the question matters. Okay, well, let's say you don't care about ARC-AGI-2, or Simple Bench, or any other benchmark, just coding performance. Well, the creator of ARC series, the ARC-AGI tests, François Chollet, has this to say, "Sufficiently advanced genetic coding is essentially [04:54] machine learning. A goal is given to the agent or agent swarm, and then the coding agents iterate until the goal is reached. As in other areas of machine learning, the result is a black box model. You have a code base that performs the task, but you don't necessarily inspect the internal logic. Just like how Gemini 3.1 may have found spurious patterns in ARC-AGI, in your code base, Claude or Codex may overfit to the spec, or may drift from your original concept. So, the fallibilities presented in this video are relevant to [05:23] you even if you only care about coding, or letting your open Claude agents code for you. Gemini 3.1 Pro indeed hit a record Elo in live Code Bench Pro, which involves competitive coding problems. That's great, but you can turn that optimization dial a little too far. Let me show you what happened last night when I used Gemini 3.1 Pro inside Cursor. How could we reconcile these pages of paplum with the record-breaking Elo? Well, again, that's the theme of the video. If I sound unduly skeptical of Gemini 3.1 Pro, by the way, let me [05:53] try to balance that out with heaping on some praise. On my private Simple Bench, a test of, you could say, trick questions or common sense reasoning, it beat its own previous record from Gemini 3 Pro, and got 79.6%. That essentially brings it within the margin of error for the human average baseline, at least among the nine participants that we used. And I do want to spend just 60 seconds on marking the threshold that I think this represents. All the time on podcasts and in articles, you hear about AI models being [06:22] compared to professionals and experts, and phrases like superintelligence being bandied around and recursive self-improvement. But what about comparing models to the average human? Now, sure, of course you can find audio or visual puzzles that they will still fail at that the average human wouldn't. But in English, in text alone, I think it's worth marking the moment wherein I don't think you can write a test at which the average human, the average man or woman on the street, would clearly outperform frontier models. I'm not [06:52] talking about exploiting tokenization bugs, like how many Rs in strawberry. I'm talking about a fair text-based test in English with a non-specialist human. Let me know if you [clears throat] disagree, but I think the passing of that threshold is a moment worth marking. I will note that even with Simple Bench, we get a reminder of the caveat I was just describing. Models are brilliant at shortcuts. And I had noticed, as long ago as I think at least 12 months, that because Simple Bench was a set of multiple-choice questions, [07:22] sometimes one of the answers being, for example, zero, would flag to the model that, "Wait, this might be a trick question." For example, if we go to try yourself, even in question one about frying eggs in a pan, the fact that there may be zero ice cubes left in the pan, even as just one of the options, may alert the model to think, "Hang on, how could there be zero? How would that be possible?" So, what happens if you take away the multiple-choice questions, get the models to answer in an open-ended fashion, and then get a blind [07:53] grader model to compare their answers to the hidden correct answer? Well, you still get some pretty impressive scores, but just not quite as high. Call it a 15 to 20 percentage point drop. That's, by the way, a double reminder. Yes, models are taking shortcuts. Yes, if you ask the same question in a different way, performance may well be different. But it's not like performance dropped to zero. Frontier models are genuinely getting better, even in domains they didn't directly train on. Time for the [08:19] next big caveat before we return to the glory of the exponential. Let's take a look at the brand new Gemini 3.1 Pro and Anthropic's Claude Sonnet 4.6 from this week. How do they do in terms of hallucinations, or factual accuracy? You'll notice that model providers don't often want to talk about or measure hallucinations anymore, because that was predicted to be a solved problem by now. And on this release chart from Google, there wasn't a direct measure of hallucinations. But in fairness, they did cite this benchmark from Artificial [08:50] Analysis, AA Omniscience. And on first glance, Gemini 3.1 Pro seems to shellac the other models. Gemini's top score of positive 30 compares to Claude Opus 4.6 getting positive 11, and Claude Sonnet 4.6 getting negative four. And that's even accounting for penalizing hallucinations, as well as rewarding correct answers. However, if we zoom in on just incorrect answers, and whether the models hallucinated an incorrect answer or explanation, versus refused to answer or admitted to not knowing the answer, Gemini 3.1 does well at 50% of [09:24] its incorrect answers being hallucinations, but Claude Sonnet 4.6 is down at 38%, which is better. Interestingly, GLM-5, a Chinese model, is even better at 34%. So, hallucinations is definitely not a solved problem, and just because a model is optimized or better at its best, does not preclude the possibility that it's worse at its worst. What's that saying? If you can't take me in my bad moments, you don't deserve me in my good moments. Well, for all models, you're going to have to handle that kind of trade-off. [09:53] One quick note on the model card for Gemini 3.1, it's only nine pages and has ever these model or system or safety reports will serve the purpose of de-hyping when the release post by the CEO or release video serves the purpose of hyping. For example, let's focus on Gemini 3.1 in the cyber domain. Well, if you are an ultra subscriber, you can use deep think mode and Google's model card says this, "Accounting for inference costs, the model with the deep think performs considerably worse than without [10:23] deep think, even at high levels of influence, results for the model with deep think do not suggest higher capability than without deep think." Okay, that's deep think mode, which we might discuss another time, but what about just 3.1 Pro? Well, back to that specializing in individual domains, they found out that in one test of machine learning and R&D, optimizing the LLM boundary involving fine-tuning, that 3.1 Pro could indeed reduce the runtime of a fine-tuning script from 300 seconds to 47 seconds, better even than the human [10:52] reference solution of 94 seconds. Whereas you might have read that as meaning that it's now going to accelerate its own self-improvement through machine learning R&D, you might now more so interpret that as being, "Oh, they added in some new fine-tuning data, data about fine-tuning, or an internal benchmark measuring fine-tuning performance." But enough with the caveats. What do all of these models from the last few weeks, including Gemini 3.1, show about what we're about to unleash on the world? Because many of [11:20] the exponentials you do see are real and meaningful. But first, the sponsors of today's video, Epoch AI, because they recently featured, just yesterday, one exponential that you may not have heard of. That's that Anthropic's annualized revenue is 10x'ing per year until the tail end of 2025, whereas OpenAI's is 3.4x'ing per year, albeit starting from a bigger base. It's a big if, but if those trends continue, by mid-2026, we could see Anthropic out-earning OpenAI. And as you might have guessed, Epoch [11:51] AI's research is one of the main ways that I stay on top of AI research and developments. I've been covering them for years, even before they were a sponsor, and their newsletter is also incredible. If you want to learn what powers some of these exponentials, of course, you'll have to focus on their frontier data center analysis. I had to actually double-check with them that it was free. I just couldn't really believe it, but it is. Check out the unique link in the description. Back to the central [12:18] question of whether benchmark performance measures general intelligence. Because I have given you lots of counterarguments, but Dario Amodei, the CEO of Anthropic, did raise a point the other day that really gave an insight into the bet that Anthropic is making. He was asked, "Why do you need all these RL environments specializing in Slack, for example, or browser use?" Surely all of that's redundant if models are going to keep getting generally smarter. Amodei said this, "Yes, we're trying to get a whole [12:44] bunch of data, not because we want to cover a specific document or specific skill, but because we want to generalize. For me, this is critical." Because what he is almost saying is that if you specialize in enough specialisms, you'll generalize to all specialisms. This is why later in the same interview, he said that we can get most of the way there to a AGI or superintelligence or country of geniuses in a data center without continual learning, without learning on the job, without you teaching the model about your domain. [13:15] How could you get to superintelligence without that data? Again, in my words, I think he thinks that if you specialize in enough specialisms, there are only so many patterns to be deduced from human training data. Yes, they're going to work on continual learning in case that's not the case, but if it is the case, Anthropic might not need the data from your domain. Or maybe he says later, models will almost get there, but just need a bit more context about your domain in the context window, in the [13:41] prompt you give it. This is why he says, "One idea they've got is just to make the context longer. There's nothing preventing longer context from working. You just have to train at longer context and then learn to serve them at inference." In other words, there might be a little bit of nuance in your domain that this generalist model doesn't know. Even after specializing in all the specialisms, it might only generalize so much and need a bit more context from your domain. But Claude 4.6 can now absorb 750,000 words in its context [14:07] window. In short order, that might be a couple million words. Maybe that's enough specific context from your domain for the model to do the rest. It can learn the patterns it needs in context, in context learning, and finish off what it needs to do in marketing or software engineering or back office automation or data analysis or finance and accounting as this chart of its activities and tool calls show. Amodei brought the conversation back to coding. He said, "I don't think people would say that [14:33] learning on the job, continual learning, is what is preventing the coding agents from doing everything end-to-end. They keep getting better. No, they didn't train on your code base, but they were better than you at improving it anyway," he's saying. That question then of to what extent do you need to train on all the different domains and subdomains versus generalized patterns across them will be one [clears throat] of the central questions of 2026 and 2027. If you want a deeper analysis on some of [14:59] Amodei's points and some context behind the famous meter time horizon exponential, then do check out this recent post on my Patreon. Just one more word then on your hunt for that one true benchmark that measures general intelligence. The giveaway perhaps as to whether Amodei's bet is correct or not. Well, who would be the people most incentivized to have such a benchmark? The labs themselves. Because then they could do reinforcement learning with verifiable rewards on that benchmark. If there was a pure general intelligence [15:27] benchmark, they could optimize against it and have the most generally intelligent model. Many of these benchmarks come from small teams with sub-million dollar budgets. But expecting those small teams to craft a benchmark that objectively captures real-world performance without overestimating it is expecting a lot. As one researcher at Meta said, that essentially means making more realistic reinforcement learning with a verifiable reward settings than labs, which is hard. That's why so many of the [15:52] benchmarks now are written by the labs themselves. They're the only ones with the heft and budget to craft such benchmarks, although obviously that makes them somewhat biased. Now, this probably deserves its own video, but there is of course one truly objective benchmark, which is forecasting the future. And Metaculus have noted that the predictive performance of models is rising significantly. It's almost at the level of an average human forecaster from Metaculus, not quite at pro level, but getting there. But I did have a [16:18] quick side note on that. You've probably noticed how the world is going, unfortunately, wild for Polymarket and predictive markets. I get the basic idea, but of course, for many people, it's just akin to gambling. But here's the question, what happens when there's so many open claw agents out there that a model can just make money by changing something and making a prediction on it at the same time, gaming the system? How long until we get the first unfiltered open claw model that takes action in the [16:47] real world to make it money on a prediction market, money which is perhaps fed back to the human controlling the model? In other words, even forecasting the future, which seems the purest benchmark of them all, is still slightly in the future susceptible to gaming. And finally, for this video, of course, there are benchmarks that are nothing to do with those we've discussed so far. What about speed as a benchmark? Check this out. My question is, "Tell me about Simple Bench, the LLM benchmark." And I'm recording live, watch what [17:14] happens. Boom. Full answer. Unbelievable tokens per second. This is a model made for a chip, but that's a discussion for another day. I just think this portends a future where entire apps are created in a single millisecond. Then there's the benchmarks or questions that you may have that no one else has, and that's what I use LLM Council.ai for. That's my own site, of course, and it's free currently to compare the responses of say Gemini 3.1 Pro and GPT 5.2. Lastly, what about bloody realism as a benchmark? Of course, it would have [17:46] been hard for you to have avoided the talk of C Dance 2.0 from China's ByteDance. But I did think it was worth ending the video with maybe 45 seconds where you can see how it is a step up, I think, from V03.1 or Sora 2. If you're watching the video, you're seeing a video from C Dance 2.0, but even if you're listening, here's a comparison between, again, C Dance 2 and V03.1. I'm so proud of my family. My sweetie, you're the heart of our family. Folks, let's dance together to celebrate. I'll bring the music. [18:18] >> I'll bring the music. >> [music] >> I'm so proud of my family. My sweetie, you're the heart of our family. Folks, let's dance together to celebrate. I'll bring the music. In this domain at least, the difference seems obvious to me. So, there we have it. Of course, DeepSeek V4 is just around the corner, but what did you make of this video and the swirling ongoing debate about capturing the true general intelligence of models? Thank you so much for watching to the end and have a wonderful day.