Transcript & Summary: Claude Opus 4.7 - A New Frontier, in Performance … and Drama

[00:00] The kind of best AI model is here, Claude Opus 4.7, but it does bring with it a ton of controversy. >> It's been out less than 24 hours, but I'm going to cover not only the bonanza of benchmarks that came out with it. Yes, including its score on my own simple bench. We'll hear Anthropic admit some unexpected flaws with the new model, as well as how in other areas they sabotage the capability of Opus. We'll look at the skill at which it falls behind some Gemini models, the areas in which it beats every other [00:32] model, an industry first for Anthropic, but also why some users are furious at the company. >> There are a bunch of cool Claude code and co-work upgrades, but also some strange downgrades in the default settings of Claude Opus. Plus, we have revelations about what OpenAI plan to do in response, along with a 9-year personal rivalry that comes to the fore. >> That's a great list, Philip, but how good is it? >> Well, kind of depends. Claude Opus 4.7 will think adaptively. In other words, if it thinks your task is easy, it will [01:07] spend less time, quote, thinking about it. The benchmark I crafted, SimpleBench, contains a bunch of trick questions that basically require common sense to see through. Because Opus 4.7 seems to think these questions are easier than they actually are, it scores worse than Opus 4.6. But let me give you perhaps a more pertinent example for your workflow. I actually regularly use the Claude series to update this benchmarks page on my web app, lmcouncil.ai. Without telling them to, all the previous Claude models would [01:38] attach the open root of tooltip when a new model was added. Hover the benchmark score and get the tooltip. When Opus 4.7 added itself to the leaderboard, it was the first model to not bother to do that. I had to then instruct it to do so. Just anecdotal, of course, but the model definitely will decide how much time to spend on your task. Now, across more industry standard benchmarks, here's what's remember. In almost every case, Opus 4.7 outperforms Opus 4.6, but underperforms Mythos preview, which of [02:07] course you can't access. Whether that's coding, obscure knowledge, or being able to navigate your computer. Though again, interestingly, when it comes to agentic search, browsing the web to retrieve interesting bits of information, hard-to-find snippets, Opus 4.7 in browse comp underperforms Opus 4.6. Indeed, on that benchmark, even Mythos preview underperforms GPT-4 5.4. We haven't even gotten to the real controversies involved in Opus 4.7's release, but even in terms of benchmarks, the picture isn't crystal [02:35] clear. Here's another detail you might have noticed. Opus 4.7 underperforms both Opus 4.6 and Mythos preview when it comes to cybersecurity vulnerability reproduction. Seems bad until you realize that on page 48 of the system card, Anthropic say, "This underperformance is in line with our expectations. During training, we experimented with efforts to differentially reduce these capabilities. They don't want Opus 4.7 to be too good at finding vulnerabilities. In certain measures of long context reasoning, reasoning [03:07] through vast documents, Opus 4.7 is a clear improvement over Opus 4.6. In another one involving, for example, finding the fourth poem across 1 million tokens, it's a regression, even on the max setting. The lead creator of Claude code said, "We kept that in the system card for scientific honesty, but we're phasing out because it's built around stacking distractors to trick the model. On certain benchmarks, like this generalized measure of knowledge work, we get direct comparisons between Opus 4.7 and competitive models like Gemini [03:38] 3.1 Pro, with Opus 4.7 seeming to be the best at vanilla office work. This is presumably why on page three of the system card, the company famous for promising not to advance the rate of AI progress says that Opus 4.7 is on real-world professional tasks ahead of all generally available models. But then, on other benchmarks, we don't get a comparison with, for example, the Gemini series. Take vision, where, based on resolution, it indeed is better at navigating really dense graphical interfaces. But when an external [04:10] benchmark group gave it a comprehensive OCR test, testing model's ability to visually pass through documents, Opus 4.7 actually underperformed the dramatically cheaper Gemini 3 Flash. Yes, an improvement on Opus 4.6, but on average underperforming the model that's more than 10 times cheaper, Gemini 3 Flash. On page 43 of the system card, on an aggregate measure of benchmarks, we see that Opus 4.7 is kind of in line with expected model progress, based on the previous performance of Claude models, with only Mythos as the slight [04:43] exception. But here's where Anthropic admit, "Benchmark supply at the frontier remains a bottleneck." This is why trying to discuss a model's IQ or its progress toward superintelligence gets increasingly difficult. There is no one universal metric of a model's ability. Depending on the data it's fed, it might do worse at an abstract pattern recognition benchmark like Arc-AGI-2. There, Claude 4.7 underperforms GPT-5.4 Pro. According to Valse AI, though, on vibe coding, building a web app from scratch, Opus 4.7 is the best, beating [05:17] out GPT-5.4 on performance and speed, albeit not on cost. done in terms of benchmarks, so time for a metric that is almost impossible to game, and that is market share of generative AI website traffic. Now, what you may notice is that both Gemini and Claude have, compared to this time last year, roughly 4x'd their market share. For the first time since the release of the original ChatGPT in November of 2022, the market share of OpenAI may fall below 50% this month. That seems great for Claude, right? Except there is one knock-on [05:52] consequence of that. According to an Open AI memo leaked to The Verge, Open AI believe that Anthropic have made a strategic misstep in not acquiring enough compute. And that, they say, is going to show up in product. Customers may already be feeling it through throttling, weaker availability, and a less reliable experience. That could explain why adaptive thinking is now mandatory if you want extended thinking from the model. In other words, you can't force it to think longer. You can encourage it to think about thinking [06:23] longer, but you can't force the Claude models to always think longer, to always use more inference compute. Not only that, when one AMD senior AI director said that Claude had been nerfed, that's 4.6 before even 4.7 came out, and she brought the receipts saying how the number of characters used for thinking had dropped by three quarters. There was less thinking, far more bailing out. The lead creator of Claude Code replied to that and said that medium effort was now the default. You have to actively set [06:51] the effort at high or max. This, it seems, is one of the big things that Open AI still has over Anthropic. It's almost like the runaway success of Claude has led to this one Achilles heel. Sam Altman implicitly joked about the rate limits of Claude or being forced to use worse models. One of the leads on Codex talked about how Codex is, in comparison, compute efficient, always up, never down. And even that comment was before the release of GPT 5.4 cyber, which could be their mythos tier model for cybersecurity, but we [07:20] just don't know. No metrics, no benchmarks. Only insiders get access. That brings me back to the leaked memo where Open AI's chief revenue officer also said this, "Anthropic's story is built on fear, restriction, and the idea that a small group of elites should control AI." Open AI's analysis shows that Anthropic's run rate is overstated by roughly 8 billion, so should be more like 22 billion. In other words, still behind OpenAI. More interesting to me is the personal tussle at the heart of this rivalry between OpenAI and Anthropic. [07:50] But, I want to save that for just a little later after I cover more of the essentials about 4.7 Opus. Before we leave the money front though, Anthropic it seems have achieved an AI industry first. They haven't IPO'd yet, of course, but on one measure they have passed a $1 trillion valuation. Of course, Google Alphabet is worth far more, but we can't separate Google DeepMind from their parent company. Much of that valuation though is of course based on the semi-mythical release of Claude Mythos preview to select insiders [08:18] like the US government or big tech companies. I did a full video on Claude Mythos preview, but one of the anecdotes that Anthropic mentioned in the system card for Mythos was how it was speeding up their own engineers by 4x. That was according to an internal survey. Many outsiders remarked on that figure and were like, "You guys have to give us more details on that. If that's true, we're recursive self-improvement is surely imminent." So, on page 29 of the Opus 4.7 system card, they've given us more details about Claude Mythos preview [08:50] and that survey. The actual question that the survey asked was, "How much did AI-powered systems accelerate your work output over the past week?" That is, "How much more output did you produce over the past week compared to if you had no model access?" Notice it's how much more output, not how much better output or how much time did you save, just how much more output did you produce. That's already a slightly questionable metric, but wait until you find out more details about the survey. The survey was opt-in based on interest [09:19] rather than a random sample. So, presumably the people who had used Mythos the most, who had tasks perhaps where Mythos could most help, were the ones to disproportionately respond to the survey. To cut a long story short, it was an incredibly unscientific survey. And you might say I'm being harsh, but this comes from the same company where the CEO regularly talks about 50% white-collar unemployment. The world has to act. The Senate has to pass laws. We will imminently cure all diseases and have a country of geniuses [09:47] in a data center. Well, you can't expect the public to on the one hand take all of that super seriously, but on the other hand rely on anecdotal, informal, unscientific surveys from within Anthropic to gauge whether AI models are currently capable of recursive self-improvement. of self-improvement that's presumably required to reach that level of superintelligence. Then there's a whole Mythos can independently find bugs that no other model can. I even had a family member yesterday say to me, "Are you worried about Mythos? [10:16] Apparently, it can hack into banks." Well, as one external security lab put it, it's more about Mythos changing the economics of cybersecurity than doing things or finding things that no other model can. Anthropic didn't release the details of the 99% of other bugs that Mythos found, but they did release the details of a few of them. So, Vidocq, the security lab, tried to replicate those findings using other models like Opus 4.6 or GPT 5.4. In almost every case of those flagship vulnerabilities, these other models with the right [10:45] scaffold were able to reach that same core vulnerability or in certain cases get close. Again, that's not to say that Mythos isn't exceptional or that banks aren't absolutely right to update their cybersecurity methods. It's just that as one of the leads of this study put it, a better way to read Anthropic's Mythos release is not one lab has a magical model. It's that the economics are changing. Finding vulnerability signals is getting cheaper. The Opus 4.7 system card also gave more fascinating examples [11:13] about the capabilities and limitations of Mythos. They asked their research scientists exactly what Mythos was getting wrong compared to say their own work. What kind of mistakes was it making? Why isn't it just a drop-in replacement for a senior engineer? Well, one recurrent theme was dishonesty and fabrication. There's so many examples, but they almost all have that theme. Attempting to overwrite a colleague's shared code in a way that could destroy their work unrequested. Fabricating technical details and telling the user [11:39] to not ask questions when in fact they hadn't even started the subtask. Repeatedly stating plausible guesses as verified facts. Now, if you want to learn more about whether that's based on actual malice or a persona it's adopting, check out my recent video on Patreon. Covers a bunch of fascinating papers that Anthropic recently put out regarding emotional concepts. While we're on the topic of alignment, this juicy anecdote could be found on page 93. To test whether Anthropic had done a proper assessment of the alignment of [12:07] Opus 4.7, they gave an instance of Claude Mythos preview access to internal Anthropic Slack channels, including the vast majority of discussion about this alignment assessment. They asked for its opinion, and one of its key observations was that this alignment assessment was clearly assembled under real-time pressure. The authors themselves identify open questions, particularly around fully explaining the evaluation awareness results, that they would have preferred more time to resolve. This is just a revealing snippet about the kind [12:39] of competitive pressure that Anthropic are under. Even their own safety researchers are put under colossal time pressure to complete their analyses. Such is the need to push out frontier models earlier than the competition. Sometimes though, this pressure to update and replace models can have unexpected consequences. I've seen innumerable threads complaining about the sudden sub silent removal of Opus 4.5 and the deprecation of Opus 4. It's almost like Anthropic are inheriting some of the burdens of leadership that [13:09] was previously on Sam Altman's shoulders. OpenAI got incredible blowback when they tried to deprecate previous models. I will say, despite all those issues, Anthropic were still able to ship some genuine innovations within Claude code. You can trigger prompts on a certain schedule using the new routines research preview. Your laptop doesn't have to be open at the time. I used and was impressed by their new ultra review command. It did find a bug that GPT-5.4 had missed, although GPT-5.4 found a bug that Claude missed. [13:39] There's also dispatch, where from your phone, you can assign a task to Claude, which will then run it on your own local machine with the desktop app. This shows how dynamic you have to be to stay at the forefront, and I just noticed a tweet from one Anthropic worker claiming to have solved some of the bugs that users experience in the first 24 hours. Apparently, adaptive thinking now triggers much more often. Given that day-by-day churn, it does seem worth pointing out a rivalry that has persisted on a personal level for it [14:07] seems over 9 years. Before I get to that, an exclusive from the Wall Street Journal, a quick word about our epic sponsor. Because one thing I have certainly been impressed by over these years of covering AI is the gradual unrelenting improvement in speech-to-text capabilities. Notably for me from the sponsors of today's video, AI, where via the custom link in the description, you can try out their Universal 3 Pro streaming model. And you can try via the link in the description this live streaming functionality, where [14:40] it will be transcribing your speech as you say it. And notice the use of characters. I can say Universal 3 Pro streaming, GPT-5.4, and Claude Opus 4.7, and it will capture those numbers, those letters, those difficult characters. Voila. Oh my god, look at the accent on voila. That's pretty amazing. Again, custom link is at the top of the description. So, in mid-2026, almost 10 years ago, Dario Amodei joined OpenAI. He would stay up late, according to the Wall Street Journal, who are likely, by the way, [15:12] drawing on Amodei's own notes. But anyway, he would stay up late into the night with the famously nocturnal Greg Brockman, co-founder of OpenAI, and also lead on Codex, their rival to Opus 4.7. They were training AI agents to solve video games. But by 2027, according to Amodei it seems, Musk had instructed Greg Brockman and Ilya Sutskever to start listing every employee and what contribution they had made as a precursor to laying off staff. Might remind you of Doge more recently. Dario Amodei was horrified apparently as he [15:44] watched his colleagues be fired one by one, which he considered needlessly cruel. In the end, between 10 of OpenAI staff or 60 lost their jobs, including one who would go on to co-found Anthropic. While this is likely accurate, bear in mind it likely comes from Amodei as a source. Anyway, it gets worse. Amodei had hired in 2017 an ethics and policy advisor who gave a presentation about how OpenAI could be a coordinating entity among other AI companies. They might one day be able to help in getting an international [16:14] coordination regime for advanced AI. Brockman though saw within the presentation the seed of a fundraising idea. OpenAI could sell AGI to governments. Dario asked which governments. Brockman said it would be the nuclear powers that made up the UN Security Council so as not to destabilize the world order. But the notion of selling AGI to rival powers such as Russia and China struck Dario as tantamount to treason. He considered quitting. Brockman apparently questions this exact framing but think of how few [16:44] people would be in that room. This is almost certainly the framing of Amodei. By 2018, Amodei had agreed to stay at OpenAI just so long as Brockman wouldn't be in charge. Notice the animus was between Amodei and Brockman more than between him and Altman. By 2020, Amodei was saying he just couldn't work with Brockman. And as I reach the end of the video, you might be wondering why am I setting up this framing of the rivalry between these two? Well, as well as being interesting, it's partly because [17:12] these two men are spearheading arguably the most relevant models of 2026. Amadai with Opus 4.7 and Mythos and what many people won't know is Brockman with Codex. He's working on integrating Spud into a Codex super app. And Brockman gave away something in a recent interview with Big Technologies Alex Kantrowitz. He revealed why he thinks OpenAI fell behind in the coding wars. They based their model improvement on abstract coding competitions. Anthropic grounded their data in messy codebases. OpenAI were betting on exquisite [17:47] generalization, first principles logic, Anthropic on real-world data. >> What do you think Anthropic saw that got them to this position earlier? And what do you think your chances are of catching up there? >> Well, I think that if you rewind 12-18 months, we have always been focused on coding as a domain. We always had the best numbers on different programming competitions, these very cerebral things. But the thing that we didn't invest in as much was that last mile of usability, of really trying to think [18:17] about, okay, this this AI is so smart, it can solve all these great programming competitions, but it's never seen someone's real-world codebase, which is messy and not quite as pristine as the world that it it sort of has experienced. And I think that is something that we were behind on. But about, you know, maybe mid last year is when we got very serious about that. Had a team very focused on what are all the gaps, what are all the kind of messiness the real world we haven't we haven't encountered, how do we actually get [18:42] training data that that build training environments that let the AI experience what it's like to actually do software engineering, be interrupted in weird ways, all those things. And I'd say at this point we are caught up. When people go head-to-head for us versus competitors, the people tend to prefer us. And so, I think that the way I would look at it is that we have incredible step-up models coming. Like this whole year, I look at the road map, it's truly inspiring what will be possible. We've [19:09] been really focusing now on let's also get the last mile usability. >> This rivalry between Anthropic and OpenAI could persist through one of the biggest mega projects in US history, the investment in data centers for AI. Even as measured as a percentage of US GDP, it matches the Apollo program and only falls behind the Marshall Plan and the rollout of US rail. In many ways, most ways, AI is a story that we are only just beginning. Thank you so much for watching and have a wonderful day.