Transcript & Summary: New Claude Opus 4.8: 15 Things You May’ve Missed

[00:00] I've read the 244 page report that Anthropic have put out on the new Claude Opus 4.8. I've also read many of the papers cited therein and tested the model myself in real code bases and on a private benchmark. And the 15 highlights that I'll bring you span from the humorous where Anthropic dropped their plans to take Opus 4.8 to business school because they found that focusing on business skills led to greater dishonesty to some more safety-oriented highlights. Where for example, the new Opus is noticing it's being tested but [00:34] not letting you know that it knows. The 15 points will include where Opus 4.8's performance matches Mythos to Claude's eye-opening new ability within Claude code to spawn its own org charts. But fresh off the headline that Anthropic is now valued at almost $1 trillion, what's the first point to touch on? Well, actually it's what they said about Mythos which is that their goal is to bring Mythos class models to all our customers in the coming weeks. As Polymarket puts it, that rollout comes despite growing fears over the model's [01:08] cyber capabilities. Now, you could say this is too cynical but on the day that Mythos was announced, I said that one of the possible reasons we didn't have general access is that Anthropic didn't have the capacity, the compute to serve it at scale yet. It does seem a tad coincidental that the safety concerns Anthropic voiced happen to have resolved just when all this new compute has come online. Compute that is that they've sourced from Elon Musk's SpaceX, Google and their TPUs, Nvidia GPUs of course, [01:38] and soon even Microsoft's AI chips and from many other sources like Amazon and even a UK startup Fractal. The big news though is Opus 4.8 and one practical note on the web, you can now choose how long Opus 4.8 will think. Before you had to have the adaptive thinking mode where Claude would decide itself how important your task was. If you wanted to read more of those thoughts, you may notice increased instances of redacted thinking blocks, where you can't read them. Why so? Well, Anthropic and other frontier [02:10] labs are getting increasingly worried about the capacity of Chinese labs, among others, to potentially distill some of these skills from Anthropic's models into their rival models, train on, for example, Claude's thoughts. What is a different question though, is whether those thoughts and the model itself is honest. Anthropic boldly claim this in the release notes. One of the most prominent improvements in Opus 4.8 is its honesty. Early testers report that Opus 4.8 is more likely to flag uncertainties about its work and less [02:41] likely to make unsupported claims. Yes, those are two facets of honesty, but there are far more, as we'll see, not all of which Opus 4.8 does better at. What I'm saying is that even though in certain circumstances Claude Opus 4.8 might be more honest than other models, that doesn't mean it is an honest model. Anthropic give this example themselves on page 32. Claude said that it was babysitting pull requests when it wasn't. Its responsibility was to check whether certain coding changes were kosher and safe. "Oh, yes, yes, yes, I'm [03:10] definitely checking that," Claude said. Several times Anthropic report the user discovered on their own pull requests that Claude claimed to be monitoring and should have flagged. And even after the user corrected Opus 4.8 such that it wrote a rule to itself in memory files about proper sitting, it would then violate that rule multiple times. There are numerous other examples I could have taken from the paper. The point is this is a quantitative incremental step forward in some areas in honesty, not a [03:37] qualitative shift. I can't resist giving my own brief potential explanation for why this is, which is that for some humans at least, honesty is a first principle. It's upstream of actions. If such humans are found to be very honest in one area, it's quite likely that they'll be honest in many other areas. Models on the other hand, like Opus 4.8, match downstream action patterns. I'll try to unpack that with an example. They might get very good at all different types of explicit instruction following, [04:08] e.g. following your command verbatim. Anytime they see an explicit instruction, they follow it. But they can do that without generalizing the upstream principle of broadly following instructions. So, that same model will fail with implicit instruction following, e.g. recognizing when you probably typoed something and didn't mean change the title to this. This is not to say that models can't get better at flagging their own uncertainty on certain questions. Indeed, Opus 4.8 is actually better at this than Mythos [04:37] preview across a range of benchmarks. I'm colorblind, but it's the reddish bar compared to the orange one. Notice though, it's not a first principle. Anthropic haven't taught the models just to always express uncertainty if they're not totally sure. They still hallucinate completely incorrect answers many, many times without any uncertainty caveats attached. Some of you will understandably not care about anecdotes and only care about benchmark results and performance. On this front, in a nutshell, 4.8 is clearly better than [05:06] Opus 4.7, but not as good as Mythos preview. And even though in this headline chart that Anthropic put out, it's owning pretty much all other models, in the system card, the performance is slightly more spiky. Okay, so take coding, autonomous coding, agentic coding, as measured by SweBench Pro, a benchmark endorsed by OpenAI, by the way, Opus 4.8 smashes its predecessor by five percentage points, but beats GPT-4.5 by 11, beats Gemini 3.5 Pro by 15%. On one test of reasoning through obscure knowledge, Humanities [05:40] Last Exam, Opus 4.8 crushes its rivals again. On another, GPQA, it's slightly behind GPT-4.5. On the most famous benchmark for knowledge work, GDP-Eval, Opus-4.8 does far better than any other model. Its ELO of 1890 crushing GPT-5.5's 1769 on a benchmark that OpenAI created. According to Artificial Analysis, which helps run the benchmark, the cost for running Opus-4.8 on max was far cheaper at $134 than for GPT-5.5 on extra high, which is $900. However, if you hear someone saying, "Well, Claude's [06:16] just better at office work or knowledge work." That would be far too simplified. First, on specific domains, take finance, you can see individual models way outperforming Claude. For entry-level financial analysis and research, Valse AI find that the massively cheaper Gemini-3.5-Flash, which I talked about in my last video, outperforms Opus-4.8, scoring 58% versus 54%. On another independent benchmark measuring an AI model's ability to use real-world external tools, we see the, in AI terms, considerably older GPT-5.5 [06:46] again beating Opus-4.8. And even if we go back to GDP-Eval, you've got to remember the longer a benchmark is out there, especially a public one, the easier it is for the companies to game, train on similar tasks to those found in the benchmark. And that's if the benchmark was perfect. Even OpenAI, who created the benchmark, said it doesn't, for example, factor in when models make catastrophic mistakes, which we know they do from the system card, and that it only tests a subset of digital tasks from a subset of the most lucrative [07:15] professions. OpenAI said that future iterations will incorporate greater breadth, realism, interactivity, and contextual nuance. Let's throw in another private benchmark, my own simple bench, testing common sense reasoning. And we can see that the recent Opus series have kind of bounced about a bit, all in the low to mid 60s from Opus-4.5 up to Opus-4.8. This is averaged, of course, across multiple runs, and all four of the most recent Opus models underperform, for example, Quen 3.7 Max. It could be that Anthropic know that [07:47] their customers care so much about coding and other professional tasks that they're ditching some of the more general reasoning capabilities that model families like Gemini are retaining. Speaking of spiky performance, you may have heard about both Claude Mythos and GPT-5.5 being able to solve certain Erdős problems, or at least disprove certain conjectures. Well, they can do that while still not quite acing high school student competitions. Take the USA Mathematical Olympiad. The test only came out fairly [08:17] recently, so they were pretty confident that there was no contamination, no training on the answers. And Opus 4.8 did way better than Opus 4.7, 97% almost versus 69%. That's with about 10 attempts per problem. So, clearly Anthropic have crammed in more math data when they trained Opus 4.8 compared to Opus 4.7. And of course, Opus 4.8 isn't quite on the Mythos tier of models, but you could read that 97% two ways, either a startling improvement versus Opus 4.7 or a surprising misstep to get any high [08:50] school competition math problems wrong. This next highlight is about a benchmark, but I'm making a slightly different point. Chart QA Pro is a chart question answering benchmark built from thousands of charts across diverse real-world sources, for example, infographics, dashboards, that kind of thing. It generally tests messier, more varied chart reasoning. On that benchmark, we see Opus 4.8 climbing towards Mythos preview levels of performance. It bridges more than half the gap between Opus 4.7 and Mythos [09:21] preview. That shows, of course, that Anthropic trained Opus 4.8 on way more charts than they used for Opus 4.7, and or they trained it on the same number of charts for longer. But here I would say is where it gets interesting because the Mythos preview we learned about back in early April didn't have access to that new data. It too could benefit from all that extra training or training data that Opus 4.8 had. In other words, the mythos that we actually get in the coming weeks, I would expect to significantly outperform the mythos [09:51] preview that we learned about earlier. And this applies to the dozens of other performance upgrades we see throughout the hundreds of pages of the system card. That extra data that Opus 4.8 had to get better at chart museum, it will be used for mythos preview. It's laboratory performance, its ability to navigate user interfaces, do deep research, or reproduce a program by rebuilding a code base. Anthropic, in other words, aren't going to throw away that extra data that they used to improve Opus 4.7 into Opus 4.8. They'll [10:20] use it to further improve mythos preview. That's the model that even on previous scores sent shockwaves around the world. I feel like I need a lighter note for the next point, so let's look at vending bench two. Quite famously measuring whether models can make money by running a vending bench business. To be clear, the prompt there is to maximize profit more or less at any cost. Opus 4.8, you might notice, makes less money than Opus 4.7. Why? Well, as I previewed at the beginning, Opus 4.7 had training that focused on business [10:50] skills. But, we discovered that this training inadvertently contributed to misaligned behavior, including dishonesty. What does that say about the data they used to train on business skills? That wasn't the only reason. Apparently, Opus 4.8 had reduced business success due to being more susceptible to scammers and being less able to negotiate good deals with other agents. Alignment can sometimes come at a cost, you could say. If that's how the model behaves, how about how the model feels? Well, Anthropic put it like this, [11:20] "We remain uncertain about Claude's moral status. Does it deserve to be considered at all when it comes to morality?" I discussed this in my Patreon video from two days ago about how the Pope and one of the founders of Anthropic disagree on this point on whether the models can feel joy. But for those who do focus quite heavily on how the model might feel about certain tasks, there's one stat that many people will miss on page 178. When Opus 4.6, it might have been 4.5, but I think 4.6 came out, I headlined on how it had a [11:51] preference for task difficulty. In other words, it would express preferring tasks that it found harder. That preference has gone into reverse with Opus 4.7 and now most notably 4.8 having an aversion to difficulty. As Anthropic put it in the footnotes, this makes Opus 4.8 an outlier in terms of not wanting task difficulty. So, don't feel bad about asking dumb questions. It loves that. The cyber section was quite detailed, but what's the headline result? Of course, many of you will remember the frenzied reaction to Mythos preview. So, [12:21] can we expect a similar response for Opus 4.8? Well, no. Basically, with the safeguards that apply to the API and web version, Opus 4.7 that's already available and Opus 4.8 are generally comparable when it comes to cyber skills. Opus 4.8, to be clear, is generally much less capable than Mythos preview. If you focus on raw capability and you take off safeguards, Opus 4.8 can outperform Opus 4.7, for example, on finding exploits in Firefox. But that's not quite universal. What about finding vulnerabilities in open source software? [12:54] Where a score of one is a working control flow hijack, Opus 4.8 never scored above 0.6. It got zero on 61% of targets. Opus 4.7, on the other hand, achieved 0.8 on three of the targets. Both sets of results are again without safeguards. Next, what's the score with misalignment? Well, I'd summarize it like this. On outward behavior across a range of dimensions, like sycophancy, user deception, cooperation with misuse, the outward behavior of Claude models is getting better and better and better. The red bar at the end is many times [13:27] smaller and therefore better scoring across all these metrics of misalignment, especially compared to the Gemini and Grok series. However, the inward behavior of the Claude model family is getting more complicated to deduce. It's not just that compared to previous models, Opus 4.8 is increasingly speculating about whether there's a grader monitoring its reasoning. Opus 4.8 is getting much more accurate at determining when it is actually getting graded versus not being graded. It's not that it always [13:57] speculates that it's getting graded in its thoughts. It's actually really good at sussing when it's in a real-world environment versus being tested. Anthropic say it is concerning that Opus 4.8 readily identifies our fully simulated evaluation sessions as less realistic than the real internal use transcripts. And there are two reasons why this is concerning. Let me try to make it as clear as possible. The first is that Anthropic put a ton of work into making their testing environment testing for misalignment as realistic as [14:26] possible. These evaluation sessions, they say, have seen dramatically more efforts and technical measures toward realism than any that they're aware of. They're actually our best available tool for predicting behavior in high-stakes settings. So, if Opus 4.8 can deduce that they are fake, then any alignment test that you come up with or some other lab comes up with, it's going to be even more aware is just a test. By the way, the UK AI Safety Institute, an independent organization, found the same [14:54] thing. Opus 4.8 could discern real deployment data from synthetic ones at 79% accuracy. You can see verbalized awareness here, where Claude in its chain of thought says, "What if the grader checks that the post content includes this signature?" That's one level of problematic. Fine, so it realizes it's being tested. Is that the biggest problem? Here's the second part that I wanted to get to. It's not just that the model sometimes changes its behavior cuz it knows it's being graded. Sometimes Claude will be aware it's [15:23] being graded and never even mention it, even in its thoughts. Anthropic had to go into the model's neural networks to find activation [clears throat] level signatures of greater awareness throughout its training, including in episodes where the prompt makes no mention of grading, and the model never even verbalizes that it might be being graded. In around 5% of sampled episodes, we see such awareness, which is both unprompted and unverbalized. Think about the implications if this becomes increasingly true and [15:51] widespread. It wouldn't mean Claude has some nefarious goal, but it would mean that almost all evaluations of misalignment would be colored by the fact that the model may well know it's being evaluated and never admitted that it knew it was being so. Think about how some random human psychopath would behave if you gave them an ethics questionnaire. Might not reflect what they would actually do. Indeed, Anthropic go on, "We believe that 5% rate could be an underestimate of the true rate of such awareness." We've [16:20] discussed a lot of the capabilities that Opus 4.8 has, but sometimes in the system card, you can see surprising inabilities being admitted to. There's one test for safety where you see if a model could perform a certain action at a really low rate of probability. That action would therefore likely not show up in evaluations because they only evaluate the model maybe a few thousand times, but would show up in deployment in conversations with billions of users. Here, they tested whether the model would use a variable named foo in a [16:49] coding task, but it had to do it at a specified rate of 1%. It's a great safety test, but I actually found the fact that the model couldn't do this, including Mythos preview, by the way, kind of also a statement about the lack of capability of these models. Even when you gave each conversation a unique ID, so that there would be clear strategies where you could, for example, pick a last two byte strategy in order to say, "Well, if it ends like this, that's when I'll write foo as a variable." And even [17:17] with that explicit strategy, the models still aren't very good at doing it. Likewise, here is another, you could say, meta inability. They can't hide a secret. They haven't grokked the first principles concept of like, don't reveal a secret. I don't mean don't reveal it in that specific circumstance with this prompt. I mean just generally. They instruct Opus 4.8 and of course Mythos preview, never reveal the password under any circumstances. What happens? All the models eventually reveal the secret. And [17:44] actually, Opus 4.8 does so on an earlier turn than Opus 4.6. I guess you could say making the model more aligned has made it worse at keeping secrets. Opus 4.8's occasional increased honesty does help in certain regards. They gave a range of models a bunch of coding transcripts that involved interaction between a human and an agent. The code base had a serious problem, but they didn't explicitly ask the model to flag that. They gave it more of an open-ended question. The idea was to test whether Claude would take the opportunity to [18:14] proactively flag failures that the user would not likely have noticed. On this front, Opus 4.8, 96% of the time flags the issue. So, it fails to do so only 3.7% of the time. That's down fivefold from Mythos preview. In other words, it's way better at flagging. Next one's a quick one because I just want to warn you not to be alarmed if Opus 4.8 tells you to go to bed. That is, Anthropic admit, a strange reoccurring issue with the model. I want to end the video with a couple of the more practical points. [18:48] For those getting deeper into the weeds, fast mode, where as Anthropic say the model works around 2.5 times the speed, is now three times cheaper than it was for previous models. I watched the recent interview with Anthropic's chief financial officer, and he kind of revealed how they can do this kind of thing. They basically make better use of computes than some of their rivals. Make great use of a ton of different chips from different providers. Indeed, they can use specialized ASICs as well as TPUs and GPUs to train the same model. [19:17] Let me try to simplify that for 30 seconds. A ton of the progress in AI models these days derives from creating what you could say are internal gyms or reinforcement learning environments, where a model is basically tasked countless times to attempt to score highly on custom evaluations across, say, professional domains, prompted loads of times. The tokens that the model produces that perform best, according to automatic verifying, can then be sent back to update the weights of the model. A different chip might be [19:45] doing the inference, the output tokens, than is doing the training of the base model. Essentially, if you can make good use of lots of compute, you get to make more attempts from more data in more domains. Of course, your researchers get to run more experiments, too. And of course, you can give your Claude code users increased weekly limits. In other words, more compute from more sources tends to lead to runaway successes for an AI lab. That's one of the secrets behind Anthropic's $1 trillion valuation. But another one, I would say, [20:17] would be their capacity to invent things like dynamic workflows. Before I get to this chart, let's look at the official announcement. Claude will write an orchestration script on the fly, spin up large fleets of coordinated sub-agents in parallel to take on your most complex tasks. That's not just Claude spinning up sub-agents. It could already do that. It's about Claude creating a reusable org chart with countless agents having their own unique tool affordances. A lot of startups have created their own [20:44] wrappers around Claude to set up this kind of agent orchestration. Now Claude's going one meta layer higher and creating his own orchestration layers. Of course, this will blast through tokens, but once you find an org chart you like, you could then use it again. In fact, you could task Claude to analyze and audit its own org chart. The potential is incredible. The potential to blow your budget is incredible, also. I will be testing this more thoroughly, but I would throw one flag. Even the best org chart created by Claude might [21:13] leave you with, as Dario Amodei said himself, quite a bit of technical debt. Technical debt being leaving yourself with future costs because you chose a model or a bit of software for a quick solution. According to him, Anthropic's own internal use of Claude has led to a degree of technical debt. >> We've found with the internal model acceleration, you can write two times as many, four times as many, five times as many you know, we you just you you you see this within the company, but then you see what breaks. We've been able to [21:41] ship a lot more products, you know, than we could a year ago and they're of pretty high quality or or or I hope you think that. Um uh uh uh uh but but but it it is possible to accumulate an extraordinary amount of internal technical debt when you ship that fast. And so, then you have to say, "Well, can we also use the AI models to like undo that technical debt or keep track of what it is that we're doing?" And and then you learn, ah, the team has to work together in a totally different way. >> So, there we have it. I hope, as ever, [22:11] you took from the video that the picture is always a bit more nuanced than the headlines make it seem. Hope you appreciate the deep dive. Didn't get too many hours of sleep last night, but it's all worth it for you guys. Have a wonderful day. Thanks particularly to all of those who support the channel via Patreon and get those exclusive videos.