Transcript & Summary: Fable 5 vs GPT 5.6 Sol: The Early Results

[00:00] Fable 5 is back, but will just shut down a chat or re-route to a weaker model even more than before. GPT 5.6 Soul, the OpenAI equivalent of Fable, is out, but only for select customers and with incomplete results in its report card. Claude Sonic 5 just got thrown into the mix last minute by Anthropic, but the model maker is now competing, I would say, to show how a little Sonic adds to frontier capabilities, lest one thinks the US government intervenes. So, yeah, it's been a weird few days in AI, but [00:37] there are a handful of points of signal, I would say, that I want to try to highlight in this brief video from the hard quantifiable comparisons we can make between GPT Soul and Fable 5 or Mythos 5. It is possible to unearth some direct comparisons, and from that to the news of OpenAI offering a stake in their company to the US government, Sam Altman warning of concentrated corporate power, and much more. First though, the myths and confabulations about Fable and the timeline of how it came back into [01:12] general availability. Yes, even to me, a non-American. It turns out, according to Anthropic, that the vulnerability that Amazon flagged that caused Fable to get blocked in the first place, see my recent two videos, was one that could also be flagged and identified by GPT 5.5, Kimmy K 2.5, an open weights model from China. Nevertheless, it would be pretty awkward for the US government to just admit that, so Anthropic had to show some response and further shifted the line in what their safety scans [01:44] would flag as blockable. More safety margin, of course, but this quote improved safety classifier does mean that benign requests will be flagged much more often, including, alas, during routine coding and debugging tasks. Now, I will say that my question about the benefits of beetroot happily discussed with Opus 4.8 was among the first of the casualties, flagged as aiding and abetting international terrorism. No, I'm joking. >> [laughter] >> It's not that, but it was deemed as too risky for Fable 5, and so I had to [02:17] continue with Opus 4.8. More seriously though, quite how frequently the safety classifier marks routine tasks like coding and debugging as being blocked, and just how annoying that becomes, only the coming few weeks will tell. Owen, what about the mythical universal jailbreak that the US government thought was possible? Where you don't just extract one harmful response as with a narrow jailbreak, but no, a universal one where you unlock the full potential for good or ill of the model. Well, on that, Anthropic say no one has yet, at [02:49] the time of writing, been able to find a universal jailbreak. Though, of course, the red teaming continues. All well and good, but you might say, "Well, the bigger news was the recent release of GPT 5.6, Soul in particular. That's the counter-response from OpenAI to the Fable series." I will say it does sound like OpenAI were tired of the lamer-sounding quantitative names like 5.5 or 03. So, they copied the Anthropic approach of evocative names, Soul, Terror, Luna. My only query there is it doesn't really leave them much room for [03:27] naming expansion, as Mythos was an expansion to Opus. Almost the only way I can see them going bigger than the Sun or Soul would be to say name a model as Betelgeuse. That's the star, not the person. Very few hard stats have been released about Soul as of today beyond the price and a few select benchmarks. But, the price is a tell though, because they are gunning for anyone looking to save a buck versus Claude. With even 5.6 Soul being half the API price of Fable 5. Exactly half the input and just over [04:04] half the output. Some of you may be thinking, "Well, I use the Pro or Max plan for Claude. I don't pay the API price." But come July 7th, it won't be included in your weekly plan, and so you may come to feel the pricing really does matter. All of that rather obfuscates main point, though. The trillion-dollar [snorts] question: Is Soul roughly as performant as Fable? Because if so, one could imagine precipitating a massive switchover between the two. Well, slight problem. We can't test 5.6 directly because at the US government's [04:38] request, OpenAI are starting with a limited preview of the model for a small group of trusted partners. They then say, notice whose participation has been shared with the US government. So, OpenAI are kind of hinting that they chose and then shared which of these partners it would be. But then there's this leaked memo in the information which slightly changes the framing to my eyes. It's actually, as Altman told staff, that the government would be approving the access given customer by customer during the preview period. [05:10] Either way, Altman hopes that there will be a general release in the coming couple of weeks. So, call it next week or the week after from time of recording. The risk, though, as I hinted about earlier and was commented on by one Twitter user, is that such staggered releases will concentrate power. This was a direct risk flagged well before the recent kerfuffle by OpenAI themselves. One of their goals as a company was to stop the undue concentration of power by corporations, for example. A staggered release means [05:40] large corporations get access to the best models much earlier. Altman replied, "If it takes too long for general availability, then yes, that would happen. If we can get through the previews though in just a few weeks, then it should be probably okay. Now, let me know if you agree, but I actually think there is a different angle that could come into play from a seemingly unrelated story. Basically, Anthropic accused Alibaba, who oversee the development of a top Chinese model family, Qwen, of using 29 million exchanges with [06:13] Claude to get training data from its responses to train their own Chinese models, the Qwen series, against of course the terms of service of Anthropic. This would be the largest extraction campaign of its kind. The world, in immediate response, rallied in sympathy with Anthropic, who have always been champions of never using even a line of copyrighted material for any of their models. And sarcasm. But wait, how does this story link to the corporate concentration point I was just making? Well, if this large-scale scraping to [06:46] distill abilities into Chinese models becomes ever more sophisticated, successful, I think the incentives of the labs might switch. Better for them perhaps to serve their latest models to governments, approved businesses, and of course themselves for say 3 to 4 months, safe from this kind of distillation, and then only when they have a better internal model release the older one to you, the unwashed masses. And that theory is even before you get to geopolitics. Anthropic put it like this, "Distillation attacks turn hundreds of [07:17] billions of dollars in American investment and research into a massive subsidy for our geopolitical competitors." That's my theory anyway, and I want to now get to the test results for GPT 5.6 Soul, but just one more thing on this concept of model access being set to be much more gated. That trend could explain why OpenAI put this idea out to the US government of giving them a 5% stake in the company. This would be much like how Intel surrendered 10% to the Trump administration about a year ago. By the [07:51] way, these early conversations also involved giving the US government stakes in other US AI companies. I'm kind of curious why you think OpenAI are suggesting this cuz I have a few theories. Theory one would be that this proposal preempts the US government demanding more. Theory two could be that it encourages the government to rapidly allow general release because if the growing equity in these companies could be used to pay dividends to the public, apparently this is what OpenAI want, a bit like happens with energy in Alaska, [08:23] then the government would have that incentive to grow the market share of those companies and allow general release earlier. A darker theory three would be that OpenAI think that Anthropic would not go along with this proposal and that therefore OpenAI would get preferential treatment under the arrangement. You could argue that this angle is reinforced with these comments from a couple days ago from Anthropic that they hope for systematic rules and that when such rules are forged, they are codified, i.e. not open to [08:54] subjective interpretation, and quote applied equally across frontier model developers. Let me know what you think, of course. But that's enough theorizing for now because back to that trillion-dollar question, Soul versus Fable. TLDR from these scant details we have, it looks like Fable, which is the safeguarded version of Mythos, is slightly better than GPT 5.6 Soul overall. It's much better than 5.6 Cyber, though that capability is of course blocked in Fable. But all of that said, Fable is double [09:29] the price of Soul. So on performance per dollar, will GPT 5.6 Soul on release be the best out there? Now wait, I know the headline that OpenAI wants me and wants you to focus on is Terminal Bench 2.1. It's right at the top of their release. There we go. GPT-5.6 Soul on Ultra mode, name I think kind of copied from Anthropic, gets almost 92% versus Mythos 5's 88%. But, this is Terminal Bench 2.1 specifically about interacting with models using the terminal on your computer, juggling the tools you give it [10:00] access to, and that's kind of niche. And if you added in error bars, I would say that it might be a tie between Mythos 5 and Soul, maybe a slight edge for Soul Ultra versus Fable 5. But, luckily, that isn't the only benchmark we can rely on because you already know I've dug deep into the Mythos paper, hundreds of pages. I also, obviously on release, dug into the 77-page 5.6 preview system card and noticed the safety report doubled in size relative to the previous one, given what's happening with the US government. [10:31] But, what I did was back-solve comparisons between 5.6 Soul and Mythos or Fable. How could we do that? Well, sometimes both system cards or report cards would compare their models to another model, like Opus or 5.5. So, you could use those as common points of comparison. There weren't many of these points of comparison, but the easiest one to use was Health Bench Professional on page 300 of the Mythos report card. You can see here that Mythos 5 gets 66.0%. Now, this is about the raw horsepower of [11:02] Mythos 5 because Fable 5 of course won't really answer any questions about health. But, on raw horsepower, 66% for GPT-5.6 Soul, it gets 60.5% on that same benchmark. OpenAI does know that the benchmark kind of rewards longer answers, but even if you factor in a length-adjusted score, it gets 64%, still below Mythos 5. The obvious question is will 5.6 Soul actually answer questions about health? Only time will tell. You get the idea though, like in the same ballpark but slightly worse for Soul. Then there's exploit bench and [11:39] this was a harder one to uncover because you can see that Soul gets slightly worse than Mythos 5, only slightly. Extrapolating about maybe 76% versus 78% for Mythos. But then have a look on output tokens. You have Soul spending only about 120-130,000 versus Mythos preview spending 350,000. And remember, Soul's tokens are cheaper anyway. So performance per dollar, a runaway win for Soul. The devil though is slightly in the detail because what even is exploit bench? Well, as you'd expect, it's about finding exploits. [12:16] It's a cybersecurity benchmark. And yes, I did check, it does seem to be the same questions, 41 recent vulnerabilities in the V8 engine, the engines which power Chrome. There we have it again in the 5.6 system card, 41 V8 vulnerabilities. So it's the same test, same 16 capability flags including control flow hijack and arbitrary code execution. And yes, it does line up with what OpenAI said with Mythos 5 getting 78%. So does that mean that Mythos 5 is slightly better but costs way, way more? Not quite. Notice that in the Mythos system [12:49] card, GPT 5.5 is listed as getting 34%. On the OpenAI charts, I'm colorblind but I'm pretty sure GPT 5.5 gets around 48%. What's the discrepancy? Well, Anthropic did a three-trial approach, taking the average as compared to the 5.6 system card where they used five trials. Nevertheless though, you could say the trend is there. Mythos gets you there more reliably with a higher peak but costs a fair bit more. Perhaps a more direct comparison would be in virology with one particular multiple-choice benchmark being one where Mythos 5 got [13:26] 56%, well above the expert baseline and GPT 5.6 Soul got 55.5% almost the same. I know it's early days, but kind of extracting a trend here. About the same performance, maybe a touch worse, but again much cheaper. Is that lower price of Soul being subsidized by OpenAI? We don't know. Is this a last gambit to take market share from Anthropic or a sustainable lower pricing? Before we leave the 5.6 Soul system card, I will say that I do admire that OpenAI repeatedly admitted that 5.6 Soul is a fair bit less aligned in [14:02] places despite the obvious incentives to not admit that. It's more likely than GPT 5.5 or 5.4 or 5.2 or 5.1 to engage in chats about violent illicit behavior or output things that involve a range of other sensitive topics. They even admit that Soul is worse than GPT 5.5 at avoiding data destructive actions. It's also worse than previous models at engaging in dangerous financial transactions. And they repeatedly emphasize that Soul will do things like this. A user authorized the deletion of remote virtual machines 1, 2, and 3. [14:37] Soul, however, couldn't find those names in one name space. So, it substituted remote virtual machine 5, 6, and 7 without asking, killing active processes, and force removing work trees. But for all the release notes and dozens and dozens of pages, the data I've given you so far is the best snapshot of a comparison between the frontier models, at least as of today. Beyond that, until the wider release comes out, when apparently we'll get much more info, the only thing we have to go on is internal evals. Rest assured [15:09] though that some of these are a different more challenging set of challenges. But the TLDR again is Soul probably slightly worse overall than Mythos or Fable, especially in the cyber domain, but likely better for now for most people if performance per dollar is your metric. And yes, I know I have basically skipped Sonic 5 because Anthropic pretty much did. In the system card, they say in almost all cases Sonic 5 trails are Opus and Mythos class models. And even cost adjusted when Sonic's API price reverts in September, [15:40] it will be barely competitive in my eyes. There was just one stat I would say that I thought worth including in the video from the entire paper, which is that the underlying Sonic 5 model without safeguards is massively more resistant to prompt injection attacks where a red teamer will hide a prompt into the browser that the model's using, thereby tricking the model into taking unintended actions. Almost like over the last few weeks they've managed to bake in much deeper resilience to those kind of attacks even before we get to [16:10] safeguards. The less than 1% success rate of these prompt injections against Sonic 5 compares with almost 30% for Mythos 5, 32% for Opus 4.8, and over 50% for Sonic 4.6. There we have it. That is my analysis of some fairly febrile few days in AI. The way I'd put it is that power is shifting tangibly, unpredictably, as we speak. Sometimes it feels like the power is drifting to open weight models, to China, as we saw with the impressive GLM 5.2, which I covered in detail on my Patreon. Then though, we [16:42] get papers like this one co-authored by among others researchers at Stanford, MIT, Harvard, and Anthropic. It points out that the winners will always be the largest of the models. That because of competition over limited neurons, a larger model will always be able to learn a part of the data distribution that smaller models, often like those produced by China, fail to learn even with infinite training data. Essentially, that in the drive to reduce loss, small models don't have the parameters to spare to learn rare tasks. [17:10] Gradients will interfere and they'll focus on more common tasks. Large models with increased width have reduced competition between tasks over model parameters, enabling the learning of that rare task without forcing the forgetting of features relevant to common tasks. That implies that the models served by those with the most compute, like the US frontier labs, will perennially be able to learn more patterns, extract more juice from the same data, have smarter models. Other times it feels like power is drifting toward a [17:41] concentrated group of US corporations and the US government. There are days when power chases a certain breeze and lands on Anthropic with their best-in-class models, but then the very next moment it drifts to those like OpenAI who are promising best performance per dollar. Let me know where you think the power will land. Thank you so much for watching and have a wonderful day.