YouTube Deep Summary

## The Capabilities of Grok 4 [00:00] How impressive is Gro 4 for you. If you look at the AIM benchmark, which is an advanced math quiz, Grock 4 scored 100% on it. You're literally running out of benchmarks. It's got to be driving uh Google nuts that Elon got this done in 28 months from a from a cold start. When he said he was going to put this huge cluster together, every AI expert in the world said, "You cannot get power laws and coherence at that scale. You just can't do it." Every AI expert is like, "Oh, god dang, he did it." The amount of compute and resources again are going exponential. Now, it's the real quality that differentiates the top models between each other. My big question is, where do we go from here. Now, that's a moonshot, ladies and gentlemen. Everybody, welcome to Moonshots. An episode of WTF just happened in tech this week. Special episode today following the release of Gro 4. It is large language model release month. An extraordinary string of new models coming up. I'm here with my moonshot mates Dave Blondon, the head of Link XPV, Salem Ismael, the CEO of Open Exo, and a special guest to help us dissect all of this is Immad Mustach, the founder of Intelligent Internet. Guys, um it was a pretty epic day yesterday. Good to see you all. Pleasure to have you. Yeah, likewise. Yeah. And this is our special Gro 4 edition. Um Immod, you're in London. Yes. Yep. Fantastic. And uh uh and See, where in the planet are you, buddy. Uh New York. Okay. Dave's in Boston. I'm in Santa Monica. All right, let's get going. So, just to jump in. goal here is dissect what happened yesterday blowby-blow what's Grock 4 all about and just to you know shadow what's coming we've got a few new model releases coming with Gemini 3 GPT5 you know and probably a few others so let's kick it off with this video like Grocer is postgraduate like PhD level in everything better than PhD but like most PhDs would fail So it's better said I mean at least with respect to academic questions it I want to just emphasize this point with respect to academic questions Grock 4 is better than PhD level in every subject no exceptions um now this doesn't mean that it's it you know times it may lack common sense and it has not yet invented new technologies or discovered new physics but that is just a matter of time. Mhm. Um if it I I I think it may discover new technologies uh as soon as later this year. Um and I I would be shocked if it has not done so next year. All right, Dave, you want to take the first bite of Yeah, it's awesome. This is actually a golden moment in time because uh it is an absolutely brilliant assistant that can do almost anything you want it to do, but like Elon said, it's not reasoning yet. So it's not coming up with the fundamental this is what we should build and this is why. So that's still in the hands of the creator, the human operator. And so this this moment in time is actually really really golden. It feels just like an Iron Man movie where you've got Jarvis. Jarvis will build the suit for you. You have to decide how you're going to save the world. Uh it's it's a really really fun time to be using these brand new like like you said, there'll be three of these in the next month or so. this is the first round and uh he's dead right you know the PhD level solution it's all measured in the in the benchmarks we'll get into in a minute uh but it it does virtually anything mind-blowing capabilities but it doesn't decide what to do and why uh I would love your take on this you've been plugged into this world you know intimately for a while how impressive is Gro 4 for you I think it is very impressive I think you know picking up what Dave said I I think it is reasoning but it's not planning as yet. And there was a question as when we got to this rona flop level I think that's the term like 10 to 28 I think flops would we continue to see improvements and part of that is the compute and part of that is the data as we'll get to later and the answer is yes and again like Elon said getting above graduate level in every sub postgraduate level in every subject it can now execute and it can reason it doesn't have planning yet. So, I mean, isn't isn't that AGI. Isn't that the sort of like kind of definition of AGI. We've been we passed through the touring test without noticing. Are we going to pass through AGI without noticing, too. It's like this hydonic adaptation. You're like, of course, it's fine. You know, but already again, if you want to get a job done, it will do the job for you of summarizing a book. Like, it will do the job for you of like writing a summary of something or translating, etc. And life is just the same so far because you haven't got that final step that Dave said and there's a few extra bits that we need for full aergentic above that. But we're nearly there because we have that final building block now with this next level of model. Yeah. Where it's reliable distinction by the way that it's it is reasoning. It has to be to solve these really hard PhD level pro problems but it's not planning. That's a great way to phrase it. Uh run a flop is 10 to the 27th. So that's the scale of these algorithms. That that was the that was the level the AI act said they wanted to ban by the way. So this would be the first ban model. Yeah, that's a great point. First I think I think one of the thing that's happening is the absolute beauty of capitalism where you've got big juggernaut companies fighting it out for supremacy and throwing taking massive risks um choosing design paths taking huge gamles and really really going for it. I think it's really ma ma magical to watch this happening. Yeah, I love this tweet from Sawyer Merritt. It says XAI was founded in March of 2023. Just 28 months later, it's now the number one model in the world, verified by independent testing. Incredible achievement. I mean, it is insanely fast compared to everything else that's being built. I remember when uh in May two years ago when Elon was first raising money and I had a chance to sit in on a investor pitch in the first round for XAI and he said I'm going to have 100,000 GPUs H100s operating by the by the end of the summer and everybody's like no no freaking way. Uh and he did just that. Um and he's not slowed down. So here we see in this image artificial analysis intelligence index gro 3 was placing like fifth or sixth gro 4 leaps to the front of the line uh are we going to continue seeing this emod you know this just leaprogging each other leaprogging each other is there is there no end in sight it's getting very difficult because if you look at the benchmarks they have there if you look at the aime benchmark which is an advanced math quiz Grock 4 scored 100% on it. I mean, so you're literally running out of benchmarks uh in order to do that. ## Humanity's Last Exam and AI Performance [07:39] And the amount of compute and resources again are going exponential uh because you need to to squeeze that out as well as have good data as well as have good algorithms. So before you could just chuck everything into a pot, slush it around. Now it's the real quality that differentiates the top models between each other. And it's become more of an engineering and quality challenge than just a brute force challenge. Insane. Can I please So what for a second, please. Um, okay. So, I've got a problem. I I would suggest that if I'm trying to answer that problem or get a solution to it, I could go to any of these and they're going to give me marginally the roughly the same answer. Yes. So, we're at a point where the the new step is I'd love to I want to get into the details of Grock to figure out why is it so radically different from any of the others, right. And I that's where I think the fun will come. Every week I study the 10 major tech meta trends that will transform industries over the decade ahead. I cover trends ranging from humanoid robots, AGI, quantum computing, transport, energy, longevity, and more. No fluff, only the important stuff that matters, that impacts our lives and our careers. If you want me to share these with you, I write a newsletter twice a week, sending it out as a short two-minute read via email. And if you want to discover the most important metatrens 10 years before anyone else, these reports are for you. Readers include founders and CEOs from the world's most disruptive companies and entrepreneurs building the world's most disruptive companies. It's not for you if you don't want to be informed of what's coming, why it matters, and how you can benefit from it. To subscribe for free, go to dmmanis.com/tatrends. That's dmandis.com/tatrends to gain access to trends 10 plus years before anyone else. Well, the funny thing is we're using, you know, we're basically going to Einstein, you know, and asking him to summarize a uh a poem for us. I mean, it's like there's such massive level intelligence and the utilization for the general public is is dimminimous. All right, let's look at what's next on this. Uh, so Grock outperforms the uh the highest level test, humanity's last exam. Uh, up until now, we've seen uh, see 03 was at 21%, Grock 4 was at 25.4%, Gemini 2.5 at 26.9%. And then Gro 4 and then Gro 4 heavy comes in at 44.4%. Uh we were talking about this a little bit earlier you know can you speak to humanity's last exam for us. Yeah this was um come up by scale AI and kind of a few others um to have an exam that even the most polymathic people in the world would find difficult. So they estimated that like some of the smartest people in the world would score maybe 5% on it maximum 10% and the top models at the time which was probably like half a year ago 9 months ago scored 8%. Now you have a qualitative leap above to that 44% level. And I think it's interesting because as kind of Salem was referring to like what are these models for. They're at this super genius level. It's like having a mega liberal arts program. And then the next step is going to be to have really useful people in the workforce on one stream and then the other stream will be to take the subcomponents of this and just push up to superhuman reasoning discovering new things at a level that we could never be have before. And I think this is one of the indications of that cuz again I I tried to read some of the questions I didn't even understand the questions. Examples I I literally just gave a presentation on this yesterday so I have it right in front of me. Tell me humanity's last exam 2700 questions. When the slide says for reference humans can score 5%. That means the very best humans in any given domain can score 5% within just the domain they understand. And I'll tell you why. Like here's here's an example question. Compute the reduced 12thdimensional spin boardism of the classifying space of the lie group G2. And then it goes on from there. Most people can't even understand one word of that. Exactly. Here's another one. Take a five-dimensional gravitational theory compactifified on a circle down to a fourthdimensional vacuum. So, yeah, these are the hardest questions and and that's why this exam is supposed to last for a long time. a 44% score is just way outside the range of human ability because nobody has that broad knowledge that spans all this all these topics. So ho how far how long before we hit 100% here too any bets. Uh two years max I would say probably next year. So you know there was a conversation years ago about AI getting to a point where um you can't understand the questions it's asking and answering. Uh, and we're not far from that. So, I mean, we're unable to actually at some point we're unable to measure how rapidly it's advancing. That becomes a little bit frightening. It's got to be driving uh Google nuts that that Elon got this done in 28 months from a from a cold start. Absolutely. largely because, you know, Elon is phenomenal at large-scale manufacturing, large-scale organizational management and and, you know, people working four or five a.m. sleeping in tents on the on the factory floor. That's that's his wheelhouse. And that's Tesla, that's SpaceX, and and because all the intellectual property was more or less open sourced by the research community at Google and Meta, he was able to pick up all that brilliant thinking and just plow it into implementation. It's also small teams, right. It's not large. I mean, Google's a massive organization. Yeah. I think there's something else here, though. Remember, we talked about this last time when Grock 3 came out, right. But when he said he was going to put this huge cluster together, every AI expert in the world said, "You cannot get power laws and coherence at that scale. You just can't do it." And he went right back to first principles, created new kind of connections between the chips and whatever, and did it. And every AI expert is like, "Oh, god dang, he did it." And so this is the this is the incredible ability he has to go into a domain with a beginner's mind, go to first principles, and just re-engineer the heck out of it to achieve massive performance. And I think um uh this is an indication of that. My big question is is as you mentioned earlier, Dave, where do we go from here, right. like what what what does it mean to have a 50% versus 44% on this test. Yeah. Yeah. I think if I can just give it a little bit of context, in 2022, Amazon built us the 10th fastest public supercomputer in the world. 4,000 A100s, you know, 2022. And that was 2022. That was 10th fastest in the world. Wow. Of any supercomputer that we were training on. And there was an instance where literally hundreds of the chips melted because of the scaling. Now they've managed to by turning this into engineering problem scale the hardware but also the inside of the model which I think is this really important thing. The reason it's above PhD level in each of these areas is that was a computation scale problem. And so what happens is that if you could scale a liberal arts person all the way up to postgrad in everything you would. And then you specialize down and then you look at some of these things. And Sem's question there, you've got just for reference everybody, it's uh the XAI cluster now has 340,000 uh GPUs. Just about $30,000 or more each. Yeah, do the math. 10 billion. A lot. I mean, this is why, you know, this is why we're seeing a billion dollars a day going into AI and why Jensen said, you know, there'll be a trillion dollars a year by 2030. Uh, and it's not slowing down. So, here's another image from the uh from the little conversation Elon had yesterday. These are the benchmarks his team put up. Um, I don't know if you want to hit on any of these, Immod or or Dave or Salem. Any of the favorites for you. Well, my favorite one is the AM AIME25 100%. You're done. You know, GPQA. These are all hard benchmarks. I think Elon want I think Elon would want to go to uh 110%. He likes 11 as 11. But the only one I don't recognize is on the bottom right. Do you know what that is. The USA 25. I think it's the um USA Mathematical Olympiad. Oh, right. Of course. So, it's about to happen. Um but but again like these are novel hard benchmarks effectively all of them and they're being saturated because ultimately the AI can reason mathematics and science better than we can. Again it can't plan just yet. It doesn't have the. ## Advancements in AI Training and Cost Dynamics [16:47] same memory capacity and the building blocks haven't been put together. But it's already superhuman narrow capability in many narrow areas. So it's inevitable I think what happens next. You know, we glossed over his quote there. Discover new physics. Uh, wouldn't surprise me if it's this year, certainly no later than the end of next year. Uh, Alex Wisner Gross has been having a field day with that all day, I bet. First of all, what does it mean to discover new physics. That's that's pretty interesting by itself. Well, I mean, you know, Alex Alex has been saying we're going to solve all of math and then physics comes next. Chemistry and biology follow quickly. I mean, this is the most exciting for me. This is the most exciting thing of these models are will they literally unwrap the president of the universe before us right here right now during our lives in the next 5 or 10 years. Well, there's a couple of specific applications that I think uh I've been watching. I want I want to I want to see an AI break and solve the quandry of the wave particle duality of light. That would be interesting and seeing what exactly is going on in this. Uh the second one would be molecular manufacturing and how do we new techniques for doing molecular because you crack that then you crack all assembly and manufacturing of all kinds right and everything the cost of anything becomes about a dollar a pound per weight a computer a dollar a pound yeah now you're in an amazing I mean listen again going back to Ray Kerszswwell's predictions right uh how he does it I still you know you know he's mentored you he's mentored me But, you know, these predictions that we're going to have nanotech in the early 2030s, uh, where is it. Where is it. Well, this is probably its parents. Yeah. Well, the one the one that's really fun to think about, you know, the quantum teleportation, Peter, that you brought up at one of our enterprise meetings. So, how do you reconcile the fact that two entangled particles can be infinitely far apart yet still communicating in real time with the fact that the speed of light can't be transcended. So, so Alex's speculation is if we can solve physics in the next year or two or three and it turns out that you can communicate using quantum teleportation that we instantly discover all these other intelligences around the universe. Yeah, we've just been listening at the wrong frequency with the with the wrong codecs. Uh these are the key takeaways. I'm going to just read these out loud and we can talk about them. They spent just as much on fine-tuning training the AI after initial phase as they did on pre-training. So that's a big change. Iman, you want to detect that for us. Yeah. So it used to be that everything was basically you take a snapshot of the internet and then you put it into this giant supercomputer mixer and it figures out all the connections, the latent spaces to guess the next word. Then you had this very weird AI that came out that was a little bit crazy. It's like a disheveled graduate student without his coffee and then you had to tidy him up with the reinforcement learning. That was the post training and that was 1% of the compute. Then with Deepseek it was 10% of the compute and now it's moved to equal because they figured out how to chain reasoning strips. And in fact I think part of what they did we've seen this with other labs is they used their frontier model to make data for the next frontier model. So having large amounts of compute to create your own training data in a structured manner allows you to take that latent space the landscape and make it smarter and smarter and smarter just like your brain adapts as you learn more and more reasoning as you see more and more things. And so rather than having to have these massive scrapes of the internet or whatever, it's more and more structured data making up these models which are making them smarter reasoners. So the the 50% additional compute dedicated to uh to the the finetuning does that mean we have a more sane version of Grock. Fingers crossed. Um, it doesn't necessarily mean that because you can still get all sorts of mode collapse within it in terms of if the latent space goes, but probably um because again you're training it just on a certain field of things as opposed to Reddit and other things. In terms of order, I'd say this is probably like a hund00 million each. So it probably adds up to one Meta AI researcher. You know, a new a new a new unit of measure in the AI world. That's funny. So, let's comment on the cost here. $3 per million tokens. Um, $15 per million output tokens and can handle long context windows of 56,000 tokens. How does that measure up, Dave, in your mind. Uh, well, it's pretty normal these days. It's it's a longer context. You know, a lot of the claimed context windows aren't real. Under the covers, the dimension of the neural net is much smaller than the claimed context window. Um, so I suspect, you know, at this scale that this is the true dimension of the network, but I don't really know. We'll have to dig in over the next couple of days and and find out. But, you know, what it means is, you know, you can feed in a 100 books worth of information concurrently. It instantly digests all that knowledge and then gives you an intelligent answer based on all of that information in one pass. So, it's just it's just, you know, the next step in what's been going up sequentially from model to model to model. Iman, do you expect we're going to be con constantly reducing the uh the price per token. Is this a is this a demonetizing curve for a while to come. 100%. I mean, so the cost of this is about the same as the cost of Claude for Sonnet, which is the second model of of anthropic or 03's cost, but it's better than both. Uh it's about 0.7 words per token to give you an idea. And so the cost of a million very good words that are smart is $20. Mhm. But next year with Vera Rubin, the next generation chip they're going to whack in there. Just by the hardware, it'll be three times to four times cheaper and they'll probably figure out some more stuff around that. So Equi Intelligence, the cost probably drops by around five to 10 times a year. So it'll be a buck for a million amazing words. It's hard to believe the most powerful technology in the world is dimminimous in cost. It's crazy. I want to I want to I want to put a a comparator though here. Um you know we we this is amazing. like we could put hundreds of our books into the thing and it would hold all of that in real time as as Dave said, but let's note that a single human cell has several billion operations going on in it at at any time point in time, right. So, we're kind of several orders, multiple orders of magnitude from modeling one cell. Uh, and so we're we've got a long way to go to try and model life or get to really big big big things. There's a coming wave of technological convergence as AI, robots, and other exponential tech transform every company and industry. And in its wake, no job or career will be left untouched. The people who going to win in the coming era won't be the strongest. It won't even be the smartest. It'll be the people who are fastest to spot trends and to adapt. A few weeks ago, I took everything I teach to executive teams about navigating disruption, spotting exponential trends a decade out, and put them into a course designed for one purpose, to futureproof your life, your career, and your company. ## AI in Medicine: Augmentation vs. Replacement [24:36] against this coming surge of AI, humanoids, and exponential tech. I'm giving the first lesson out for free. You can access this first lesson and more at dmandis.com/futureproof. That's dmandis.com/futureproof. The link is below. Let's talk about Super Gro Heavy. You know, I gotta love Elon's terminology, right. It's we we've got Falcon Heavy, now we've got Super Gro Heavy. Um, he loves his terms and I love them, too, actually. It makes me I smiled when I saw that. Why heavy, by the way. Is there a name reason for that. Falcons, the Elonverse. Yeah. No, I mean, like, you know, Falcon Heavy was able to have, you know, three boosters to launch a heavier payload to orbit. So why not why not uh talk about heavier capacity. So I mean uh in in reality right Falcon Heavy had multiple boosters and this has multiple agents. So super next one will be heavier and the one that will have to next next one will be Grock Starship. It'll be it'll be it'll be BFG BFG. Yes. So the price point here sets a new high bar. Uh that's going to scare a lot of people. Um, I I say the same thing I said last time. You know, try it. Burn the 300 bucks for one month. You can turn off the subscription, but you got to try it to know what you're what you're missing or not missing. A lot of the use cases, you know, the day-to-day use cases, it won't matter much. But if you're building something complicated, writing code, uh, or or designing mechanical parts or whatever, you're going to get addicted to it. What I'm really curious about is the margin at 300 bucks a month. Are they actually chewing up all that money on compute for you or do they have significant margin at that price point. Because one thing I've been predicting for a long time, it's inevitably going to happen soon is the use cases where you need that extra intelligence. Like when you're when you're building a software product and you're prompting it, you absolutely need that extra level of intelligence. It makes you dramatically more efficient in moving forward. And if you look at the cost of an engine software engineer's time, you can afford to go up another factor of 10 or or even more in price point for this and still be glad that you paid it. And so I think the escalation of pricing is is going to come soon. The counterargument is that the competing models will then commoditize it. But I think people will pay a lot for marginally better improvement because the the effective product you get out the other side. It it really accelerates your time to development or the quality of the design or whatever the solution to the math problem you is right rather than wrong. ## The Future of Gaming and the Metaverse [27:16] Makes a big difference. My guess is they're losing money. You think so. That's what that's what OpenAI said for their pro level whereas the level below they make money. So I think the way that I view this is a loss leader because if someone's paying 300 bucks, you enterprise sell them up. Mhm. And then you do team things to get everyone doing it because basically right now what we have is a UI problem. The reasoner is there. The way to hook it up and make it usable for as many people on your team isn't there. You know, this is what Andre Carpathy calls context engineering. You know, like what are the new UIs that will enable us to use this most efficiently and get our data in there. If you can crack that, then 300 bucks a month for a high level knowledge worker is nothing. Yeah. You know, zero, right. Just like we used to pay a,000 2,000 bucks a month for Bloomberg when I was a hedge fund manager mostly for instant messaging. But, you know, like again, it's just not quite there, but it's about to flip there. Yeah. Well, like a lawyer will cost you that much per hour or or three, five times that per hour. Will this do the job of your legal document better. I I can't wait. That's the one profession I would love to replace is lawyers. All right. Uh you you mentioned enterprise level uh emod let's go there right now. What else can GU do. So we're actually releasing this GUG if you want to try uh right now to evaluate run the same benchmark as us. Uh it's on API um has 256k contact length. So we already actually see some of the early early adopters to try guac for API. So uh our polo neighbor ARC Institute which is a leading uh biomedical research uh center is already using seeing like how can they automate their research flows with gro. Uh it turned out it performs is able to help the scientists to sniff through you know millions of experiments logs and then you know just like pick the best hypothesis within a split of seconds. uh we see this is being used for their like the crisper uh research and also uh you know Grog four independently evaluated scores as the best model to exam the chess x-ray uh who would know um and uh uh on in the financial sector we also see you know the graph for with access tools realtime information is actually one of the most popular AIs out there so uh you know our graph is also going to be available on the hyperscalers so the XAI enterprise sector is only, you know, started two months ago and we're open for business. Open for business. So, Iman, you've been working on medical related um AI. Uh it's, you know, the block here isn't the tech. It's going to be the regulations. It's going to be when will an AI be able to fully replace a radiologist or fully replace a um you know, any profession of. ## Hollywood vs. Video Games: The Battle for Attention [30:10] in the medical world. How you think about that. Well, I think it's the augmentation first. Reduce errors, increase outcomes, and then eventually it's replacement because Google had their AI medical expert study which showed that it was doctor doctor plus Google search doctor plus AI and then AI by itself. Yeah. But just do self-driving cars just I want I just want to touch on that because it was a really important article that came out. uh if you again the physician by themselves was getting something like 80% of the cases correct the centaur the physician plus the AI was getting like 87% the numbers are approximate and then the AI without the human bias without the human biasing the output the AI by itself was outdoing all of them at like the early 90%. uh extraordinary well again it's what you said it's better than any postgrad at the moment but right now I think it's about the empowering and the acceleration in terms of the integration and you're way off the liability profile of replacement I don't think you need replacement right now what we need is less errors in something like medicine right I I think the doctor number by itself Peter was 70% because I remember Daniel Craft saying when you go to the doctor you get the wrong diagnosis about 30% of the time, right. That's a staggering number of errors, by the way. That means out of four of us, one and a half got the wrong diagnosis the last time we went to the doctor. I mean, we need to figure out who that was. Uh, that's really ridiculous. And so, you need an AI to take over that whole field. Well, human bias and getting human bias out of that is also even more important as we can. The number of types of scans and sensors you can do is way way outstripping any human ability to look at all the data that comes out of it. So, a lot of a lot of it isn't trying to beat a doctor. It's trying to assimilate data that never could have gotten into the diagnosis before. That's a great point. That's a great point. Yeah. Just All right. Let's go on to Let's go on to our next uh next one. Uh so, available uh for an API. All right. Uh we've covered these areas already. Let's move on. A quick aside, you've probably heard me speaking about fountain life before and you're probably wishing, "Peter, would you please stop talking about fountain life?" And the answer is no, I won't because genuinely we're living through a healthcare crisis. You may not know this, but 70% of heart attacks have no precedent, no pain, no shortness of breath. And half of those people with a heart attack never wake up. You don't feel cancer until stage three or stage 4, until it's too late. But we have all the technology required to detect and prevent these diseases early at scale. That's why a group of us including Tony Robbins, Bill Cap, and Bob Heruri founded Fountain Life, a one-stop center to help people understand what's going on inside their bodies before it's too late and to gain access to the therapeutics to give them decades of extra health span. Learn more about what's going on inside your body from Fountain Life. Go to fountainlife.com/per and tell them Peter sent you. Okay, back to the episode. All right. Uh, I love this. You know, Elon is a gamer and so it's not unreasonable for him to be talking about using Grock to make games. Take a listen. Yeah. So, uh, the other thing, uh, we talked a lot about, you know, having Grock to make games, uh, video games. Uh, so Denny is actually a, uh, video game designers on X. So uh you know we mentioned hey who want to try out some uh uh gro for uh preview APIs uh to make games and then he answered the call. Uh so this was actually just made first person shooting game in a span of four hours. Uh so uh some of the actually the unappreciated hardest problem of making video games is not necessarily encoding the core logic of the game but actually go out source all the assets all the textures of files and and uh you know to create a visually appealing game. I think one of the challenges is what we do with all of our time in the future and we may be playing a lot of video games. You know, this could actually light up. ## The Evolution of Coding with AI [34:20] the entire metaverse world because building the metaverse world and building those environments was the big limiting factor and now you can do it at a very rich level. This could be really interesting to see what comes from this. Yeah. When did you guys first hear that Gro 4 was going to come out last night. Well, he said a few days ago, didn't he. I mean, a week ago. I mean, he was saying it was going to be this weekend and then it got pushed to to yesterday. Yeah. Because I feel like we had about 48 hour notice plus or minus a day or two. But it was amazing the if you look at the presentation, the raw presentation from last night and compare it to Google IO. Google IO was was scripted and staged with multiple presenters and, you know, clearly planned way in advance. Uh this last night was like, is it done yet, guys. Is it done. Does it work. Okay. If it works, we're launching tonight. Let's go. Get on stage. Let's go. And and I think that's the way it's going to be in the future because uh you know, it seems like getting to market one day, two days sooner actually matters a lot in this horse race. So this is kind of the dynamic we should expect going forward. But by the way, that narrator, that's the AI voice of a geek who is living and breathing it. And that's what you want in there. That's what you want. All right, let's let's take a listen uh on Elon on video games and and movie production for example for for video games you'd want to use, you know, Unreal Engine or Unity or one of the one of the the main graphics engines um and then gen generate the generate the art uh apply it to a 3D model uh and then create an executable that someone can run on a PC or or a console or or a phone. um like we we expect that to happen probably this year. Um and if not this year, certainly next year. U so that's uh it's going to be wild. I would expect the first really good AI video game to be next year. Um, and probably the first uh half hour of watchable TV this year and probably the first watchable AI movie next year. Yeah, it's amazing with the the fragmentation of those industries is going to be incredible because, you know, normally we think of a video game coming out in a release, all of your friends get the exact same release. It's a release that's maybe good for a year or more and you're all on like FIFA 23 now or whatever 25. Um, but here because it's only four hours to create the next. ## Advancements in Video Model Training [37:00] iteration, then you can say, well, no, I want a customized version or I want there's going to be all this fragmentation and the version of the movie that I saw isn't the same ending that the one that Sem saw. So now we're debating on how it we're not even on the same page and how the movie ends because we saw a different a different AI generated version and it's going to be great. It's going to be it's going to be really really cool because everything's we're gonna have a lot to do with our time. I mean I listen you spent so much time uh as CEO of Stability in this market arena of entertainment and video production and such. Uh when I asked you earlier whether Hollywood is you know going to be disrupted you said no. Um can you can you explain that please. So I think the thing that won't grow is people's attention. So if you look at Netflix, their biggest competitor is video games, which is why they're going into video games. You only have so many hours in a day and you're a consumer. Video game sector right now, I think, is $450 billion. The movie sector is 70 billion. That's how fast it's grown. Like education around the world is like 10 times larger. So it's 10% of education in terms of size. So if you think about that, then for Hollywood Studios, this is great because the costs have coming down and it's been a dramatic shift. To give you an idea, the first video models, stable video I think was pretty much the first. We released that in 2023. And now with V3 from Google and others, you're pretty much a Hollywood level, close to it, but you need one more generation to get there. And the average Hollywood click length is 2.5 seconds. It used to be 12 seconds. Now it's 2.5. And we can generate eight. And soon we'll be able to generate more. So you're getting to this point where you can make that. But again, people like having common stories to talk about Barbie Oenheimer and things like that. So these marquee things, they can get the license of Carrie Grant from back in the day and make him a star again. you know, don't you think don't you think that uh there's going to be so much supply and if I have a chance to watch, you know, a new episode of classic Star Trek, but you know, I'm the character playing Captain Kirk and uh and you know, you're playing Spock and my friends are taking the roles. I I mean it I don't know why I would not be buying that entertainment uh from a source other than you know outside of Hollywood. Well, you'll buy that too, but I think one of the things we've seen in the AI world, what's it about. Distribution, distribution, distribution. So, you'll buy your interactive games and put yourself in the game, but you'll still have your marquee things and the cost of that will reduce dramatically and the distribution cost will decrease dramatically and the impact will increase. So again, for companies, this is all great. For the individuals working in the industry, this is terrible. And so I think this is the key thing. For the individual creators, this is great because you can finally tell the stories. So we'll see richer stories, but you've still got to distribute them. It's like one of the examples I had to give is, you know, Taylor Swift, bless her heart, it's not the best music in the world, but she still causes earthquakes, you know. Yeah. Yeah. No. Uh your your point that uh I think the video game industry bypassed all other media combined. Uh I think I read that and it's on a much faster growth trajectory as well. But I think the video games are far more compelling with AI components, AI players, AI voices, voices that are talking directly to you. Uh and so that interactive media is going to get even more accelerated by this trend. So I whether you call it movies or video games or other the media is going to change, right. It always does. So it may not fit exactly in those swim lanes, but it's clearly the interactive talk to me part is going to grow much much faster than passive watching part. Yeah, I think it's the quality part and it's the feedback for you to find flow. So the movie industry's grown from like six 50 billion to 60 billion in the last 10 years. Average IMDb score 6.3. Video game industry is like doubled inside, quadrupled. It was 170 billion, now it's like 500 billion. The average score has gone from 69% on Metacritic to 74%. Games are good now and you need to be good to compete. And again, I think what we can see from this technology is I as a creator can create the best things better because I can control every pixel. This is what Jensen has said. Every pixel will be generated exactly what's in your mind. maybe you know you have to use a keyboard it just comes straight from your mind can be on that screen you can tell the stories you want and on the other side you've got the fast food so you know the general content farms get even better so you got your gourmet and you've got your fast food and both of the quality of those will increase right every day I get the strangest compliment someone will stop me and say Peter you have such nice skin honestly I never thought I'd hear that from anyone and honestly I can't take the full credit all I do is use something called onskin OS1 twice a day every day. The company is built by four brilliant PhD women. ## The Future of AI: Grok 5 and Beyond [42:05] who've identified a peptide that effectively reverses the age of your skin. I love it and again I use this twice a day every day. You can go to onskin.co and write peter at checkout for a discount on the same product I use. That's oneskin.co and use the code peter at checkout. All right, back to the episode. Uh, of course, Grock for coding. Let's take a quick listen. Right. So if you think about what are the applications out there that can really benefit from all those very intelligent, fast and smart models and coding is actually one of them. Yeah. So the team is currently working very heavily on coding models. Um I think uh right now the main focus is we actually trained recently a specialized coding model which is going to be both fast and smart. Um and I believe we can share with that model with you with all of you uh in a few weeks. Yeah. Yeah. I still remember Immod when you were on stage with me uh like three years ago at the abundance summit and you said no more coders in five years and it was it was front page throughout India. I I got I got hate mail about that you know. Oh my god. You scared the daylights out of and and it's true. I mean there's I mean it's a big issue. It's a big issue. Why would you be able to talk to a computer better than a computer can talk to a computer. Yeah, you know, well, hold on. Let me drill into that just for a second. Don't you think we'll end up with really good coders just creating 100 times more code. No. Because what you'll have is really good context engineers directing to build things. code is an intermediate step of language because the computers and the compilers couldn't handle the complexity of what we wanted to talk about. Now you can talk to the AI all day long about anything and it understands to a reasonable degree what you actually want and once we get the feedback loops really going as we've seen with cursor and other things like that like there's a reason it's got to $500 million in revenue in a year you know there's a reason that anthropics got to $4 billion probably twothirds of that is code. Mhm. Yeah. Crazy. All right. Disappointing that we won't have this for a couple weeks. We'll have to get back on the pod and and check it out when it's out. Somebody told me you can get to it through cursor right now. I'm looking at cursor as we speak and I don't see it popping up as a as an option. But cursor is very much linked towards anthropic. So it probably like lobomize it. But Grock 3, Grock 4 already heavy is a pretty good code. It writes clean code and the coding model I think will be even better. But again, how much better are you going to get when you can. ## The Race for GPU Supply and AI Development [44:39] output a 3D video game like that or just about anything. And I think this comes to think, are you if you're trying to create content, the AI is good enough already for just about anything. If you're trying to create something creative, this is the final part that requires planning and coordination and multi- aent systems and the UIUX isn't there yet for the feedback loops, etc. Yeah. I can use all the horsepower they can give me though cuz like when you're writing a little code module it's all pretty much perfect already. But right now I can go to the best claude model and say build me a dashboard for this function and just give it that prompt and most of the time it comes back great and even thinks of things that I wouldn't have thought of for that dashboard and I can use another step up of capability in that area. So I'll use it up as quickly as it comes out. Believe me. all the tokens to Dave. Okay, let's hear from Elon about uh his video model training. What's coming on input output. We expect to be training our video model with uh over 100,000 GB200s uh and uh to begin that training within the next three or four weeks. So, we're we're confident it's going to be pretty spectacular in video generation and video understanding. So 100,000 GB 200 uh more than anybody's thrown at this uh what is that how does that how does that hit you. So when we trained the state-of-the-art first video model two years ago, two years ago, that's right. We use 700 700 700 H H100s. So like uh let's say they're three times slower. So the equivalent of 200 of the chips that he's about to use cuz these are the integrated GB chips from um Nvidia. The top level models right now, if you look at the Lumas of the world, the bite dance models of the world, the V3s, use 2 to 4,000. Wow. He's about to use a 100,000 of those. And the thing about video is when you train a video model, it actually learns a representation of the world through computation. So once we made a video model, we extended it to a 3D model that could generate any 3D asset. It understands physics and more. So actually video models are world models that can be used to do all sorts of things like improve self-driving cars by creating whole worlds and other things like that as well. I think that's the reason why given they've got 300,000 chips, they're putting a 100,000 of these to their video model. Well, and they're planning a million GPUs by the end of this year. You know, let's you know, it's like it's like no small dreams here. Mod, when you pioneered this just a couple years ago, like you said, um the video model was trained completely separate from the large language model because, you know, it was just too much. You couldn't put everything into one mega model. Is he going to do um a monster retraining of this model with video data or is it a separate set of parameters and a separate model. This will be a separate model. So, we took the image model and then we created the video model from that and then we created the 3D model from that. Now they're doing from scratch training because the technology we developed for stable diffusion 3, the diffusion transformer matching it is able to do that all at once. And this is similar to what V3 and others use. And with optimizations, you can just pop that all straight in. Now the arch that they use, like the Grock model for the image, is actually the same architecture as for the language. And they may do the same thing. I'm not sure how they're going to train this model cuz again they're super smart. But it's a different model entirely. But they may all end up being the same model because if you want a model that understands physics and the wonders of the universe and what's the question to get to the answer 42, you probably want to train on everything that a human sees and more because it'll train on everything a million humans can see and understand and read and all sorts of stuff. I mean, you know, I'm excited about the idea of there's so many of my favorite science fiction books that have never been made into movies or TV series, right. I mean, the ability to just say, "Hey, uh, you know, like one of my favorite books is, uh, the Bobverse series, uh, by Dennis Taylor. Uh, and you know, I love it. It's a four book series. It's it's extraordinary. Make it into a movie for me. Make it into a 20 part TV series for me. Um, here's a hundred bucks. 100 bucks. Really fun actually if you took took the the best books that have ever been turned into movies already and use that as training data. So like this book turned into this killer movie. Make the changes necessary to get from point A to point B. Okay, now here's a book that never got made into a movie. From what you learned about those patterns, make the movie that's most compelling. The thing is you won't even have to do that. like just with the pace of chip improvements as we go through the generations in 2 years you will have live 4K TV so you've already seen some people do like live low resolution stuff interactive stuff when Jensen says every pixel will be generated he literally means it like with the next generation chips and a bit of more improvement in the algorithms and optimization of the models you can have live streaming 3D or video where every single pixel is generated on your screen within a few years. And so you can just say, "Stop, try this, adjust this, and that'll be the feedback loop." It'd be fun to take some old movies and and make them way better. Like take the old Kona and the Barbarian movie and make it really a proper movie. Oh my god. You know what hits me. We're sitting here having this conversation in four different cities around the world where, you know, we've taken so much for granted in this video channel and like, you know, 10 years ago, what do we have. We had just barely had Skype. Um, and now, you know, it's it's crazy. So, we humans adapt so rapidly to awesomeness and we take it we take it from we normalize it very fast. It's like your second Whimo motor ride, right. Yeah. Your first one's like, "Wow." And your second one was like, "Okay." Oh, for sure. So, any any closing thoughts on Gro. I have a question. I have a question for Emod. Uh, you've been in the space for a while now. We have Gro 4, right. What are the types of things that Grock 5 will be able to do. So Grock 5 will be a multi-agentic system, but rather than having four boosters, it'll have 60 or 600 or 6,000 depending on what you want. It'll probably have a world model plugged in and it'll have interconnectivity, and this is something that Elon mentioned yesterday, to every major type of system. So it knows how to use Maya, it knows how to use advanced physics simulators, it will write its own lean code and optimize it for mathematics. And so it's just going to be like an incredibly versatile worker. And just like he's going to unleash millions of Optimus robots, he's going to unleash billions if not trillions of these things, GPU demand withstanding into the economy and that's going to be a bit crazy. And I think the way that you'll interact with Gro 6, probably Grot 5, is you'll have a Zoom call with it just like you have now. Mhm. Hey folks, Lim here. Hope you're enjoying these podcasts and this one in particular was amazing. Um, if you want to hear more from me or get involved in our EXO ecosystem on the 23rd of July, we're doing a once a month workshop. Tickets are $100. Uh, we limit it to a few people to make sure it's intermittent and proper. And we go through the exo model. What we do there is we basically show you how to take your organization and turn it into one of these hypers growth AI type companies. And we've done this now for 10 years with thousands of companies. Uh many of these use the model that we have called the exponential organizations model. Peter and I co-authored the second edition a couple of years ago. So it's a 100 bucks June July 23rd. Come along. It's the best $100 you'll spend. Link is below. See you there. Uh Gemini 3 and uh and GPT5. Let's talk one second about what you expect there. Are these going to just leapfrog Gro 4. Are they going to be, you know, sort of diverting in different directions. Immad, your thoughts. I think they'll probably all be kind of the same plateau. Now, it's really about the UIUX and then how you wrap these into agents and then multi- aent systems and then how you make it so just easy for anyone to use like this. So, you know, Google in the work that they've done with their AR glasses, um, you know, enabling you to have a conversation with your AI and being able to have it see what you see. That's a great step forward. uh you know open AI uh with their their voice their voice mode has been fantastic. Uh are there any versions of I you know user interface that we haven't seen yet. I mean BCI will be one of them for sure. I mean I personally think again the interface is just the interface that you have with a remote worker and all the technology is almost in place for that like get on a call hit him a slack pretty much and you just don't know. That's my AGI. My AGI is actually more like actually useful intelligence, right. Like this is I think probably what Selem would like just I don't know it's an AI or not. It just gets the job done and it doesn't sleep. And this final part of it as well is that the task length of these AIs has gone to like 7 hours now. I think I've seen from various entities now they're getting that up to almost arbitrary length. So you can set teams away and they have organizing AIS and others. They get the job done. They check in whenever they're unsure about something. And then this is that next step up for all these technologies. But I think the 10 to the 27 models will, as you said, all be pretty much similar cuz they're already above PhD and everything. Now it's about making them super useful and getting them out there. And the demand for that is in the billions of agents. Dave, you know what I find interesting is Elon's got basically a limitless capital supply. Yeah, you know, it's every time he's gone to raise money, you know, I've asked, well, how much can I get in the next round and it's like, well, we're over subscribed already. Yeah. Yeah. No, the constraint isn't going to be the money. It's going to be the the GPUs. I have a question for you, Mad, about that actually because if you say, okay, the the you know, GBD5 will be out soon, couple weeks hopefully. It'll be on the same plane, probably leak frog, but in the same genre and then Gemini 3 will come out and it'll be somewhere similar, maybe a little better. Um, but the chip supply, you know, Google has huge amounts of GPU and and a massive cloud computing platform, plus they make their own TPUs. Then, um, you know, you got a million chips going to Elon. We just talked about that Sam at OpenAI has had a little bit of trouble with Microsoft recently. There's there's definitely some kind of falling out there. And the way OpenAI got ahead of everyone in the first place is getting access to the compute from Microsoft. And so is he going to have a problem getting catching up to a million concurrent GPUs training a single massive model. I mean, I think Stargate is in that order of magnitude when you look at the kind of gigawatts and now Amazon's just announced poor Anthropic using Tranium for something that's even bigger than Stargate with their latest kind of trip supply. Google's the leader in this. So, they have 3 million odd. But the thing that I come back to is Open AI basically slowed down when everyone was making Giblly memes. And so if you think about order of compute of Giblly memes compared to order of compute for useful work like it's that versus that right Google is okay because Google are actually landing millions of their own TPUs and they have the full stack and it has better interconnect for large context length. It's actually really good 7th generation hardware. Elon will get the supply because he's a beast. And I think again OpenAI have the capital, but they're moving more and more towards consumer with the Johnny IV acquisition and things like that. The dark horse here is probably again meta to be honest because Zuck is going to drop a hundred billion. Yeah. On this. He dropped 30 billion on the glasses on the metaverse. He thinks AGI is coming and Meta is a $1.7 trillion stock. will easily drop a h 100red billion. Yeah, he's got $70 billion of free cash right now to to use and can pump it up. Well, I I did an interview of Yan Lun at MIT not super long ago and they had committed and already bought a million GPUs for internal use at Meta. So, he he had those on order already then. I'm sure they're in house now. So, he has the the compute inhouse. So, basically all the top guys can get a million. The next step is 10 million. Well, there's only 20 million in the world. This is where it runs into a bottle. You can't even keep a straight face, can you. Well, well, but again, think about every pixel being generated and think about again the economic activity of actually having a single useful teammate or account. I mean, we're talking about like accountants and lawyers and other things like that on the other side of the screen. We're not even talking about super genius PhDs. Is Nvidia just going to just keep going going going. Is anybody going to displace their their production at all. uh all of the top chip manufacturers are good enough to run these models. The only question is who has enough gating supply. So the reason for the hopper thing was actually the packaging of the chips. You know the co-ops. So you have different supply channel constraints just like robots. In two years robots will be good enough to do what 90% 95% of human labor. The only reason the entire global economy on labor isn't going to flip over from do $2 robots is supply chains. So what we've got is a complete replacement of the capital stock of the economy from GPUs for virtual workers and robots and it's just supply constraint. So Nvidia number one, you don't go wrong. You don't get fired getting Nvidia, but you'll get chips from wherever you can get them because those chips are orders of magnitude cheaper than your team members. H I just asked actually um Gemini in the background here what it costs today's market rate to train a Ronoflop. So one of these models just to compute cost is 312 million. So like you said Ahmad it's it's like one signing bonus over at OpenAI these days. So that's not the cost is not the issue. It's it's who has access to the compute. What's amazing to me in this entire conversation we haven't said the word Apple once. Yeah. and and Apple controls about a third of the manufacturing capacity at TSMC for their M3 line, M2 line chips. So, they could easily become a player in the the get a big data center up and running game. They'd have an incredible asset having that manufacturing towhold with TSMC. It's just incredible that they haven't done that. Well, I think this comes down to the thing. These models have economies of scope in that once you train a model that's good enough do you really need another one and then it becomes like electricity it becomes a utility so your genius models become utilities and then what matters is the model that runs on the M3 or whatever you know like liquid AI just releasing edge models those things become even more important because the M3 has capac M4s have capacity yeah that's a really big deal. By the way, liquid is uh I didn't appreciate how big a deal it was until recently, but um people are going to want to use this stuff immediately. I mean, it's so addictive and the inference time compute is severely constrained. Uh and liquid, you know, runs fine on the edge on these M3s. It runs really, really fast. It runs on the chips in the cars and it's about, you know, they say about 100 times more efficient than just trying to run a brute force transformer. So that could be a huge unlock for people having access to AI, you know, at least more access to keep up with the demand. Exactly. Because you'll have your gated stuff and then they might increase prices because they have to because there'll be so much competition for chips even as you get them cheaper. And then you just got this AI with you, but that AI will be smart enough to do your day-to-day. And so you'll have a whole curve of intelligence just like sometimes you need to have steady workers and sometimes you need your geniuses. I forgot you were the you were actually the first guy to to see liquid when it was just a research project. Yeah, I I gave them all the compute to get going. Yeah, that's right. That was amazing. And now they're what $2 billion valuation. So, listen, when you come back and join us next week, I think we have it scheduled. I want to hear all about the intelligent internet. I'd love you to break the news on what you've been working on in secret for the last, you know, year or so. Uh I I've seen pieces of it. It's awesome. But hopefully you'll you'll spill the whole master plan for us. Uh Dave, Salem, my Moonshot mates, thank you guys. Uh Gro 4 special edition. See you at Grock 5. Yeah, we got Gemini 3 in like three weeks. We'll be back online soon. All right, see you all. Thank you for joining us. Take care, folks. Bye, guys. Bye. If you could have had a 10year head start on the dot boom back in the 2000s, would you have taken it. Every week I track the major tech meta trends. These are massive game-changing shifts that will play out over the decade ahead. From humanoid robotics to AGI, quantum computing, energy breakthroughs and longevity. I cut through the noise and deliver only what matters to our lives and our careers. I send out a Metatron newsletter twice a week as a quick two-minute readover email. It's entirely free. These insights are read by founders, CEOs, and investors behind some of the world's most disruptive companies. Why. Because acting early is everything. This is for you if you want to see the future before it arrives and profit from it. Sign up at dmmadness.com/tatrends and be ahead of the next tech bubble. That's dmmadness.com/tatrends. [Music].