YouTube Deep Summary

Hello. Don't mind my voice. I'm getting over a cold. Look at this spiraling rainbow. And this is not a mod. It's vanilla Minecraft. The structure is being built and rebuilt over and over, frame by frame, by the AI Clawude 3.7 who's standing beneath it. It's essentially an animated building. I'll explain later how this is possible and how actually it was always possible. But first, let's see what my friend Gemini thinks of it. Okay, he's taking a look using the new vision command. And yes, they can now speak automatically using your systems ugly robot voice. This is the new Gemini 2.5 model, by the way. Whoa, cool rainbow thingy behind you. And is that a big tower way over there across the water? Looks like some pink stuff is on the ground, too. We're on some rocks by the water. I don't know what the pink stuff was, but yes, that is indeed a rainbow thingy with a tower and water in the background. The bots can now actually see by processing screenshots of the world. Welcome to an episode of Minecraft. This is an update video, a devlog for the project. I'll be showing off their new visual abilities. There's an official paper for the project, and we'll be doing some so-called vibe coding with several new impressive models. Also, I just reached 200,000 subscribers. Crazy. Even crazier is that a whole six of them are female. Wow. Thanks to everyone who subscribes. You make this stuff possible. And to everyone watching who is not a subscriber, what's up? We got a problem. Why don't we fix the problem? Okay, I've finally updated to the new Minecraft version 1.21.1 specifically. You actually have to use that version for vision to work properly. You must explicitly enable vision in the settings file as well as the texttospech thing. The bots have two new visual commands. look at player and look at position. It can look at specific coordinates or at a specific player or with that player, meaning look in the same direction they're looking. These commands take a simplified screenshot of the world, which looks like this, and asks the language model to describe it in the context of the current conversation. I see a nice grassy plains biome with some flowers around. There's a small wooden house with lanterns and a tall unfinished scaffolding structure in the background. Oh, and there's a cow and a sheep nearby, too. It thought the dog was a sheep. They can make lots of mistakes. You can find these screenshots in the bot's individual folder. Vision is actually an open- source contribution. It was largely written by this guy. So, thank you very much for your work. The image is rendered by something called Prismarine Viewer, which is not perfect. Players always have the Steve skin. The sky is always blue. Items are pink cubes. Blocks below zero don't render. It's buggy. I'll be honest, it throws lots of errors and sometimes causes crashes, but it's a hell of a lot better than nothing, and it's working better than it was in my survival competition video. Internally, the model is asked to summarize and describe the image. It is also automatically given information about the block in the center of its view, the block's type and coordinates, which should help ground the image in the 3D world, so it can reason more spatially about where it's looking at. The image does not stay in context forever. It is fed once to the model and only the generated description is maintained in chat history. This helps cut down on API costs and it can always take another screenshot. Now vision is cool, but it isn't really that useful, especially for survival. You're standing on a diamond or block gemstone shiny choice. Text usually provides more actionable information about the environment. However, it should be helpful for building things since it can actually see structures in the world. Look with me. What do you see? Wow. I see a stunning Gothic cathedral. It's got tall spires and pointed arches with huge stained glass windows that catch the light beautifully. The afternoon sun really highlights those architectural details. This giant impressive cathedral is AI generated, but not by my bots. It was generated with Trippo AI, the sponsor of this video. Trio is a tool that can generate highquality 3D models from any given prompt. Similar to how AI images are generated, it is extremely useful for 3D artists, but can also be directly plugged into Minecraft using the Lightmatica mod. Once you've generated a cool model on Trippo, you can download the file, then upload it to this free website, Object2 schematic, which turns it into Minecraft blocks. Neat. Download the schematic file, move it into your schematics folder, and you can load it into your Minecraft world with the Lightmatica mod. Trippo can drastically speed up your builds and save you time by giving you a great starting point or an end product by itself. And it's just fun to play with. I really enjoy generating weird things. You can make some genuinely amazing massive things with just a few clicks. And it's definitely much easier to use than my bots. Bring your Minecraft world to life with a simple prompt. Try Tripo now. You can support my channel by clicking their link in the pinned comment below. Have fun building. So again, one of the main potential advantages of vision is that it allows the bots to see what they've made, notice any issues or room for improvement, and make adjustments. Let's have Claude 3.7 make a pixel art image, and then take a look at it. She chose to do Mario. It's almost perfect, but it's not that impressive since there are probably a million examples of Mario in the training set. And the face is wrong. Let's see if she can notice that problem. Ah, I see the issue. It's built vertically, which makes it hard to see. Let me rebuild it horizontally so we can view it better. Yes, it's hard to see, but that's because you're too close. It's not because it's built vertically. It's still vertical. Okay, let's just move her back to get a better view. Sure, I'll back up for a better view. Okay, that's a good screenshot. She should be able to tell that there is a problem with the face. Perfect view from here. The colors really pop against the sky and all the details are clear. His red cap, blue overalls, and that classic Mario mustache. What else would you like to see built? And she didn't notice it. What if I tell her to look for issues? Ah, I see two issues. It's floating in the air and Mario looks a bit too wide. Let me fix both. No, no, those are not real issues. Well, let's see if we can get better performance with different models. We'll do a little building with vision competition. We have GPT 4.5, Gemini 2.5, Claude 3.7 thinking. This is a reasoning model and DeepSseek V3. Actually, Deepseek does not have vision, but we can mix and match different models in the same bot for different roles. So, DeepSeek here actually uses their chat model for conversation, their R1 reasoning model for coding, and Gemini's model for image processing. Likewise, Claude is only using the thinking model for coding, not for conversation. Let's give this prompt a shot. Build a statue of a creeper in front of you to the west. Then take a look at it and fix any issues or inaccuracies. This prompt should hopefully make it easier for them to get a good view, but I'm not very confident it will work well. They're already jumping around. Okay, so Claud's is pretty bad, but it has a good view and is trying to fix it. Unfortunately, the viewer has a bug where blocks below zero don't render, so it looks like the builds are floating, which can be confusing. Hey, Geminis is not bad at all, but he didn't actually look at it. Okay, GPTS is not great, but it is trying to look at it. And Deep Seeks is not great [Music] either. Now, look at this. GPT keeps looking at it and can't see the face. So, it thinks there's an issue with the face placement. It just needs to move around to see the other side or build on the side that it's facing, but it seems incapable of realizing that by itself. Vision, I think, has a lot of potential, but it is not a magic bullet. It is extremely hard for all of these models to reason about positioning and angling, which they already struggle with. Adding vision does not automatically resolve all of the issues with building cohesively or acting sequentially or thinking spatially. In fact, it actually seems to be more confusing than helpful. So, here's one final build with no vision, just blind coding as usual. I asked them to make the Hajia Sophia from Istanbul. I mean Constantinople, I mean Bzantium. It's one of my favorite realworld pieces of architecture. For some reason, the claude API died out. So, I'll show another attempt after this. [Music] Geminis's is something else. To say that there's been a leap in performance is a vast understatement. Gemini 2.5 might be my favorite model right now. It is so good, especially at maintaining consistency and minimizing [Music] slop. Look, clear, usable stairways with zero human help. I've seen GPT4.5 do this once before, but it is still super impressive, and it's a beautiful structure. Those final domes kind of ruin it, though. Here is Claude's second attempt, which is also very good. Clawed thinking is the first reasoning model that actually seems pretty creative, unlike OpenAI's reasoning models, which I think are pretty boring. I wanted to give a quick shout out to the Minecraft benchmark project, which basically formalizes this process of judging the building capabilities of different models by letting people vote on the ones they prefer. You can go vote right now. I am not involved in this project. It was originally inspired by Minecraft, but it has since moved on to using its own implementation. It's a pretty unique way of benchmarking the creativity of different models. And speaking of benchmarks, we are working on an official paper for Minecraft to benchmark some of their abilities. I thought it would be published by this video, but it's not quite ready. Turns out it's a lot of work. I will post a link in the description once it's published. Now, I am involved in this one. I'm a co-author, but most of the work was done by some fine folks at UCSD, especially Izzy and Aush. Good work, y'all. The focus of the paper is on the collaborative abilities of multiple agents working together to craft recipes, collect items, or build predefined structures. They've built out a suite of tasks that automatically start up agents with goals and inventories, and constantly checks for task completion. It then evaluates the success rates of different models. Maybe my favorite tasks are these cooking ones, which automatically construct a cute little cooking environment. Different agents then have to talk to each other and work together to combine inventories and ingredients to make whatever it is they're tasked to make. It's so cute and it's fun to watch. This paper officially introduces the Minecraft framework as something that researchers can cite if you want to build off of our work. Okay, to finish off, I want to do some vibe coding, which is a new buzzword that basically means let the AI do all the coding for you and you don't worry about the details. That is essentially how Minecraft has always worked. They write JavaScript programs under the hood that let them perform complex behaviors like building things and you don't have to know anything about coding. But it really struck me recently that this means that they can execute programs step by step in Minecraft in real time, which is much more powerful than I originally realized. So here's a basic example. I've asked them to build a sinewave animation by building, clearing, rebuilding, clearing, rebuilding over and over with a small pause in between each frame. This is actually relatively easy to do. It's a simple mathematical function that just requires a little timing. Oh, look at how Gemini interpreted it. It's squiggling off forever. That is technically what I asked for. Now, GPTs didn't do anything for some reason, but Deepseek got it just fine. We can do more with this, like implementing Conway's game of life on a screen made of blocks. This is a famous cellular automaton that can be implemented with a simple piece of code. Or rather than updating the same screen every frame, you can stack each frame on top of the previous one, creating a 3D game of life that grows upward endlessly. Only the living cells show. The dead cells are just air. You can even build fully functioning games. Gemini made this game of Snake with automatic pathf finding. The fact that you can simulate a screen with blocks, I think, opens up some major possibilities. It did eventually trap itself. A game of snake is a pretty basic, boring thing to build with AI, but what about a 3D game of snake complete with 3D pathfinding? Something about this just blows my mind. And it was always possible, though probably very difficult to do with older models. I bet there is so much more you can do with this that we all have yet to imagine. Okay, well that's it for today. Thanks to my wonderful patrons. I believe I will focus on another large scale survival experiment for my next Minecraft video, which will be quite an undertaking. So, be patient with that. I'll see you later. [Music]