YouTube Deep Summary

[00:00] this is a Roomba the iconic Robot vacuum [00:03] that usually ends up in the back of the [00:05] closet collecting dust or being sold on [00:07] Marketplace for $20 but there's so much [00:10] potential in these things Motors sensors [00:12] a whole robot just waiting to be hacked [00:14] what if we could make it go faster see [00:17] through its eyes and reprogram the whole [00:19] thing using a $7 Raspberry Pi everyone's [00:22] building apps and websites and they all [00:24] kind of look and feel the same and [00:26] everyone's using the same AI tools and [00:29] having AI generate most of the code but [00:31] with Hardware it's different you can't [00:33] just prompt your way around this stuff [00:35] you've really got to get in there and [00:37] break things and figure out how it works [00:40] you could hack your coffee machine so [00:42] that you can text it to brew you coffee [00:44] or control an RC car through a web [00:46] browser or just take apart a RBA because [00:48] some guy on the internet said it was a [00:50] good idea because that's where the real [00:51] fun is coding something different this [00:54] is the Rumba motherboard that controls [00:57] everything on the Rumba and it's plugged [00:59] into the the speaker and the charger and [01:02] a whole bunch of infrared and liar and [01:06] other sensors but we are going to [01:07] completely unplug this from everything [01:10] and get rid of it because we are going [01:12] to use a Raspberry Pi Pico and control [01:16] everything using python at least that's [01:19] the plan good thing we have no one [01:20] underneath us on the other side of the [01:22] rber if we look underneath well first [01:25] there's the battery and I really should [01:26] have taken this thing out first and then [01:29] we can see the these four Motors there's [01:31] the two motors for the wheels there's [01:34] this brush motor thing and then there's [01:36] the main motor that actually controls [01:38] this thing vacuuming and the first thing [01:40] I want to do is try and see if we can [01:42] control these motors so here is one of [01:45] the wheel Motors and if we take a look [01:48] inside of this if you can look past the [01:50] dirt you can see there's a couple of [01:52] wires here that detect whether the wheel [01:54] is actually on the ground or not there's [01:56] a few wires coming out of the motor [01:58] itself to tell how much the wheel has [02:01] actually turned and we can figure out [02:03] how to work with all of these sensors a [02:04] little bit later on but for now all we [02:08] need are these red and black wires that [02:11] are coming out of the motor because if [02:12] we connect just these two wires to a [02:15] battery then the motor will turn and if [02:19] I flip around the battery then the motor [02:21] turns in the opposite direction and this [02:24] battery is around 4 volts so it spinning [02:26] kind of slow but if I take four of these [02:28] and link them up in series then I now [02:31] have a 16vt battery and we get a motor [02:33] that turns way faster so Motors aren't [02:36] that difficult to control and that's [02:38] where we're going to start make a remote [02:40] control vacuum just by controlling these [02:42] four Motors we're going to be using the [02:44] raspberry pi peo to try and control the [02:47] RBA because it is a $7 microcontroller [02:51] really cheap that supports micropython [02:53] and it has all of these pins on the [02:55] outside these gpio pins that means that [02:58] we can write some code that that [03:00] interacts with things in the real world [03:02] it supports C and C++ and some other [03:04] languages but if I Google programming a [03:07] Raspberry Pi Pico Raspberry Pi has a [03:10] decent amount of documentation on how to [03:12] get started just writing some basic [03:13] python to control things so we'll start [03:15] simple I have this Pico already set up [03:18] with an LED plugged into pin 17 I should [03:21] be able to run just a little bit of [03:23] python code to set up my LED on pin 17 [03:28] which is where I've got it plugged in [03:29] right now if I toggle this pin now it [03:31] should light up that led and if I toggle [03:33] it again yeah every time I toggle it on [03:36] it's sending 3.3 volts through that led to light it up and then when I toggle it again it just turns it off and we're basically going to do the exact same thing but instead of a light we're going to make a motor turn and I also have this python script here it's going to loop around and it's going to increase the brightness of the LED gradually and then decrease it back down so if we run this we should see yeah you can kind of yeah you can definitely see that the LED is getting bright and then it's going back down and getting dimmer so again we can use this exact same technique to increase and decrease the speed of the motors but we can't just plug the motor directly in where this led is because the Pico isn't going to supply enough power to the motor it doesn't have enough current or voltage to make the motors turn so instead what we're going to do is we're going to add in this battery pack we could use the rumber battery pack if it still worked but mine was completely dead so I'm just going to use this custom one for now but we take that battery pack and we plug it into this motor controller and then we can take one of the motors plug in the red and black wires of the motor and this thing is just a $3 motor controller that is going to act basically like a dimmer switch the Pico can supply that little bit of voltage to this board and it will allow a certain amount of voltage from the battery to the motor depending on what the Pico is telling this board so if I plug one of the motor controller pins Direct L into the same pin that's being used to control that led and then I just need one more pin so that the motor knows which direction to spin around in and I'm going to use pin 18 for that plug in the last battery here and then this motor should turn as long as I remember to plug in the ground connection there we go okay so now as the LED gets brighter the motor will spin faster and then as it gets dimmer the motor slows down so this code is controlling the motor we can control the motors with a little bit of python now the next goal is to be able to control this over Wi-Fi so that I can set the speed wirelessly so I've added a function that will connect the PCO to my home Wi-Fi and then I'm using this micro do python Library which creates a really lightweight HTTP server on the Pico itself so if I run this code again I can see it's going to attempt to connect to the Wi-Fi it should connect and then set up an HTTP server running on the Pico so this is its IP address I could just try WR a curl request to that which is so sick it's just running an HTTP server there I could even visit it in the web browser if I really wanted to there we go there's my Pico web server working on my Wi-Fi but to control the motors I've set up a custom motor class and it takes three pins these two are for the direction and this one is for the speed then I've set this up as a websocket route I'm just expecting a one byte binary integer minus 100 is full speed backwards and positive 100 is full speed forwards so it's going to expect just that tiny bit of data to come in and then it's going to update the speed based on what it gets so I just need to create a websocket client for this so that I can start controlling the Pico and for that I have set up a react tab it just connects to the Pico websocket server and then has some components to send that data up so if we open this in a web browser this is what it currently looks like I got a little toggle here so if I toggle it up and we look at the light and the motor we should be able to see them there we go there's the motor turning and that's 100% and the light also lights up still cuz I got them connected to the same wire and you can see it working in both directions you can see it logging out the values on the server of the pico2 so this is browser controlled rber and it won't turn off I don't know why must have put a bug in there now I just need to clean up the code a little bit and get all four of these motors connected up to these motor drivers this whole part here is the part that picks up all the dirt and that's these brushes that are responsible for it on the back of this thing here right here this silver thing is the motor it's the silver part on all of the pieces that's the actual motor and again we're just looking for these red and black wires that's all we need to control this thing and the way this works is that the silver part is the motor itself that we apply the voltage to and then somewhere in the plastic part there's going to be some gears that then make this part turn with a little bit more power and and then the second wheel motor and again here is the motor that's that part there's the red and black wires somewhere and inside this piece right here there's a whole bunch of Gears that reduces the speed of the wheel but increases the torque which makes it much better for driving and then finally this brush motor which actually doesn't have any wires on it it just has these two pads so I think I'm going to have to solder a red and black wire onto this thing all of the motors have now been attached back to the bottom of the rumber I have all the wires coming out from the bottom to the top here each of these can control two Motors individually so I just need two of them for all four Motors and you can kind of see these red and black wires coming up and plugging into each of those and then the Pico is plugged into each of those controllers individually and I also added this little 5volt converter here so I can attach any battery to this and it will make sure that the Pico just gets the 5 volts that it needs I also added another cell to the battery pack so it's five now so it's a 20 volt battery which is kind of the equivalent of just plugging a drill battery straight into this thing and on the code side of things I've now got these four Motors on the server for the right left the brush and the vacuum and in the web socket code the server is just listening for these speeds to come in and then it's just going to update each of the motors based on the values that it gets and I could control all of this from a react app from a web browser still but I thought it would be much more fun to control all of this from an iPhone app because I can actually connect this to one of the game controllers built for iPhone and then in Swift I can just import this game controller Library where I can just detect these left and right joysticks and then send that data to the Pico websocket server so if I push these back and forth you can see those values these toggles will be for the left and right Motors and then I just have this slider in the middle to control the vacuum speed so now really the only thing left to do is test if this actually all works and I should probably test that on the floor goam bro go ham oh my god oh did you see that the fuse just burn out immediately I I drove it 2 in oh that's so stupid the good news is I had a fuse on the battery in case anything went wrong the bad news is I just immediately blew so let's see what happens if I actually oh it blew again I think I might have something wrong with my circuits I mean it feels a bit feels like the red and black wires are shorting and if they are shorting that would also explain why the fuse is blowing every time but I don't even know how oh that could be shorting them okay I think that's normal I think I fixed it I think I'm a genius I'm a little scared to try it now I think the right motor is okay all right so let's try oh wait it's working oh that is so sick turns so nicely oh my God that's so fast imagine if rers were like this they'd be terrifying that is that is so much fun oh that's so good and with the vacuum on wait oh so loud that is so loud be so fast at cleaning though hell yeah okay this is awesome oh my God I love this now I know there are so many more things I could do to make this a better vacuum and integrate some of the other sensors that I've been ignoring for now but the only thing I want to do right now is attach a camera to the front of the rumber live stream that to the iPhone screen so I can get this firstperson perspective of what the Rumba sees but before I do that let me tell you about this video sponsor because when you're hacking a Rumba it is really important to have good programming and problem solving skills and you can build these skills right now for free using brilliant where you learn by doing with thousands of interactive lessons in math data analysis Ai and programming each lesson is filled with Hands-On problem solving that let you play with the concepts interactively so you learn by doing which makes it fun but will also help you build a better understanding from the ground up and while you're learning brilliant is helping you build these critical thinking skills through problem solving You're Building real knowledge on specific topics and you're becoming a better thinker and Problem Solver and my favorite part is that you can do these lessons whenever you have time get started with just a few minutes every day and try out their mobile app to make that daily habit even easier you can have a lot of fun leveling up in Python while also learning how to build a language model so to try everything brilliant has to offer for free for a full 30 days visit brilliant.org Sam meard or scan the QR code on screen there's also a link in the description and you'll get 20% off an annual premium subscription it looks like the Pico isn't really designed to work with cameras so there's not much selection but I found this camera right here it's an ov2640 which is a really standard camera to use with a microcontroller but it's $26 just for this module so way more expensive than the Pico and that's because the sensor right here the camera itself is a fairly cheap piece of Hardware I think you can get them for like $5 on alexpress but to get it to work effectively with the Pico they have to add all this other expensive stuff but I've got one right here and we just need to plug it into the Pico and see how this thing works here is the camera plugged in taking up eight of those Pico pins and we can quickly test this out by taking a picture on the camera and serving it up over an HTTP request so I got some camera code from the camera's web it is just a big python class that handles setting up the camera and everything I'm setting it up to take pictures at 640 by 480 pixels and then I have this HTTP Endo so when we go to/ jpeg it's going to take a single jpeg image from that camera and it's going to return it as a HTTP response so if I run this code right now we should just be able to take this IP address and go to/ JPEG to see a picture oh all right that's what working so I'm going to hold this up a little bit higher in front of my face and if I just keep refreshing I'm going to keep getting a new J frame so all right this is kind of like really crappy video but that was pretty easy to get set up now what I really want to do is instead of having it be a JPEG each time I really need a video stream so if we look back at the code I have a different endpoint here that is slash stream and this is just going to be an mjpeg stream so I can still just visit it in a web browser and essentially instead of just sending back a single image it's going to continuously keep taking images and send that down as an HTTP stream to the browser so if I go to slash stream okay so there's the video stream and it is really laggy that frame rate is awful absolutely awful hold on if I just focus it's kind of okay all right and that's really just due to the limitations of the technology we're using so if I go back into the code and I reduce this size so instead of 640x 480 we're going to go with 320x 240 so it's going to be much smaller images but this should give us a much better stream yeah there we go okay so that's that's a little better it's uh just need to reduce that size so that's pretty decent we're getting we're getting a stream we're just streaming jpeg frames so this is what it looks like on the iPhone I am getting that Real Time video feed if I hold it up a little bit you can kind of see my face there yeah coming through on the iPhone screen so overall not bad at all and I have these LEDs on the Pico right here to kind of simulate the motors for now so we can see that I can stream this realtime video and I'd be able to control the motors at the same time so that all still works I'm getting this little bit of a live stream and I mean it's mostly it's working that is that is working but we can do better and this is an ESP 32 S3 which is about the same price as the PCO but this board is set up a little bit better for doing things like video streaming it has more RAM it's got a faster processor so to get a bit of a better video stream I'm going to use one of these and see what the performance difference is like and I went and got this version of the esp32 S3 it's from Seed Studio it costs $14 and comes with the same camera that we're using over here it's the same sensor module but this entire thing costs $14 so we've got a cheaper setup with a more powerful processor and we could stick with micro python for this but I find it incredibly easy to use C and C++ with the PP IDF so I've Rewritten the app and it does basically the same thing but instead of 100 lines of python it's like 2,000 lines of c and C++ so for the camera setup I'm using SVGA which is 800x 600 so more than double the size of the image we were getting with the Raspberry Pi PCO and we're basically doing the same thing where we're going to grab a JPEG frame from the camera on the ESP and then we're going to send that over the web socket connection so that the client receives each jpeg frame as a separate web socket message and I'll post all of this code on GitHub if you want to take a look through the entire thing but let's plug this in and run this application and here it is on the iPhone stream I'm going to hold this up a little bit and that is definitely a crisper shot the frame rate is so much better and we're just all around getting a better video stream in this tiny tiny little package and only $14 and I'm just going to show a side by-side comparison of this video quality difference because the Pico is great but for video streaming clearly here there is a winner this is the Pico camera I've got the camera attached right there with some double-sided tape and I think this is ready to go and for the ESP I didn't want to take the other rumber apart so I just put together another one so it's got the battery pack the same the ESP is at the front there with the camera I got an antenna on the back to get a little bit extra range there and I managed to get all of the electronics inside of the empty compartments here like where the battery used to go so it looks a little bit cleaner too now let's see if they actually work okay I always forget how fast this thing drives that camera feed that is sick that the camera is working but it's incredibly difficult to see what's actually going on when I'm driving and I don't know if I completely stop though like this could be an okay spy cam or something it is sick that I have a live streaming camera on the Roomba but I think I need to try with a better quality wide angle to see what this is like so that's the wide angle camera oh that is so much better that is sick that's actually way better than I thought it would be my God that I can actually drive this thing just using the game controller screen that really exceeded my expectations are you ready for the cleaning test what's that what is that what the is that I hate you where did you get that are you ready I hate you are you ready for no it's okay the rumba's going to clean this up you ready I don't believe you what the hell oh I really hope this oh my God it's glittery you ready vacuum mode can you see the sprinkles I can see yeah like through the camera I can see exactly what I need to clean up I need a reverse camera too so I can see what I've work cleaned up you know like this is cleaning just as well as it did before I hacked it so that's impressive on its own full speed mode with the vacuum oh yeah that is no problem cleaning this up oh hell yeah I don't know why rumo doesn't just put live streaming cameras on all of their robots this is so sick let's see how good a job the RBA did see that is still that is still a decent vacuum that is so sick that's awesome okay you get to clean all this and I really wanted to test out the Pico cam as well but something happened when I plugged it in and a bunch of smoke came out around here and the fuse blew again so this one is officially broken right now but we have to move on because it turns out we can just plug a microcontroller stra straight into the rumba's motherboard using this port instead of having to rip everything out which is great because these motherboards have already been set up to be able to work with all of the motors and the sensors inside of the rumber you can see the port on the motherboard itself and it appears right here on top of the rber and all we have to do is connect three wires into these ports from the microcontroller and now these two can send messages back and forth over these wires using the RBA open interface which is a Serial communication protocol that is really well documented there's commands for everything you can do like controlling the motors getting data on any of the sensors and there are a lot of sensors on the Rumba and you can even program a song into the Rumba using these different notes and then have the Rumba play that song back to you and I'm embarrassed to say that I had no idea this existed until West boss casually mentioned this to me and then made a video on how to control a Rumba using JavaScript where he goes over all of this in way more detail but I've got my microcontroller plugged straight into the rumber the red and black wires go straight to the rumber battery so I can just use that power and then the green yellow and blue wires are all for that communication and I've got them plugged in to my ESP over here and I just made my own little circuit board to clean that up a little bit and I have my voltage converter so it's going to take whatever the rumber battery is giving and convert that to to 5 Vols for the ESP the esp's got that wide angle camera lens on it and then I've still got the little antenna on the back of the rumber but this is definitely looking like the least dangerous version of my rers so far all right you ready mhm do you think this is going to work so this controls pretty nicely without having to hack the whole thing apart it's definitely slower than my custom versions cuz we're not just hooking up a custom battery straight to the motors but with this version because I have ACC to all the sensors on the front I've implemented that video game style crash I can see how much current each motor is drawing and I've have a list of a bunch of other sensors that I want to integrate into this app and there are so many sensors you can work with but my absolute favorite [Music] [Laughter] whoop