YouTube Deep Summary

## Intro [00:00] We've been using AI tools for a long time in engineering. I believe we were the first company outside of GitHub to use GitHub Copilot. And the reason Yeah. The reason I know that is because when Thomas DMK became the CEO of GitHub, the same day I emailed him saying I would like GitHub Copilot. This is 2021, a year before Chat GBT. He messaged me back saying, "It's not available for commercial use." And I said, "That's not what I asked. I don't care what you have for commercial use. I would like this deployed for all Shopify engineers as soon as humanly possible." I think it took him about a month. And we were not charged for 2 years because there was not a skew to charge us. And we said in exchange we'll give you lots and lots of feedback. And so we were using copilot for a long time and then we worked with them as we started looking at their road map. We then deployed cursor internally as well. Actually we're very new to cursor about a year into cursor. So we kind of trying all these things and as we try them we want to see what's working not working and then we let more engineers use it. The most interesting thing about cursor is that the growth in cursor at Shopify is happening a lot outside of engineering and outside of R&D finance sales support those are the teams using cursor. What happens when a company goes allin on AI? Shopify did exactly this a few years ago and Shopify's head of engineering, Farhan Tavar, told me exactly how it's going so far. In this episode, we discuss how Shopify works closely with AI labs and why Farhand paired for an hour with an engineer at Antrophic working on the cloud code team. Why Shopify is planning to hire 10,000 interns in a year and how they incorporated AI into their entry process. why Shopify has no cost limit on how much an engineer or team can spend on AI tokens and many more. If you're interested to know what an AI first tech company operates, like this episode is for you. This podcast was recorded as a live podcast at LDX3 in London. If you enjoy the podcast, please do subscribe on your favorite podcast player and on YouTube. So, welcome everyone to this very special live podcast with Farhan Tower and we're going to talk about AI at Shopify. But before we start about AI, I just wanted to talk a little bit about you, Farhan. So I did a little bit of research. I talked with a couple of engineers at at at Shopify and they told me, you know, your role is the head of engineering and engineers told me that they see you in everything. You have reworked Shopify's ## Shopify’s philosophy: “hire smart people and pair with them on problems” [02:07] intern hiring program. You've defined what Shopify does. In fact, you even were in the weeds to deploying an internal hackfest to more than to getting the Wi-Fi right for a hackfest. So I'd like to know what does a head of engineering do at Shopify and specifically what do you do or maybe what do you not do? Yeah. So I think what's interesting about Shopify is that we use this line. We're not a swim lane company which means that we don't try to put people into these roles where you are like you're in product so only think about product or you're in engineering only think about like how the code is written or architecture. We're very much just curious problem solvers and so something's broken we expect uh curious people to go and look at the problem even doesn't matter what their role is and try to solve it and so last year at our uh Shopify summit which is our employee event where you know 7,000 people come to one space the Wi-Fi did not work super well and so I spent all my time in that summit for three days trying to fix the Wi-Fi I became known as the chief Wi-Fi officer and so this year which was it was two weeks ago um I deployed uh 300 00 ubiquity uh APS, 800 switches in order to have access points. Yeah. To have a much much smoother Wi-Fi experience, which is how uh that all got started in terms of and they the team built lots and lots of memes because they thought the Wi-Fi was not going to work, but it did work. And so I did publish all those memes. Uh I I I need to ask you because a lot of leaders would say, look, like you know, your time is super valuable. You're kind of ahead of an organization. How many people are 3,000? 3,000 people. you're kind of up there and your time is now, you know, if you measure it in dollars, it'll be expensive, right? It's a lot better to just get a specialist to do these things. Why do you still do these things? And do you disagree with with that advice of like focus on on the high leverage? Yeah. So, we don't love the there's that notion of like hire smart people and get out of their way. Instead, what we say is like hire smart people and pair with them on problems. M so the difference for us is we really like to bring in obviously lots of smart people hands-on people and instead of just saying hey you take this area and then go away and just solve it for me and come back we like to pair with them and say you're smart I'm smart can we work on this problem together this episode is brought to you by work OS if you're building a SAS app at some point your customers will start asking for enterprise features like SL authentication skim provisioning and fine grain authorization that's where work OS comes in making it fast and painless to add enterprise features to your app their APIs are easy to understand and you can ship quickly and get back to building other features. Work OS also provides a free user management solution called OKIT for up to 1 million monthly active users. It's a drop in replacement for Ozero and comes standard with useful features like domain verification, rolebased access control, bot protection and MFA. It's powered by Radics components which means zero compromises in design. You get limitless customizations as well as modular templates designed for quick integrations. Today, hundreds of fast growing startups are powered by work OS, including ones you probably know like Cursor, Versel, and Perplexity. Check it out at work.com to learn more. That is works.com. If you want to build a great product, you have to ship quickly. But how do you know what works? More importantly, how do you avoid shipping things that don't work? The answer, Statig. Static is a unified platform for flags, analytics, experiments, and more. combining five plus products into a single platform with a unified set of data. Here's how it works. First, static helps you ship a feature with a feature flag or config. Then, it measures how it's working from alerts and errors to replays of people using that feature to measurement of topline impact. Then, you get your analytics, user account metrics, and dashboards to track your progress over time, all linked to the stuff you ship. Even better, Static is incredibly affordable with a super generous free tier. a starter program with $50,000 of free credits and custom plans to help you consolidate your existing spend on flags, analytics, or AB testing tools. To get started, go to statsick.com/pragmatic. That is statsig.com/pragmatic. Happy building. So this leads really nice into the next thing I heard about you which is at Shopify you are have gone kind of AI first and we're going to talk a little bit about that but one thing that you do is you have access to ## How Shopify works with top AI labs [06:22] some of the top AI labs and what you told me just before that you pair with them. You just told me that the other day you were pairing with anthropic engineers. Can you tell me about this whole pairing and how you work with these AI labs? Yeah, because again we're like super curious problem solvers and so when there's something interesting happening in the industry, we want to be close to it. So obviously chatbt comes out and then Anthropics builds their model, Gemini has their model, Coher has their model, we want to be as close to the leading edge as possible and so we try to be close to those people and what they're working on. And one of the things we do is when something comes out, so this in this example it was cloud code. We deployed inside of Shopify. We saw people using this was in May, right? Three or four weeks ago. Exactly. Well, so cloud code came out before that like let's say 6 months ago and then we started spending time with anthropic. I wanted to see how Anthropic was using cloud code internally similar to like OpenAI is now releasing codecs. I want to figure out how they're using it internally. And so what I did was I reached out to Anthropic and said, "Hey, I'd like to pair with one of your applied AI engineers. We spent an hour building a bunch of things and I learned how he sees uh cloud code being used internally at Enthropic." Then I can take that back and say, "Well, here's how we're using it at Shopify and we can together figure out where we think this thing can go." Is this just a matter of like you just asked them? Yeah. I mean, because we are Shopify, we're very lucky. People want to see how we work. We tend to work in an unorthodox way and they also are interested in how we use it, but they also use it. And so, it was more of a me again, a pairing. It was a true pairing. It wasn't like me as a customer trying to push them and say, "I really want." It was we have a shared Slack channel. I peding them there saying, "Hey, who'd want to pair with me for an hour on something I'm building inside Shopify?" Somebody put their hand up, an applied AI engineer, and said, "We'll show you how we work and we show you how how we work and we can together figure out um where we think this product could go." So, I do that with a lot of the providers. And as you're pairing, is it just you pairing and then you later tell people or like do you Yeah, I might I might record the session and then share it with the internally to say, "Hey, I just did a pairing session with Anthropic." They might use it for a case study that they're building. But the idea is again because I'm just curious when somebody says something I'm like, "Oh, how do you how does that work? How does that uh how can we use this?" And so I think it's a way for us to just learn. And one one recent thing that I I just wanted to to go into before we get into some more of the AI things. It's a really fun fact that most people don't know. You just came out of a code red for seven months. Uh I think this is the the first time we're building. What was this code red? Uh what was about and and what did people outside and inside of Shopify see of it? Yeah. So we were uh internally seeing a lot of things that weren't bubbling to ## The recent Code Red at Shopify [08:50] the surface, but that can actually affect the operation of a system. So you would typically call it tech debt, but we had a lot of signals showing that the tech debt was growing, not just the general ones like uh taking longer to update a piece of software or build a version two of a product or people having a hard time uh maintaining some of the software stack. We saw things even not externally but internally that were shooting out signals lots of exceptions uh growing even in some cases because we build things at the very low level of the stack. So for example we patch we're core contributors to Ruby. We patch my SQL I believe we have the second largest MySQL fleet in the world outside of meta. Um and we were seeing things like seg faults and things like that that I was like not willing to feel like we should move on and just build features as normal. And so we actually got together spent uh we didn't think it was going to be 7 months we thought it was going to maybe be 3 months. And we took uh something between like 30 to 50% of engineering. We didn't have an actual number. We just said these things have to not grow anymore. Like exceptions, unique exception counts have to shrink. There should be zero seg faults. We should understand everything that's happening in the system. And so we spent almost 7 months uh doing that over the last uh you know from November till now. How did you make sure that you fixed the right thing? Did you like talk to the seam saying hey bring up the things? And and also how did you decide when you will stop? Because like whenever I hear these code reds, it's always, you know, a lot of companies do code yellows, code reds, but sometimes they can drag on forever. Yeah. So we had m multiple metrics that we used. One was literally what I said, exception counts, unique exception counts and seg faults, which would literally would say like seg faults are at zero now and exceptions should be going down. But we also looked at other metrics that we have uh like you know we have four we want to have four lines of reliability across our different surfaces storefronts merchant admin uh point of sale like we looked at all these things and we also used the 28day rolling average of those things and once those were all green and we saw that uh again seg unique exception stopped and was shrinking we then felt like we're in a place where we can start building features again and now going on ## How Shopify became early users of GitHub Copilot and their pivot to trying multiple tools [10:47] to to AI so you're a huge early adopter of of AI coding tools tools. Can you tell me on what tools you started to use and what you're using right now and what what what your what you're liking, what engineers are are liking specifically like so focus more on software engineer but maybe even outside of a software. Yeah. So we we've been using AI tools for a long time in in engineering. So we have we were the I believe we were the first company outside of GitHub to use GitHub copilot. And the reason yeah the reason I know that is because when Thomas DMK became the CEO of GitHub the same day I emailed him saying I would like GitHub copilot. This is 2021, a year before Chat GBT. Um, he messaged me back saying, "It's not available for commercial use." And I said, "That's not what I asked. I don't care what you have for commercial use. I would like this deployed for all Shopify engineers as soon as uh humanly possible." I think it took him about a month and we were not charged, I think, for 2 years because there was not a skew to charge us. And we said, "In exchange, we'll give you lots and lots of feedback." And so we were using Copilot for a long time. And then we worked with them as we started looking at their road map. We then deployed cursor internally as well. Actually, we're very new to cursor about a year. Really? Um, yeah, they were a tiny startup back then. Yeah. So, yeah, I I think that's super late. You're saying it's super early. I feel like it's very late to cursor. We knew about cursor for a long time. We like to have one tool at Shopify for the things that we're doing. And so, we don't like to have multiple tools. So, like we have like Figma. We don't try to have lots of other uh visualization tools. We have MySQL. We don't have other databases. So, we try to focus on one tool. And so, our our bet was VS Code. But because of AI's proliferation and we don't know what's going to happen, we then started making a change in our stance around trying more tools than just one. Maybe we consolidate, maybe we don't. But right now we have cursor and VS code as our two AI tools. And then cloud code came as the agentic workflow and we started using cloud code. We tried Devon last year. So we kind of trying all these things and as we try them we want to see what's working, what's not working and then we let more engineers use it. The most interesting thing about cursor is that the growth in cursor at Shopify is happening a lot outside of engineering and outside of R&D finance ## The surprising ways non-engineering teams at Shopify are using Cursor [12:49] sales support. Those are the teams using cursor. What do they use it for? So what's happening is they're because cursor is actually so ubiquitous on uh building things. They're using it to build MCP servers outside of regular uh engineering domain. So for example, Salesforce, Google Calendar, Gmail, Slack, they're using they're building MCP uh access MCP servers to access those services and then building like homepages for themselves. Like you're a salesperson, connect to Salesforce, connect to Google Calendar, connect to my email, and tell me what what opportunity I should be working on. So with just going back with with an MCP server, when you have a service, you can put an MCP server and then you can access it with for example with cursor or with with an agent. Exactly. Do engineers help build that MCP server? Sometimes yes, but a lot of times no. Now that these services are coming out with their own MCP agent so that you can literally just download GitHub's MCP server, right? You don't have to actually build it yourself. But a lot of times they're actually just building it themselves and they're watching like videos and they're actually just deploying it on their own without any engineer intervention at all. Now it's it's it's end of one, right? They're building software for each individual person, not like infrastructure for all people. And does this remind you of anything? You've been around in the tech industry for a long time, but the fact that you know the non-technical people are now kind of building that does does it do you see any parallels from from the past of did did this happen before or is this is this new? this coding part is new although if you remember like like wissywig editors and like people starting to like selfs serve I think it it it reeks of that but they are I mean they're not reading the code right they're really much vibe coding and if it doesn't work they just delete and start over but even if you read anthropics documentation on cloud code like they say onethird of the time you can kind of get something working with one shot and so we're starting to see that and people are getting used to the fact that if it doesn't work they just delete and start over and and try to get it working I I find this really interesting on on how this is happening. Does is it change your dynamics with engineering? Because one thing I noticed when you know working at places like Uber the everyone was a bit jealous of engineering like they wanted engineering resources. They couldn't get it. They really want one of these things. Do you see any of this change? I'm going to say ## Why you have to understand code to submit a PR at Shopify [14:53] yes and no. So there's a weird thing happening whereby so let's say you're a PM. Unless you're a technical PM. You now probably have enough knowledge to vibe code some new feature you've been waiting for engineering to build for you yourself. The question is, can you submit a PR and should the PR be accepted by engineering? I'm going to ask you first and then you and I'll tell you what I think. Well, I think you can always submit a PR and engineering will tell you why it can either be accepted with if if it's good or what is missing. Uh may that be conceptual, may it be coding standards and all those things, but at least you know they're they're they're and this feedback for them. They're seeing what this person wants to do. I think it should be a great thing. Yes. I'll say yes with one caveat. And the caveat is the problem with vibe coding today is you might generate 10,000 lines of code for a very simple uh application feature that now the burden on engineering to read the 10,000 lines is there. Yeah. So what we say is yes you can submit a PR but you have to understand the code you are writing yourself before exactly before you submit it because I could do the same thing and I can say hey I'm going to write a blog post generate 20 pages and then send it to you and be like hey edit this and post it on your blog and now you have the burden of having to now if I said I read it all and it is my voice and I agree with it then you might be more likely to read it. The interesting thing that we're probably going to see this problem with open source projects because it's so easy to to use AI tools and they're going to get a lot of these things. So I think policies like this are interesting. Exactly. Now I wanted to get your take on one thing which is going around in and social media. There's a lot of people predicting it's the end of SAS because people can create their own SAS. Now what you told me is some of your non-technical people are creating like small solutions for ## AI tools' impact on SaaS [16:42] themselves for themselves. How do you see this potentially changing uh SAS vendors or not? or are are you you know and and you're also someone who actually you know buys a lot of SAS vendors you understand them. Do you think this changes anything or not really? Is it just more personal software? Yeah, I'd like to think we buy less SAS than most companies only because we try to consolidate down. That being said, there is this notion we're in this middle zone right now. The middle zone is basically this idea that you can vibe code something maybe for yourself. Um it's unclear whether you can vibe code as a platform or vibe code something that gets into the infrastructure layer of what you're building because you do really have to understand again like you mentioned it earlier like you might be putting yourself in a very precarious situation. If you're building on top of something and building on top of something you don't understand what you're building right maybe it's a prototype that's fine or a proof of concept but if you're building infrastructure for the internet you likely want to understand how that's being built. So that is not we're not there yet. Is it coming? Yes, it's for sure coming where someone can vibe code something and the you know anthropic or open AI or Gemini or who know who cares who model build something that's actually architecturally elegant and the right architecture for what you need. That's we're not there today. So where we are is um this notion of still human in the loop and still like start over and like this is not what I meant and like prompt engineering English as the programming language. Um and so I do think you can get yourself uh very far but we're not I'm not worried yet about SAS. I think we I don't think we should be worried by the way because let's take another view. How much software do we think should h be should there be in the world? Probably 10,000 times as much as there is now 100,000 like there's a lot of software that should be in the world and we are we are satisfying 000 or 1% of the demand. So now that everybody can generate software um we should welcome them right into the software world. We believe everyone should be writing software just like my sales team is and everybody becomes more productive. I'm not yet worried about the software industry. I still believe this is Jevan's paradox. The more we get, the more we want. Yeah. A really good analogy that has stuck with me was Simon Willis was saying, "How has the filming industry changed with this? Everyone has this used to cost like5 or $10,000 a few years ago, the this camera, everyone has one in their pockets have." And you know, there's still a professional movies industry, but now there's a bunch of, you know, there's YouTube, there there's Tik Tok, there's all these things. So it it has become bigger but the barrier for the the pros has not really changed. You know Dune and some of these amazing movies are so hard to make. Right. Jevan's paradox. The more you have the more you want. We all want to be filmmakers now. We can be filmmakers and then there's still going to be people at the high end and the low end. The I would say the controversial maybe controversial spicy take is that these phones actually benefited the experts more meaning the camera like bas you know like the being able to pull out your iPhone and take a take a video actually benefited those who were really really good at it already and the same thing is maybe potentially true of vibe coding. It's like the AI agents are going to help the best engineers more than the mediocre engineer. Mhm. So there was a memo that went around in the media about a month ago. Toby sent uh a memo. It was meant for internal ## Tobi Lütke’s AI memo [19:50] consumption. It was about reflexive AI usage and it got I guess it got leaked and then Toby posted the whole thing. And this is about saying that hey everyone is expected to use uh AI uh at at Shopify. Can you tell me what why that why you felt that memo was sent out? What was the response internally and has it changed anything or was it just kind of stating what what was there already? Yeah, it was a fundamental statement that this is something new and it was meant to get everyone to move from passive to active, right? So yes, there's AI in the world and chat GBT and all these things existed, but we started building infrastructure internally to make it easier to use and we ex we basically said the following. You don't have to use AI in your in your workflow, but we're going to expect that you have this plethora of tools available to you such that the expectation is you're going to be like your impact is going to be evaluated as if you had the tool. So for for example, you know, imagine not having like Excel or Google Sheets and you had to like, you know, like do a complicated analysis. You don't have to use those tools. You could use a pen and paper, but we're going to expect that you have these tools available and we expect the level of analysis to be treated as if you had these tools. And so by us doing it, what changed internally was actually probably more of a change outside of R&D. Like I mentioned, R&D was already leaning in. Now, of course, there were some folks who were waiting for I don't know. R&D is engineering. R&D is engineering, product, data, and uh design. They were more likely to use these tools because they tend to be more forward-looking and uh and uh wanting to use the latest and greatest. However, some people were waiting um that it changed their shape because we said, "Hey, look, we're going to expect that you use these." But outside of R&D, we saw it even more. Again, sales, finance, like customer success, help centers, like all those roles started using AI as well. And we again gave them the infrastructure to use that. So, we have like an LLM proxy. It has all of the uh APIs and all the models available. You don't have to be worried that your personal information is being leaked to a model. You can kind of use those tools. Let's talk about these things. Let's talk about the tools that you either created or adopted early. I I I heard can you talk a bit ## Shopify’s LLM proxy and how they protect their privacy [21:46] more about this LLM proxy and also I heard something about MCP. Yeah. So we are um when I talked to people so let me start with LLM proxy. The problem at the beginning was people would go to chat GBT or Gemini or claude and they would like you don't want to put customer data there. You want to put customer data there or even employee data. You want to use it to write your employee reviews. You don't want to put like, you know, Simon, you know, is is having problems with this and like put their first name and last name and here's a project they worked on and the code names leak. So, we wanted to quickly have an internal LLM proxy. Plus, we built on top of Libra chat, which is the open source uh native chat product and we're core contributors to that as well. And with the LLM proxy, you're sure to be using the enterprise APIs. We can check to see which we you get a token. You ask for a token. We can see which teams are using um how much uh like what what cost is being incurred by that team or by person. We have a leaderboard where we um actively celebrate the people who use the most tokens. Oh, really? Because Yeah. Because we want to make sure that they are if they're doing great work. Of course, if not if they script like that's not what I mean. I mean they're doing they're doing great work with AI. I want to see how did they spend $1,000 a month in credits from buy a cursor. Maybe that's something they're building something great and they have an agent workforce underneath them. So we have we have this proxy and again it's used to build anything that you want to build and you can ask for a token and use it. ## How Shopify utilizes MCPs [23:00] And then we have uh we we very early on MCP. We're on the MCP steering committee. We are fans of MCPing all the things. And so when I talk to people about how to get access to um some piece of data inside the inside the company, we quickly will spin up an MCP uh endpoint for them so that they can just use it. So for example, our internal uh wiki is called the vault. It has information about all of our projects that are happening. It has information on every internal talk that has been done. It takes the transcripts of all those. So you can say something like when did the point of sale at Shopify launch and it'll come back and it'll say oh in 2013 2012 there was a hackathon where the point of sale was started and then in the based on wiki doc based on the wiki and then in the 2013 Q1 board letter it was mentioned and it would tell you all of the history of what happened in Shopify. And so we do that by just having all you do is put up an MCP and now the Libra chat has access. It's like an internal perplexity or something like that. Exactly. And so we basically any internal document uh is is crawled by this thing and now you can have access to it via MCP. And so any new system comes online, we put an MCP in front of it and you now and now everybody has access to it. This episode is brought to you by Sonar, the creators of Sonar Cube, the industry standard for integrated code quality and code security that is trusted by over 7 million developers at 400,000 organizations around the world. Human written or AI generated, Sonar Cube reviews all code and provides actionable code intelligence that developers can use to improve quality and security early in the development process. Sonar recently introduced Sonar Cube advanced security, enhancing Sonar's core security capabilities with software composition analysis and advanced static application security testing. With advanced security, development teams can identify vulnerabilities in thirdparty dependencies, ensure license compliance, and generate software bills of materials. It's no wonder that Sonar Cube has been rated the best static code analysis tool for 5 years running. So join millions of developers from organizations like Microsoft, Nvidia, Mercedes-Benz, Johnson and Johnson, and eBay and supercharge your developers to build better, faster with Sonar. Visit sonarsource.com/pragmatic security to learn more. And And how many MCP servers do you have? You have wait two dozen. Two dozen. Yeah. And they're growing quickly. They're growing because people are building a new thing again. There's one for Salesforce. There's one's for um you know, Figma, there's there's a there's an MCP in front of anything. But I tell anybody who's looking at building AI systems internally, the way to make this easy is start putting MCP in front of everything. And now your data is accessible. We have to use Libra chat. You might use something else. But you don't have to be even in cursor, right? You just use the chat interface. Yeah, you can do chat. You you you can do the the LMS. And I like how you spun up basically a platform to build the emp is it is some AI platform or or is just a platform team infra team just took it over. Yeah. Data platform. I don't know if I don't I'm trying to figure out because they're inside the data platform team. I don't know if they have a a name outside like I call them LLM proxy but and and what was this more like you or someone up said like hey we need to do this or or they just like hey this is a good idea. that's kind of bottom up or it was more top it was more top down because we felt like this was going to grow internally. We wanted to make sure that we could we didn't want people signing up for their own personal tokens and it's also a good way for us to see hey this team is growing is the usage growing what's happening there this person's uh usage is yeah because number one push back I get from from people or even engineering directors working at companies is the data security issue and obviously when you host it yourself it's it's good but then you need the infrastructure so it sounds like at what point did did you decide what was this month ago was this more than a year ago to like actually invest probably I mean I would say every week it's getting better right I would say I would say probably a year ago we started thinking about it this way. We had the LLM proxy when when chatbt came out we actively have a warning if you go to catchb says hey by the way like you can go to the internal version that was a long time for a long time yeah now you mentioned cost and I am hearing stories of CTO's engineering leaders saying okay we were paying for GitHub copilot I don't know it's $10 or $20 per month per engineer ## Why AI tools aren’t the place to pinch pennies [26:59] but we have all these seats or maybe we have for everyone and they're telling me cursor feels a bit too expensive cuz they're now assuring 30 or 35 and we're not going to go there and and in gerally uh there's a big push back on cost. People are saying look like AI should make engineers more productive, but it's now like it's kind of ridiculous to spend like $1,000 on an engineer. You think about cost differently. Tell me about it. Yeah, I did I did a tweettor storm on this because I think that people looking at it differently. So, think about it this way. If I could give you a tool that could make your engineering team more productive by even 10%, would you pay for it? The answer is yes. Would you pay $1,000 a month for it is the question. And I my um hypothesis here is $1,000 a month is too cheap. Like it's too like if I can get 10 percent more, it is way too cheap. Like it should be $5,000 a month. Like it's insane that people are looking at this like $100 a month, whatever. Now, I did meet somebody who said their engineers were spending $10,000 a month per engineer. And I said, I want to talk to them because I want to see either they're doing something very very smart or very very stupid, but I want to meet them. And so I literally said, please introduce me because I'm I'm I'm unclear how you can use that much. It's like either they've got fields of agents building amazing things or something is going very very wrong. But I would like to understand because there's something interesting happening and you should not be penny pinching on AI tools because actually the productivity gains are there. We don't know what they are because we actually don't know how to gauge developer productivity but it is it is it is clear that there's something happening there and you should be uh very open with your pocketbook there. And so again we celebrate those who spend more and of course we talked to them and what did they do? So they were getting the Do I understand that even though you cannot measure it right now or maybe not as accurately you believe that there's productivity gains and you believe that people need to use it to get these so you're kind of treating as an investment right now like for a year for two years whatever that may be and then right probably forever like I mean again if you like you wouldn't turn off spellch check and grammar check and and G Suite and like you wouldn't turn off these tools right I mean there's probably a debate on when you turn off sort of turn off slack because it's like that could be a productivity but but like there it's it's We're seeing the gains and I don't think that uh we should be penny pinching. Now over time we might realize there's a equilibrium and like oh this is the amount of tokens they people should be using. Actually if anybody's using cursor here the number one thing I'll tell you is please move away from the default model because the default model is nowhere close to like actually using a sophisticated model and by default cursor puts you there. And so we tell people please move off the the the small models move into something more uh bigger. We'll even say things like please use 01 Pro for this project or please use 03 um 03 Pro because those they're more expensive. They're more expensive. Actually, I'll go even further. Miky, the CTO will say something like the following. If you you um don't pay personally for 01 Pro or Gemini Ultra or Claude whatever it's called advanced um the $200 a month one you are crazy because you can afford it and actually you are missing out on actually all of the progress happening in LM because you are just stuck to the $10 a month or $20 a month chat GBT like that's how far he would go. One interesting thing that I've heard when I ## Farhan’s projects and favorite AI tools [30:02] talk with engineers they're saying leadership is leading by example. Toby is is hacking and if you go on his Twitter or social media, you're going to see all the all the stuff that that he's doing even a year ago. I want to ask you like what are you doing on on on the side or experimenting or or playing with it and what tools are you excited about? Yeah. So, when I was building with when I was pairing with that engineer with an Entropic, I was building something for myself, right? Yeah. I was building a couple things. So, one great way to use company time, I guess. Exactly. Well, I was like, let me see what they can build because I didn't we didn't want to go super deep into the Shopify API. But we do build a lot of things that um you know, one was actually a commerce flow. I was trying to build a commerce flow via like operator where you can actually record the browser session and I wanted to see how the uh session was storing like uh credit card information or uh secure tokens as we were getting through a commerce flow. And I was trying to figure out I was trying to build almost like sub subscription product without using our subscription product from Shopify. And so it was an interesting experiment where we were trying to see how do we store those credentials especially when you want to store like credit card is it stored in the cloud already is it stored by the vendor are we storing it in the browser session that's how I kind of stay as close to it as possible I I use cursor pretty often to build my own workflows um in this case it was cloud code we actually even use gum I don't know if you heard of gum loop gum loop is an automation platform that is very interesting at its own layers oh here's one example we tried to build um I I did something super dumb I was like let's figure out when the next SanDisk Extreme Pro 8 TBTE drive comes out. Okay, we tried to build in cloud code. Took us an hour. We did not get there because it actually tried to do something at the wrong layer. It was trying to like read the JSON and it was like try to like get the image from the SanDisk site and figure out if the 8 TB drive is out like it was actually quite complicated. We built it in 2 minutes in Gum Loop. Not because what did Gum Loop do? So Gum Loop is basically like it was browser based. We said go to this website look at the like go through the search box like using English commands and they came back right away and said there's no 8 terabyte drive available and I said okay cool run it every week and send me an email it was just they're different tools for different jobs and an hour with cloud again nothing wrong with cloud code I use it all the time it was not meant to do this kind of like web scraping because what it is what does it do it tries to build a web scraper versus gum loop it already knows how to scrape websites so it's it's actually just a different layer of the stack and what I would encourage people to do is figure out where what you're trying to And we have a lot of automation happening in gun now because people have to do something. I want to scrape a LinkedIn profile. I want to find out if the what platform this company is using behind the scenes. I want to scrape a website. Like all of these things are it's built for the right thing at the right time. You have a lot bigger tools. Such exactly. So you should figure out where you want to work, right? If you're writing code, plot code. If you're writing if you're doing web skyscraper, use gum loop. So far it was an AI AI. And when this uh letter came out externally, there's been a lot of some speculation saying, hm, is this a sneaky ## Why AI-first isn’t about freezing headcount and the value of hiring interns [32:50] way for Shopify to freeze headcount because it says you shouldn't hire unless you can check if you can do it with AI. But then you told me something interesting. Uh I'm not sure how how big Shopify is, but you told me that you're applying to hire a thousand interns. Can you put put that in context with the size of Shopify? And can tell me why are you hiring interns where a lot of companies are saying let's not hire entry level people cuz we have AI. Yeah. What are you seeing that others are not seeing? Yeah. So a couple things. One is there there's this real notion of like generations as they come up in different uh like that generate that graduate from different uh programs out of university or even coming from high school. And you really want to be close to the next generation of of people. One they're like you know they know things that you don't know. And so we restarted our intern program in in full thrust this year. So last year we had like 25 interns a term. Uh I convinced Toby to hire a thousand interns this term mostly because I felt that they would Yeah, I know. They would be more AI reflexive than everybody else. That was our hypothesis. So my tweet actually said, I want you to come to work with an LLM and a brain, not one or the other. Yeah. The idea was that bringing these like centurs, AI cents, they know how to work with an LLM to solve their whatever workflow it is, whether it's a finance person or an engineering intern. And we really focused on them coming in and changing our internal culture. And so a thousand engineers over the year meant like 350 a term. And what we saw there was one, they're excited, they're hardworking. We also we also have them as a cohort, meaning they come to the office. We're a remote company, but we have the interns come in because we felt like the younger people need to have like younger people around them to work together versus just being remote in their like condo or whatever. We're changing the culture that way as well. And we did see this. They come they have their AI reflexive kind of we saw the same thing in mobile. You were in mobile as well. And so was I in my last decade of work and I I hired lots of interns because they grew up with mobile phones. And so this next generation is growing up with the internet and they're growing up with uh phones and they're growing up with LLMs. So we wanted them to come in and change our culture around that that axis. So, do you feel like people are learning from We all We always We always were. Interns are the secret weapon. You always learn more from the interns. Like people think about internships as like community service. Oh, that's so great, Shopify. You're helping the next generation. We don't do it for that. We do it to learn from the interns. That is always the reason. So, if you don't have an intern program, an early talent program, I encourage you, you should be bringing those people in because you will learn from them more than you will get like out of them. Like, you will you will take a net positive benefit from them. And of course, for those who like tweet at me and say, "Please don't hire interns. You have to pay them. I'm like, we pay them. I don't know why people think that you don't. We pay them like very well actually. And like they we do hire I think I've hired almost a hundred from the in the last in the this year from the intern pool to come into our program. And we're very much thinking about the only way to get into Shopify as an entry- level uh engineer is through the intern program. I'm fully with you with with on the paying. I never understood why companies are are cheaping out on this because sometimes interns will reject uh some of the offers because it doesn't cover their living cost, which is wild. Of course, you have to pay them. actually one of the best intern I it was another company's loss and in Amsterdam they offered 500 euros per month no housing and that intern would have loved to work there what a great experience but they applied to Uber because they said it's just too low like and then we gave them a new grat salary they could afford housing they could afford to travel and it was that intern could have done amazing things at that company and now this person is still senior at at at Uber you know been there for 5 years so you know their loss so think about it 350 engineers uh interns to about 3,000 engineers about 10% at any one point. Yeah. So if I join Shopify as as either ## How Shopify’s engineering department operates, including internal tools [36:20] a software engineer or or an engineering manager, can you tell me how stuff gets done for first as an engineer? Like what what do what what what tools do people use? I'm assuming it's just no longer you know VS Code do code review. There's going to be like AI tools here and there. Yeah. Well, first on we have a very specific way of writing software to the point of we built our own uh project management system. Like we don't use really Jira, we don't use linear, we don't use Yeah. And the reason is is because we believe that uh there's that line like first you make the tool and then the tool makes you and so we we believe that and so we make the tool so that it can make us versus we it's not nothing against those amazing tools. I'm a big fan like linear is beautiful and like you know uh I've known Jirro for a long time. I'm a pivotal fra guy myself being a pivotal ex pivotal person but we do believe that we have a specific way of writing of building software and if we use one of those tools we would be adopting someone else's way. Mhm. Yeah. And so instead we have a tool called GSD which stands for get done. And it is it is a tool that allows us it's a program management tool like a pro a product management tool which allows us to think about what are we building who's on the team what's the latest weekly update. It has metrics. It actually pulls in PR reviews so you can see like uh it pulls in PRs and you can see there's activity happening on this project. Y here are the core contributors. Here's how long it's been going on for. This tool is how you kind of get work done and you're forced to like write an update every week. And now actually we have an AI here's an interesting one for you. We have an AI tool which will um pull in the latest PRs and the latest conversations from the Slack channel and it'll write a update for you. And then you can look at it and say this looks good and you can also tell it what did I not think about and say please emphasize this and it'll help you write the project management update every week. So this is interesting. I I want to push you a little bit on this because I think there's two schools of thoughts here. One says your weekly update like I appreciate that that you're doing it. The weekly update point should be for you to look back on on what you did. Obviously, everyone hates doing this, by the way. And with by AI doing it, would you not lose that kind of reflection? Well, this is why we pair you with the AI. So, we don't we we we do auto we do autopublish it if you don't do anything. But if you autopublish it and don't do anything, we look at the stats. um there is a little bit of loss from you because you are now like losing out on the context that you were supposed to gain as the champion of the project by looking deeply. However, we expect everyone just like a PR written by AI to have read the code and read the update. Yeah. So, you're responsible. You're responsible. And so, we're early days in this. We're like literally 3 weeks in on this project and we may revert it and say, "Hey, by the way, it turns out everybody stopped looking at what was happening week to week." Yeah. But we also every six weeks at a company level with Toby go through every project in the company and we see if we think that it's on track, not on track, should be kept working on, has the right resources, is on too long, is it aimed in the right direction. You better know what's in there because literally we're going to review it at the leadership level. Yeah. I I I think you have no I I I like how because I feel like with with with AI like I kind of want it to automate the stuff that I really don't want to do. Yes. And ironically it's really good at automating stuff that we like to do sometimes which is coding but like writing an update or writing documentation. I don't think as engineers we loved doing that. And so as as an engineering again we're trying to reduce toil. If we find it reduces actual context we will we will revert. I I I love the experimentation and and and and going back and forth. As an engineering manager, what what additional things might might people use? Do people build their own tools to stay on top of things? Do people go deeper because of Yeah, as an as an engineering manager and a director, we try to surface again metrics so that you can see how your team is doing. So, for example, we have tools that allow you to see things like um focus time of your team, how often they're are they in meetings, um AI adoption, are they using like AI tools or not? Like we're trying to surface this information. um how many of your people are on a GSD project versus not because maybe a project ended and they're like freed up to work on something else. So we try to surface that and engineering managers too we try to get them to be a lot of the best ones come from IC land like they they started as IC's so we try to get them to do if they're hired outside I start as an IC at jobify before they move into management. M oh here here's a fun one that I think we should talk about. You ## Why Shopify added coding interviews for director-level and above hires [40:31] mentioned something super interesting to me when you're hiring engineering directors and above. In the past it was the usual interviews you know culture fist strategy all that stuff. You added a coding interview for every single engineering director and above hire. Can you tell me about this? Yeah so it's interesting like it's maybe shocking to folks and I know like Rodney's sitting there. He used to work for me. I did the pairing interview with him and so what happens is it is shocking for VPs especially to be like whoa there's a coding interview. I'm like, "Yeah, because we believe that the best leaders here were ones who were not running away from coding. They just felt like they got better leverage from running a team. They still are deeply in love with technology and they still, in Rodney's case, still codes on the weekend, right?" And so we it worked out super well. But our whole idea is that you're not running away. You're running towards technology and this is just a better way for you to express it. So I pair with the with the candidates and they also see that even though I'm not writing code every day, I'm still deep in the weeds of technology. I still love technology and I still want to talk about technical topics and so we pair and some people believe that that's not the best for them and they can you know there's lots of great companies out there where that's not the requirement but at Shopify we believe people should be as close to the details as possible and we are such a tech nerd company like again our sales people are vibe coding now we want our engineering leaders to be coding as well and so that doesn't mean coding day-to-day but you should understand code and how code works and a lot of it comes back the muscle memory of coding will come back in these pairing interviews yeah and I I guess especially now we have these coding tools when you You know, you can generate something as long as you can look through it. Like you This is the best part about co-pilots is that a candidate comes, the co-pilot generates tons of code, and now I'm like, great. Is that good code, bad code? I love Hold on. So, you're using AI during your interview process? Yes. Oh, you're not running away from it. You know, one of the interesting stories that we just learned, Cursor has disbanded on the interview process. They're the AI tool company, right? So, you're embracing it. We're embracing it. Okay. How's it working? Tell me. I love it because what happens now is the AI will sometimes generate pure garbage. So, so you're screen sharing and you say literally use that screen sharing and they're using I say co-pilot or whatever let them use whatever they want. Here's what I'll say. If they don't use a co-pilot, they usually get creamed by someone who does. So, they will have no choice but to use a co-pilot. Sometimes I will shadow an interview and you do the questions myself. I've never seen them with a co-pilot and send it to the interviewer and say, "Please mark my assignment as well uh against the candidate. I have not lost yet." If they have not if they don't have a co-pilot, they will lose. But when they do have a co-pilot, I love seeing the generated code because I want to ask them what do you think? Is this good code? Is this not good code? Are there problems? And I've seen engineers, for example. Um, when there's something very easy to fix, they won't fix it. They will try to prompt to fix it. And I see, are you really an engineer? Like you I I get the the nuance of like just prompt and prompt and prompt, but sometimes it's right there and they will not prompt. They I'm like, change the one character and they won't and they won't change it. And I'm like, okay. So they they they're interesting. I I don't want you to be 100% AI coder. I want you to be like 90 or 95. I want you to be able to go in and look at the code and say, "Oh yeah, there's a line that's wrong." So I want to ask you what is a standout senior or staff engineer today look at Shopify and what has changed let's say 3 years ago when we didn't have any of these tools. I'm trying to pro here like how you think AI or AI tool usage might have changed of what we ## What has changed since Spotify added AI tooling [43:40] expect an amazing engineer and you think of the the person who you think is like this fantastic person. Well, I would say the thing that's changing is I'm seeing some of my engineers now actually use these AI tools to like do the infrastructure that they've always wanted to do but never felt like they had the time. So like tech debt reduction, refactoring, u making things easier to read, like those are the things I'm seeing the best engineers at Shopify use AI tooling to actually do and they just never felt like they had the time or the resources. Like the engineer who always felt like I wish I had like six months to do a code red now can use an AI refactoring tool like a cloud code or a open AI codeex to like unleash these agents in parallel to then and then review all the PRs but then unleash it and have it and feel like they have a team themselves. So that's what I that's where I'm seeing and they're not afraid of of trying things that may or may not work. They might unleash something for 24 hours and like a Devon like tool and it comes back and like utter garbage they delete it all but they said it was worth trying. Yeah. And as a closing question uh in and and a lot of people are are thinking we'd love to be like Shopify like I'm ## Farhan’s advice for implementing AI tools [44:40] either an engineer, I'm either a staff engineer, director of engineering, etc. I' I'd love to transform my culture to be a bit more AI first. What would your advice be? How can people start? Like not everyone has a luxury of having the CEO saying okay I I love these things. Do you have tips to that you give to your peers? I know you're in CTO groups for example. Yeah. So I think that the number one thing is role modeling is the best example. like I haven't seen anything work better than role modeling and the way that happens is like you have to do it. So if you are coding and you are showing people your workflow and you are showing you are in those same channels asking like hey I was trying to get this working and it wasn't working or I I was able to get this workflow going via this prompt. We have a prompt library internally where you can literally grab prompt and yeah and like say like you know um hey this prompt worked for somebody let me try it and modify it for my use case. That's the number one way in which I see people um trying to adopt AI is because they've seen other people do it. And so we try to share like AI use cases. We have again prompt libraries. We had a hackathon like in our as a part of Shopify Summit a few weeks ago and we leaned heavily into AI and it wasn't just junior people learning AI. It was senior people trying to embrace cloud code and one guy said I haven't coded in six uh six weeks. I've only used cloud code. Then I pushed him and he said okay I have to make changes like here and there. I said cool. You're 95% AI but that's the point. You're not supposed to be 100. And uh role modeling is the number one thing I've seen that and sharing examples. So role modeling and then sharing especially because no one has it figured out. So we No one has exactly no one no one's no one's there yet. Well this this was wonderful. I I I love how much you're experimenting. I I love how you are not afraid to you know like take some churn to learn more. And I love how much you're sharing internally. I I think we got a little window of this but I I think it it's just something I hope more more more people more companies will do. So this was wonderful. Thank you so much. Thanks for having me. I found it refreshing to see just how practical Shopify is about LLMs. It feels to me that they understand that you need to invest and experiment to get results. One surprising thing for me was understanding how Shopify expects to become more effective using AI tools by hiring interns who come in with an open mind towards AI tools and can come up with clever use cases for it. For more details on how Shopify operates, check out Deep Dies in the Pragmatic Engineer linked in the show notes below. If you've enjoyed this podcast, please do subscribe in your favorite podcast platform and on YouTube. This helps more people discover the podcast. And a special thank you if you leave a rating.