## Intro [00:00] what does actual development look like when you join fabricator is our internal code review tooling fabricator is well integrated with sand castle which is our internal CI tooling which is integrated with on demand which are internal Dev boxes which is integrated with land castle which is how code makes its way out to users and sort of the thing goes from there and then it doesn't just stop at code review it's integration across the rest of the developer platform for example meta built its own task system and so tasks and poll requests could be very deeply integrated in a way that we just don't see out here one tool that I kind of experienced because of fabricator was called heral heral if I recall correctly was the rules engine which later got replaced by butterfly bot and the idea of both of them was the same it was be able to match events that might happen in the normal course of code review I used to work on internal developer tools there when we were deprecating apis you might go ahead and do something like create a heral rule that would say something like if a user calls this function or uses this API post a comment letting them know hey this is deprecated also the new syntax is this and then as you got later and later into the roll out you could start to do things instead like hey notify me if anyone's adding call sites of this API so that I can go ahead and go leave a comment saying hey like no we're really trying to deprecate this meta is famous for his customer internal tooling Dev at the company use the likes of fabricator Sand Castle or butterfly bot but what are these tools and why did meta not use a more standard stack to build software we get answers from Thomas rer who was previously at meta working on developer tooling before co-founding graphite of devel Vel productivity platform today we talk about what stack diffs are and how they became the defecto way Dev at meta work with polar requests why men I move to a monor repo and why there seems to be an industrywide Trend in companies moving from poly repos to monor repos why AI coding tools will make code reviews a lot more important and more if you're interested in how effective engineering teams work these days and approaches trending in this group this episode is for you if you enjoy the show please subscribe to the podcast and consider leaving a rating this great to help the show and with this let's jump in Thomas ## An introduction to Meta’s in-house tooling [02:00] it's great to have you on the podcast great to be here so you before you started graphite you worked at meta and meta is known for a lot of their in-house tooling can we talk about like when you joined what was interesting tools that you didn't see at other companies before so funny it's a great question too and yes certainly it's when I joined I was a new grad so I just graduated college I joined meta I used a lot of the tooling the my frame of reference was personal projects I used to do a lot of hackathons back in college and internships and so I think when I learned when I joined meta a lot of it came in sort of uh eyes wide sort of like uh I was about to say eyes wide open but I mean the opposite of that if I came in sort of just like blissfully unaware that this was an industry standard uh when I left was when I had the bigger shock because I think I had used tools like GitHub before I'd used analytics tools before I had left meta after after having really gotten sort of uh immersed in all of this wonderful tooling and then when I had to start to do all of these same activities myself I found myself reaching for tools that didn't exist and there weren't clear analogs for yeah it's usually the other way around right like usually people who work in Industry at a startup or even at a larger company go to meta and unless you've been in like Google where you know it's all custom is just a shock as I think most people imagine like okay you know they must use GitHub or or at something similar but what what what does actual development look like you know when you joined right like you you went into boot camp I assume what what did it look like to like you know like they didn't even push a p request but that's not what they call it do they no no they call it a diff um good good good trivia for by the way um I think that though if I had to use one word to describe meta internal tooling it would be integrated everything everything everything integrates with each other and I I tell this to people and they never really understand the extent of it meta builds its own internal calendar right so meeting room scheduling can be integrated in with the rest of the tooling and that really sort of permeates the whole thing right fabricator is our internal code revieww tooling fabricator is well integrated with sand castle which is our internal CI tooling which is integrated with on demand which are our internal Dev boxes which is integrated with land castle which is how code makes its way out to users and sort of the the thing goes from there and then it doesn't just stop at code review its integration across the rest of sort of like the developer platform for example meta built its own task system and so tasks and pole requests could be very deeply integrated in a way that we just don't see out here what was sand castle so sand castle was the internal CI system so out here a lot of people use uh GitHub actions they use build kite and Sand Castle was just our flavor of that it allowed us to build versions of our apps and uh websites before we released um and went ahead and tested them for us and you mentioned this like integration like you know when you say it it sounds ## How Meta’s integrated tools work and who built the tools [05:07] like it's just it's just a word but like in practice what what did it mean you know you did mention that for example tasks which is I guess we can think of it it's the jira or or you know linear or whatever that is that was integrated with the other stuff and this was like you know like 10 years ago right or so um yeah actually um yeah I think integration here I think one of the best examples I have of integration was actually a translation system so at scale internationalization becomes a real problem right you need to be able to include internationalization as product developers start to update the website create new pages Etc there's always this question or this tension that happens of well how do we know that the string that we're pushing has a translated equivalent at Facebook that could actually just be built into the code review tooling right and so the code review tooling can tell you something like hey this diff is using a string for which the translation don't exist and because of that you can't deploy it yet or what was more interesting to me was when you looked at a diff that had been merged that had been landed um that was what we called it you would see on that same page like hey this diff has been rolled out to employees this diff has been rolled out to 1% of users this diff has been rolled out to 10% of users also if this diff included any feature Flags or sort of like AB tests here are the results of those AB tests and so you had this wonderful ecosystem where everything was tightly tied together and so you as a developer could just go to one place and be like I know what I'm looking for I'm looking for this piece of like code that I landed let's go see like how it's changed statistics or how uh how it's impacted users and you could see that all in one place and then there was a bunch of other stuff that started to make its way into the public tool chain now of if you wanted to revert a diff you could just go ahead and do that directly from uh from fabricator this episode was brought to you by swarma the inuring intelligence platform for modern software organizations swarma gives everyone in your organization the visibility and tools they need to get better at getting better enging leaders use swarma to balance the investment between different types of work stay on top of cross team initiatives and automate the creation of cost capitalization reports inuring managers and team leads get access to a powerful combination of research back engering metrics and developer experience surveys to identify and eliminate process bottleneck software Engineers speed up their daily workflows of swarming as two-way slack notifications working agreements and team Focus insights you can learn more about how some of the world's best software organizations including mirro Docker and webflow use swarm to bit better software faster at swarm mia.com pragmatic that is s swmi a.com pragmatic fabricator was a really central part of this I know because at Uber we use fabricator it was it was an open source tooling and I think Uber and you mentioned Dropbox also took it on maybe a few other companies but but not not as many so for a while we kind of used it I I saw some of of this integration but it it sounds a little wild because I I don't really know any other company maybe outside of Google where you can have this thing that you know as as you say you're you deploy a code and you can see the the experiments the localization the the roll out all from one UI usually it's different systems even at ruber it was different systems we we had a you know like an experimentation system it was called Morpheus at the time or or Flipper the feature Flags it was you know and we had I I had this list of like systems that I you know like a cheat sheet of where I would go to check the roll out the internationalization etc etc so all of this was really kind of like you one click away I think that's the way it exists for a lot of people today you know I think when we look of right now best-in-class Dev tooling is starting to become more integrated I think around like linear it's now starting to post the linear issue in the GitHub in the GitHub poll request and you can see and it updates the status as the poqu moves through review um you're also seeing the same with Rell right when you create a PR Rell can go ahead and be like hey went ahead and like made a preview environment for you you can go ahead and click that there I think what was so magical about Facebook is that all of that was native and it didn't appear to be Integrations but just part of the system and who built this tool right like uh like the the developer tooling you know fabricator uh the sand castle all of these things was it just like devs like okay I'm I'm going to build this obviously it was internal teams right but was it like dedicated teams or or people just kind of went and like went and tweak stuff I think it really depends on so it really depends on the tool some internal Dev tools were just built by developers themselves they saw problem they're like nothing's addressing this I need to fix it some came more from the com isn't it it's a it's an wonderful culture to be a part of where you can solve your own problems um but some of it came more from the like oh we as a company have notice that there's an issue here we need to address the issue let's spin up a team to do it and then the place where all of these teams Al or all of these tools ultimately found their home was under the dev infrastructure team so developer infrastructure at other companies called developer platform and developer velocity really maintain the bulk of uh Facebook's developer tooling yeah and I ## An overview of the rules engine, Herald [10:20] guess like one tool that I kind of experienced because of fabricator was called heral I'm not sure if if you know so so it's it was of course you could set up special rules on on the code base anything like if if basically on on a commit anything happens like weird rules could could could run were there like can you like do you remember some some like there were some really creative examples of like using Herald rules for developers to start to create their own rules or code ownership or that kind of stuff totally um so Herold was Herold if I recall correctly was the rules engine which later got replaced by butterfly bot and the idea of both of them was the same it was be able to match events that might happen in the normal course of cod review something like someone puts up a PR that includes some line of code or calls some function or CI fails regarding something and then what it would do is it could take a variety of actions so some of the examples that I remember was when I I used to work on internal uh developer tools there uh when we were deprecating apis you might go ahead and do something like create a Heralds rule that would say something like if a user call this function or uses this API like post a comment letting them know hey this is deprecated also the new syntax is this um and then as you got later and later into the roll out you could start to do things instead like hey notify me if anyone's adding call sites of this API so that I can go ahead and go leave a comment saying hey like no we're really trying to deprecate this like please stop um and then you could take it from there other examples were simpler um I remember there were some cases where there were uh let's call it privileged or important parts of the code base and what you'd say is like if someone goes ahead and creates code that touches that part of the codebase just make sure this team is added as a reviewer or at very least as a subscriber subscribers were an idea and fabricator of being able to CC people it it wasn't that you expected a review more just that you wanted to keep their eyes on it um and there were a handful of rules like that and and do I remember correctly that code ownership was super ## The stages of code ownership at Facebook and code ownership at Google and GitHub [12:20] super important for fabricator and for Facebook and it was just really trivial to say I own parts of the code and then set up different rules around that no so that's a great question so code ownership actually became uh code ownership went through a few stages at Facebook there was a while where code owners were required at Facebook and so you could have like different parts of the code base could have different uh owners then the company took it away and they said actually we don't believe that we should have code owners we're a collaborative environment we should go ahead and get rid of that it was then reintroduced and I think it was then again taken away wow um with the idea of like we're not sure the company alternated between oh they're really privileged parts of our monor repo and we can talk more around sort of how Facebook structured their code that need certain people to uh review it and well actually we should trust our developers developers can developers need to be able to understand organizational practice one thing that was always existing within the platform though was this idea of if you're in some part of the codebase tag the correct reviewers and so I think one of the things that actually I most missed when I got out here was the ability to structure repository in a way where you could say oh if you touch this part of it we should tag these people in I think in GitHub code owners is a very coarse approximation of that where you have to Define one file it's at the root of the repository it's difficult to use a person who I've heard has done this even better or company uh is Google so in Google uh code reviewers were assigned hierarchically and so in every folder you could basically say like hey if you touch code in this folder like you need to go talk to these people either as a heads up or as a requirement and it allowed people to get um there were two uh really nice aspects of that one was that it was really easy to Define ownership and then the second was that it was easy as you were writing uh poll requests there to be able to go ahead and say like who does own this code who can I go ask for and so if you're in a say that you using an API that you weren't certain of you need to figure out like who can I go talk to about this API what you could do is you could just navigate the file tree find the team or people in charge of it and then just go DM them and so that became a really powerful system there now you've you've gone back and forth between you know at Facebook there was code owners no code owners and you're now working at a startup you actually work with a lot of startups and and larger companies what ## Tomas’s approach to code ownership [14:39] is your kind of unfiltered view on code ownership is it a good thing is it a bad thing is it is it necessary only when you're bigger or or when you're smaller I'm sure you've got some opinions now I work going to start up now graphite we have the privilege of working with some of the largest St tools teams in the world which is really cool at this point I think I've heard all sort all sides of the story I've heard the we are in a financially regulated market and because of that we have a legal requirement to have code owners and so I've heard that side of the world I've also heard the side of the world of oh we could possibly do code owners here like it would just slow us down too much and so I think it really depends on the nature of the business I once saw this Matrix which I really liked which is if you trust your PE if you trust people a lot and are willing to tolerate like small mistakes you should lean on culture not process um as you start to get into a place of uh oh actually maybe I trust the people less or I can't tolerate mistakes at all you move to Automation and enforcement and so it really depends sort of where your business falls on that I also will say that I think that as Facebook's grown um the way they treat different parts of their code base has changed whereas some parts of the codebase absolutely cannot go down right things that deal with privacy cannot go down that's not a place where you can move fast um some other things perhaps uh maybe the way that you display comments if it's uh not a big deal that actually is a little bit easier or a new product even um there you have a little bit more leeway to move quickly and so maybe code owners isn't the right match yeah and you know one thing that i' I've kind of like learned over time is maybe ## A case for different constraints within different parts of an organization [16:15] inside a company as much as you'd like to think that there's one inuring culture maybe that's not the answer maybe you know like a new a new startup inside a company like an established compl met a new startup maybe it should operate differently cuz maybe they'll be shut down in in six months if they don't find product Market fit and you know they should just throw away the things it's it's a little bit uh at the same time I think companies don't really like to admit it but you know like the and and maybe even inside a small company right like if you start uh just a small prototype project see if it goes anywhere you're probably not going to have the same rigid thing as when you're deploying to you know like customers who actually count on it it needs to stay up I think that's a really wise point I think that ultimately and uh engineering anywhere exists within a system of constraints and I think those constraints within a company can be very different and so recognizing what are the constraints on the product I'm building right now and architecting your systems around it is an engineering problem like any other right sometimes you have constraints of uh I think the the place where this actually comes to me uh most clearly is when I deal with developer tools teams who release mobile apps versus web apps so web apps tend to release like on a like hourly basis um maybe maybe multiple times a day but not slower than that whereas mobile apps release on a weekly Cadence because of that just the um the shape of problems they deal with are so different even within a company so mobile app teams generally don't have to hold pagers because even if something goes down what are you going to do submit a fix to Apple great that's two days yeah um versus a web app team like paging is an important part of the culture right as you're shipping things really quickly things will break things will go down and you need to you're on the hook to fix them um and so watching the difference between those cultures I think is really is probably the like two extremes you see within a company and then again as you start to work in different parts if you work on Payment Processing versus uh abuse and fraud detection versus like the consumer front end versus authentication um you actually have different constraints which are subtly different and I think that as you're saying engineering organizations can develop different cultures around how they need to treat them I I I like this thinking of constraints and how you know it leads different practices now one thing that really surprised me at Uber which came really from from meta uh from from fabricator is we use this thing called stag diffs and we did a deep dive into pragmatic engineer uh it's the it will be linked in the notes below but it's something that it just like really ## The problem that stacked diffs solve for [18:42] clicked for me and also everyone around me everyone I talk with at at meta uh you've now actually built a like a business on top of this but can we talk about what is stack stack diffs and you know how did you come across it and why why is it so you know like useful once you get a hang of it absolutely uh um so you're right we we've spent a lot of time talking to people about stacking about stack diffs and so stack PRS as as they're called on GitHub um and I think that we've uh we've had a lot of practice about how to talk about it I think the the way I like to introduce stacking is by talking around the problem that it solves and the problem that it solves is in any given engineering organization you have a bunch of Engineers who at any given moment might tell you they're blocked they're blocked on code review and I think the when you really sort of like start to ask well why are they blocked why is code review blocking them what it comes down to is in most engineering organizations the way development works is you have a main line so you have a main branch or a master Branch you go ahead you Fork off of that you go ahead and you create your feature branch and then you kind of get stuck because before you can continue to develop on top of that feature Branch it needs to merge merge back because exactly um you need to merge it back because you only create feature branches off of the downside of that though is that your development your personal development process now becomes blocked on review because you've created some PR you've created the perfect way to do uh I don't know authentication loging and you now want to build forgot password and so your options here are you can either add forgot password back into that Branch but suddenly that branch is going to become really big and your reviewers is not going to be able to review it or if they review it they're going to give it a like rubber stamp looks good to me yeah yeah that kind there's no point in a Cod review when you do that exactly and so uh Google has a funny academic paper where they refer to it as reviewer frustration as as poll requests get longer um reviewers buy out of the process and end up just approving things rather than actually reading them uh and so the solution uh the solution which meta uh stumbled upon is what they called stacking Google I've heard has stumbled on something similar uh there might be some cross-pollination with meta there but the idea is you create your feature Branch you put it up for review and rather than waiting for it to be reviewed for you to start building you just Branch off of that and keep going and you create another feature branch and then you put that up for review and then while you wait for that to be review you create another feature branch and you create that to be reviewed now people who have been using it long enough most of them have tried something like this and I think inevitably you end up with a few questions one is well what if like someone rightly calls out that that earlier PR needs to be updated um the general answer is that's actually fine um generally when you need to update a PR you're not changing the abstraction boundary between PRS um an example of how you might break it out is if you're building a new feature I have server and then I have front end and so even if that someone's like hey can we refactor This Server usually the endpoint uh abstraction is actually staying in place and so the next PR can continue to be stacked on top of it the other question is uh well uh how do I know what I'm supposed to break up uh and I think for that generally and this this is actually really interesting to me generally developers build uh their systems in order right you might start building the server and then building the front end I think stacking is just a way to communicate that same order to your uh to your reviewers a lot of people do this automatically using commits right they just say like oh I'm I'm going to build commits in the order I would build them and you should review them by commits sacking goes a step further and says well if the first three commits in our Branch are approved why not merge those while you wait for the latter ones to be approved and that has a few other benefits you can run C on them separately you can uh reduce merge conflicts and you can t uh and you tend to be able to reduce your time to merge because if there are sticky things uh you get stuck there but can still merge a lot of other code the other advantages then happen sort of afterwards of in the case that there's an error or something like that you can go ahead quickly identify which PR was offending and if you tell me a 10line PR is offending as opposed to like a 2000 line PR it's much easier for me to identify the issue so so basically you get to work in just like smaller increments and it's almost like forking off of your your current Branch you know like smaller and small have a bunch of small Forks except it's just a tooling right that that like so wh why do people not do this you know like just forget about like the tooling that met have but you could just literally you know create a branch and then create yet another Branch but I guess it's just really complex to look at what's happening complex toerge it back I think git makes it scary and I think GitHub makes it scary so so in git the most basic way to do this would be to create branches off of branches but then if you need to update something you need to go ahead and rebase it now that's where I just use a really scary word for a lot of people rebase Reb oh we hate rebasing I I mean okay I can't talk for anyone but I'm going to admit it it always freak H having a the rebase conflict and figuring out what I need to do yeah it's scares me and I've been doing this for years I actually have a stack Overflow post bookmark which is the three types of rebase because depending on how many arguments you give it it behaves differently and so super frightening I think that's the place I'm very fortunate that I learned this at meta because I think what meta has is and what I've learned Google has and what I've learned uh places like stripe and Uber have built are internal tools to handle this for you where you can basically say like I'm creating a stack now I now need to update a branch in my stack do that for me and under the hood it takes care of this intricate set of rebases that while you can do by hand are easy to mess up now there there's a lot of tooling that does that that does that client side portion the other half of this is the is the uh po request side is if your host does not if your wherever you have your PO request does not have support for stacking I think Stacks can look very foreign because what you end up looking with is a lot of PRS that appear to be merging into like that appear to be recursively merging into different branches that they were all based off of and so it can be hard to understand what's going on and without any clear explanation or affordance in the UI your reviewer might become confused too so I'm kind of like ## How larger companies drive innovation, and who stacking diffs not for [25:01] wondering if if you have a theory why did you know like a few companies like uh meta and Google invent this and why why is why did why is it just not part of get or or GitHub I'm sure you've speculated about this you must have um certainly so um why why I think some companies built this I think first and foremost the current tools weren't solving their own needs right Curr tools being and C like and or whatever they had at the time right um at uh mercal mercal yep um I think they were shipping a high volume I think they were really early on monor repos and I think that they were getting to this point where okay I'm blocked on review progressively larger parts of this org blocked on review and as you sort of like start to grow up um as a company you realize like oh like if developers are spending 20% of their time waiting that's actually a lot of inefficiency that's being created AC the or right most people that run an or of over 100 Engineers would be like I would to have more Engineers um second thing is that uh they saw the potential impact so I think for a lot of people they they sort of like saw that lift of like okay um Facebook for example really prioritized being able to improve their ship velocity for whatever reason it mattered to them a lot um Google I think it as well of well we're a web company if we can like continuously deploy we can continue to improve things um and thirdly I think they had near infinite resources and this is a thing that I think most of us don't have which is they saw the potential impact they saw a problem they had the resources and so they could build a solution I think it's much harder when you're a startup or even just a smaller scale company than one of the what three most valuable companies in the world uh to be able to say like I'm going to take a few engineers and I'm going to build this yeah I me these are companies who are printing money despite having you know thousands of Engineers even in tend of thousands so so they I mean obviously they're going to like like be rational they're not just going to create a team out of the N but this is very interesting because what you mentioned is I remember I talked with Nicole with dror uh with niol forren uh about developer productivity and you know one of the things that came up when we discussed is like well you do want to like measure certain things just to get a sense and one of them is typically code review like how or how long does it take from writing the code to get and get in there so I have a feeling that companies like meta Google and some of these bigger ones who did have the resources they started to measure it earlier therefore they decided let's build better tooling and one interesting part that I've only learned recently I was talking with one of the three uh Linux fellows uh Greg uh he's you know there there's and he told me something interesting that git was created by the Linux Community well specifically not stols but it was to solve their issue of Linux development where they had about three to four changes per I think per per hour or so and it was email based list so they created G to solve their own problem and now I think they have something like 40 50 changes per per day or or per hour something like that I think it's per day actually that that's that's their rate and you they built it it's all for them they open sourced it but it was for them it was there's this Linux there's this big it's also a monolith uh the changes are pretty small for the most part there's not as much let's say you know like user facing code Etc but they sell for themselves so I wonder if they just didn't really have the problem that stack diff does because they they don't have uh if you look at the average Linux change it's a lot smaller than let's say you know like a new feature being added to some of the Cod bases at like met or Uber I think uh you hit the nail on the head with it's really important to use tools that are built for your use case I think that open source development generally and Linux specifically uh stacking makes no sense for so people always ask me sort of like who is stacking not for my answer is open source and the reasoning is that in open source you're dealing with a lot of untrusted authors who come to you they're like hey I have this feature it's partially built I don't have tests for it yet I haven't fully tested it can I merge it and it's funny in that in a company if you're like I have this feature it's partially built maybe you even have some tests for it but the feature is not totally there you're like I know you're going to be here tomorrow right and so cool thank you for doing job I appreciate that you're contributing back to the codebase and that you're going to prevent me from having to resolve rebas conflicts later merge conflicts I think that um in Linux you kind of see the opposite right and if you've ever spent time on those email lists you'll see people throw back and forth sort of like patches and what will happen is that they'll say like oh like this isn't complete you're not going to work on this Edge case right and rather than saying like you know what like we can put this behind a feature flag and merge it they're instead like go take this back fix it fully finish it rapid and tests and documentation and everything else and only then can you merge it um and that makes a ton of sense in open source right having worked on open source repos myself the biggest fear is that the contributor is just going to disappear it happens more often than we like to admit they get busy something happens and so in that case you don't want to land partial Stacks you want to make sure that the thing is fully complete before you even consider like reviewing it truthfully yeah and this is actually like when I had a discussion this trust ## Monorepos vs. polyrepos and why Facebook is transitioning to a monorepo [30:25] came up so much with with open source so it's interesting how like you know as you said the context like inside a company where you have people you can actually trust them that they'll follow up or if they don't follow up it's not a big deal you know like it it might be deleted interesting how how this context like like makes such a big difference now another thing that is unique I guess maybe not just to to large companies but it kind of started there monor repos yeah so what what was the you know meta and Google were probably one of the two companies to start doing monoral you've seen it at at meta what what was the history with uh with with with the monoral story there and and the kind of pain points you saw yeah totally um I think you have to remember so first things first you have to remember that Source control at meta has a funny history when that company started and this is well before my time rumor has it the way they did Source control is emailing back and forth zip files and so at some started yeah um I mean that wasn't so uncommon right get itself is a a fairly new innovation came around in 20072 2008 um and so I think that when Facebook started to do Source control uh they had this idea that well we want to be able to version everything together and I think there are a lot of advantages to that I think one is it makes collaboration a lot easier it makes Discovery a lot easier um I think it makes it easy to put dependencies between different services so I think all of us have had that case where it's like oh well I need to change but that also means I need to change the call sites or that also means I need to change this other thing um monor repos allow you to do that right because what it's saying is there's one like unified state of the world that we can look at can make sense and everything's good we can test against even uh poly repos the opposite of that when an organization has many many or uh repositories I think have different advantages um you can version them separately you can uh have different practices within them you can uh have different entirely different owners within them I actually think that GitHub on many companies pushes a culture of poly repos because it's what's open it's what's common in open source right so again going back to this Narrative of sort of like oh open source is different than closed Source I think in open source you actually want things like I don't want to have to version at the same time that all of my dependencies are versioning on that doesn't make sense I'm not really collaborating with my dependencies yes I depend on other open source projects but perhaps I'm not contributing to them I think within a company those constraints change right um Within a company you're looking at constraints of like I want to collaborate with other people I want to be able to version things and so you get pushed towards sort of like one unified source of Truth for like what is the state of our code base now uh by the time I was there Facebook wasn't even one monor repo it was multiple M repos I remember hearing someone refer to it as a poly lith which made me laugh a lot uh where there was the main web mono uh there was Instagram which was separate there were the mobile apps which were separate than that there was tool Le yeah which was completely separate and they were on a mission to try and unify these into all of them because into one massive monor repo because the negatives of not having one monor repo were starting to affect the company some examples of that might be that if you wanted to do an end test on mobile uh apps I remember there was some test that would pull down a copy of the other repository um spin up a server from that repository and then spin up the mobile app such that they could test against that and that was that led to all sorts of conflicts right like you're it makes your CI flakier it makes it hard diversion against when we had dependencies between repositories that was a nightmare and so by bringing Everything Under One Roof it allowed you to really uh collaborate across the company more actively and uh it allowed you to collaborate across the company more actively and allowed you to share uh dependencies so like face Facebook was being slowed down despite having you know a few Mega repos if you will or these large ones so so they saw the disadvantage in the end they they they move to one unified monor between iOS Android and even web when I was leaving they were on the journey to um I remember hearing recently that they're still on the journey to that's that that's really it's such a different code especially for Native mobile but yeah and but the advantage of it though and again going back to it is I think that it allows you to enforce a common set of end practices and uh requirements right like the idea of like oh every repository should have at at least one One reviewer right in a company yeah that probably is a companywide requirement uh but in open source like I can't be guaranteed that every open source project wants the same set of requirements and it's interesting because it's not just meta like other other companies when they grow a certain size they start to start have the same Journey you know I've seen it at Uber as far as I understand you know Shopify has is is is doing similar things I assume that the as companies go larger they'll have these typically starts with like oh what about a vulnerability of a Library being updated would it not be nice if it was in one place now you've you've seen ## The advantages of monorepos and why GitHub does not support them [35:31] more of these through obviously through with with graphite when you're interrogating with with some of these companies do you see some trends of like when monores are starting to become important and and if if they are starting to become more important why do you think we're not seeing GitHub support too much of this still it's a great question so uh first and foremost I think that we're watching an industrywide move towards monor repos most companies we're talking to are already in a monor repo or migrating to a mono at least for the people that we talk within Silicon Valley a lot of people have decided you know this is actually the future I see the advantage the tooling there someone should be moving here um and so we're actually seeing a broader sort of like Trend towards monor repos why do I not think GitHub supports it I think it doesn't make sense for open source I think we're just starting to see the like uh Advent of monos in open source I think react is closer to a monor repo than not I think that versel follows the same pattern probably a lot of the same energy there um I think we're seeing a lot of repos say like well maybe multiple packages should be in one repo um but there are a lot of advantages to have to being able to put them all together um but I still think it's not the norm right when I when you look at like the JavaScript ecosystem or the rust ecosystem or the python ecosystem we still live in a world of many separate packages for many different authors with many different cultures and the way that manage those is through poly repos yeah and then when you say you know you see a lot of companies having can you just get a sense of what types of companies in terms of the the size the the funding stage you know like I appreciate you might not be able to say specific names here who do not advertise but it was a pretty surprising for me to say that a lot of the companies you work a lot of them are moving no um I'd say that graphite tends to work with companies with Silicon Valley based companies that are somewhere between call it like 100 and high thousands of people maybe 10,000 people and so for us I think it's a lot of the uh forward thinkers who are really investing in developer velocity as you were talking about earlier that are starting to invest in the like how do I speed up my team are very quickly realizing that the coordination overhead of poly repos um makes it harder to speed up your team it both slows your team down in code review in weird uh Insidious and second order effect ways um and it makes it hard to enforce a common culture right so when we talk to De tools teams and they're like oh we want to uh migrate CI when you're in a monor repo that is a that is a somewhat straightforward operation when you're in a poly repo that is a multi-year journey that you're on oh yeah and yeah I I I think I think I can emphasize with this yeah and and I think what you said like it needs to be some investment like it it doesn't just happen like this there I mean there's better tools now but you need to have like you dedicated people uh looking at vendors who are doing it Etc so probably makes sense that the the the teams who are investing in better tooling are probably the ones who are looking at these Alternatives now and that there's probably some selection effect in who we talk to yeah right of when teams when teams want to invest in tooling they start talking to us and at that point they've already started to consider them monor repo Journey no well I mean you don't need to tell me because I've seen the benefits of monor repo I mean I see the costs as well but it was so good at at Uber so many benefits and like I would you know if I had a magic wand I would say let's let's zoom on a repo obviously there's a cost so I I see this I'm just surprised that that it Act kind of an encouraging way that there's so much of this happening I think another place where um things got confused let's call it is that people associate U microservices with many repos and I think again that's because that's a way that um I think GitHub very much encourages you to think of like oh you have abstractions between code you should have different repos and I think that I I have a friend um also from Facebook who likes to joke that uh the discovery of monor repos is really just the discovery of folder structure where it's like well yes you have multiple you have multiple uh different pieces of code but you can just put them in different folders that's okay you I think you know the probably one of the best examples Uber is known for advertising how they have 5,000 plus microservices and they do have monor repos I think per language like four or five but yes you you can it's it's not dependent it's totally independent from one another exactly right now one other Trend that's happening now everywhere especially with the Forward Thinking companies is uh AI tools for code ## AI’s impact on software development [39:55] generation or for coding how do you see this changing how code is written uh both through the companies that you work at or maybe even at graphite it's incredible it's been a whole conversation at graphite and it's definitely been a whole whole conversation with a lot of the companies we work with I think the um most immediate impact of how will AI impact software engineering is there will be more software written right there is going to be more code written as the tools make it easier and easier for developers to ship code I think what's uh not as obvious as the second order effect right if you have more code written well what needs to happen well more code needs to be reviewed and if more code's reviewed more code needs to be tested and more code needs to be merg and more code needs to be deployed and I think that's suddenly the place where we're starting we're seeing people run into issues where they say okay I think that I think that we're ready to adopt one of these AI tools that helps us generate code that helps our developers code faster um let's do that and then what immediately happens is we see the volume of poll requests go up we see the size of those poll requests go up we see the amount of bugs or bills also go up and I think getting ready for that transformation is a thing that the industry is just starting to Grapple with this episode was brought to you by centry buggy lines of code and long API calls are impossible to debug and random app crashes are things no software engineer is a fan of this is why over 4 million developers use Sentry to fix errors and crashes and solve hidden or tricky performance issues Sentria debugging time in half no more Soul crushing lock sifting or vague user reports like it broke fix it get the context you need to know what happened when it happened and the impact down to the device browser and even a replay of what the user did before the error centry will alert the right div on your team with the exact broken line of code so they can push a fix fast or let autofix handle the repetitive fixes so your team can focus on the real problems sent your help monday.com reduce their Errors By 60% and SP up time to resolution for NEX door by 45 minutes per Dev per issue get your whole team on Century and seconds by heading to sentry.io pragmatic that is sen n y./ pragmatic or use the code pragmatic on sign up for three months on the team plan and 50,000 errors per month for free now you know this is speculative ## The problems that AI creates, and possible solutions [42:15] but it's it's good to speculate sometimes how do you think successful companies will cope with it and this is we we can just speculate right like we know there's going to be more code being turned out devs will be able to generate stuff faster build features faster and yes it will have all these a lot of them will just tap tap tap except they're going to not notice certain things being there we even have like attacks of like you know some companies trying to have malicious B like you know stack Overflow copy paste and you have the memory leak will have more of those what you know let's just do the thought experiment like how how will good teams deal with this I think AI necessarily is both the problem and the solution so I agree with I think that the fastest moving teams are going to adopt AI I think we've moved past the world where where you can say like ah this is a fad this will pass AI is going to mean a lasting change on the software development industry period I think that you're right it's going to create some problems right you have developers just clicking tab non-stop probably importing code that they neither we already have it right it's called Vibe coding yeah I keep I I love the Twitter memes about it but yeah Vibe cing is a thing right you have developers at your company who are probably just talking to their computer telling it like yeah create this website no I'm not going to read that code and they're putting up for code review and I think that to me that just emphasizes the point of code review because if the author didn't read the code then someone else certainly should and the question is how are we going to make sure that happens and so I have two answers for you one is I think a lot of the practices that already existed at these major companies at the Google and Facebook and to a lesser extent um but still an extent Uber Twitter Snapchat um which are used to seen a high volume of changes um written by a diverse employee pool um a lot of those practices are going to percolate down because those used to be the problems of like well you only run into this issue if you have 100 developers we see it on Twitter people are saying teams of 10 can start acting like teams of 100 developers and so that also comes with the problems of hundreds developers the second solution is I think that AI is going to transform the way that we view code review I think the uh the largest change which has happened uh something which we are trying to trying to uh keep up with internally is that I think we're seeing the same Advent that happened when something like grammarly or the grammar checker in word happened where AI can actually check a lot of the minutia or mechanics of code review it can tell you that like yes this code works or yes this code does what it intended to do the question though of code review then isn't is this code literally bug free it's is this aligned with what we wanted it to do right is this I I originally we spec out the system it was supposed to work this way um or maybe we have built the system and it does work this way what are the other effects of that how does this interplay with everything else we do at this company what's going to happen there and I think that we're going to see AI tools built we're we're building one ourselves our AI reviewer that let developers um automatically check this code for sort of like the the simple stuff allowing the reviewers to focus on the more complicated stuff do you think testing will become more important in this in this world where we ## How testing might change and the testing AI coding tools are already capable of [45:25] we do have like a lot more AI tools a lot more AI code a lot more you know pushing to production I think testing's going to change and so I I is it going to is it going to so is it going to become more important probably in some form definitely because as I said I think that de if developers are not putting thought into every keystroke but instead we're just seeing like this code written quickly checking that code for correctness both the like literal like does the function work as intended and the endtoend correctness of like are the buttons in the right place and is it usable or critical now I think what's really interesting or the trend that we're seeing is that it appears that AI can actually write tests for you in some limited uh cases and when we look at some of the more advanced AI agents they actually do this as part of their coding right if you ask an agent to like generate a generate a change and also test it it'll do a pretty good job um and I think that a lot of the previous types of tests that before were uh restricted to QA people or to humans of like look at this website and try to like click around it with the mouse where before we had to use uh we had to use an anend testing tool it was hard we'd kind of be like oh go click at like XY location this you can we even hired you know like testers who you we call it sanity testing you describe what to do go there take a credit card and like that was cuz I was on payment team like we actually had some of the fake credit cards that take take one of them enter there and we did it because we couldn't really automate it for different reasons the payments is a bit tricky but yeah it was it pretty terrible job to do but someone had to do it and someone was being paid so exactly and it's uh I think that um again the the machines can start to do a lot of this work I think the question that we have to Grapple with then is so what is the purpose of this review step right why and I think that for a lot of people uh this is a question we used to talk about like three years ago and I think with time with advantage of time we've become a we've had a lot more clarity of like the purpose of code review isn't just the mechanical it's not just hey can you proofread my essay for me because I want to make sure there no grammar mistakes in it I think there's also a type of shared learning that happen of hey I want other people to code review this because I want to distribute my knowledge of how the system works and I think that there is an alignment checking that happens of hey there's uh this is how we build software here and there's tribal knowledge that maybe more senior staff level engineers at your company have that they disseminate through the process of code review and I think that as uh we take away these mechanical parts or remove their reduce their need um you're going to see those lighter two become a lot more emphasized I am a little worried though about there ## How developer accountability might be a way to solve bugs and bad AI code [48:15] is a big we do see that AI tools are very capable of generating code and I think you know like experienced developers who've been been in the industry for 10 plus years and have SE for example the they work with the teams of Junior engineers and you see what happens you know it's a bit of a mess then you need to really clean it up like they I think you know they see that this has limitations right like it's it's a very powerful tool when you know what good output is when you can check it when you can step in and say this is BS stop or or or you just like stop prompting after a while because there's no going anywhere and you start take over and you just write whatever you need to do but I I do feel that people over index on how good a generat code and I have a feeling that a lot of companies are or especially you know like company leaders even CTO they'll they'll kind of ignore this part of of learning cuz it's invisible right like you you don't really see the difference between like two teams one uh there's no code review or just AI code review everyone's doing their own thing and you know and they're they're both shipping for a while and at some point something breaks and this team where people are actually doing code reviews they're thoughtful they understand the code they can start you know nothing stols down they build the new features they in innovate whereas this one kind of gets stuck there just bugs no one knows why but because it's not visible like I'm I have this feeling that there there's there might be I don't want to like you know be all negative about it but it's it's just so many steps removed from from the work that I I wonder if like few companies will appreciate it a bit like how few companies used to appreciate developer tools you know like it's only fashionable these days right like like if you think about it 10 or 15 years ago like everyone was looking like why is Google having a dedicated team to build internal death tools that's I think it became popular with uh I I'll give a lot of credit to Heroku I I think I remember just like growing up in that era there was uh there was a lot of love for like oh this is what good can look like you know and I think it really set a standard of like oh we don't have to accept worse we can actually we can have better tooling sort of like in the popular opinion and then a lot of Heroku developers went on to go to companies brought a lot of that energy and feeling with them and then we started to see it seated there um but yeah I remember I remember when Google uh there was a lot of sort of like question marks around like what is Google doing like why why would you invest so much in this and meta too um I think that where we where this will be different with AI is the fact that um at the end of the day code has really profound business opin business decisions right like if you if you decide that you want to give a refund or you want to charge this or the user experience wants to look like this or this is the way the product works that's at the end of the day a business decision and I think that what leaders will agree with is that they don't want the machines making bus decisions and so at that point at some level that means that a human needs to be revie it and now now the question is sort of like what is that level and what is that granularity that we review on but I think that we can we we can we can find a like we can find a common understanding and like okay humans will need to at least check for intent the same way humans prompt and and that's not to say that like prompting b or it doesn't work it's more that just I think fundamentally like human natural language communication is flawed right like you I have a conversation and you can understand something different than what I mean and so at a company the way we get around this is we then check on the other side right we align at the start and we check at the end uh and I don't think that's going to go away yeah I I like this thinking and also I wonder if we might have more companies discover again I'm just a little bit inspired by Linux as I recently talked with uh with GRE from there that the way Linux works you know what I when I talked about how the whole project works I was really surprised in how much it's about people and the core the core concept there is when you take a patch they call a PR a patch there you now own it and you are personally responsible for it and this actually you know if something goes and this is how there's a pyramid of like subsystem owners system owners and then in the end there's L tal so every single release for Linux it goes through it still to the day it goes through him he owns it uh and you know he's he's very happy to take that responsibility but and you know Greg was telling me that because of this like if you think about it and it's very open the whole world sees your changes so because of this uh there you know people take pride in this they will both review but there's also trust levels right so if you're a system owner and there's a subsystem owner you work now for 10 years you know this person is solid you're not going to spend all the energy and I wonder if you know the the correct response to AI is like yeah just go wild use these tools but responsibility maybe at least to go back to developers like you know like so we we'll see I I suspect that is what we'll see um and and as I said that's not to say that AI is not going to solve real problems I think AI will actually free the humans to focus more on those bigger picture things I think AI is going to be able to like take away the like oh does this function say do what it intends to do because they can understand that it can tell us that but I think it's going to be a tool and then saying and so does this holistic change do what you wanted to do yeah so speaking of change you ## Why stacking hasn’t caught on and Graphite’s work [53:20] know one thing that I I was really surprised that when I worked at Uber I talk with other companies and stacking didn't take off anywhere else cuz fabricator was just not there what why do you think that didn't take off and how did you get the idea to start graphite around this idea of you know building this tooling to to make it accessible for a lot more people yeah of course well so I'll start by saying I actually don't think that's stacking caught on inside meta at the start oh so um originally when it was built um it was built into Mercurial there were uh it was advertised about the company there were a few senior Engineers who saw the vision felt blocked and picked it up and I think that not in an exaggerating way they saw themselves get 3x faster and what happened was those senior Engineers started to become evangelists I had one on my team who was like the stacking evangelist and they used to go throughout the organization and literally give PowerPoint presentations of like this is how stacking speeds you up like you need to believe me I've seen the other side and at the time I think you had a lot of developers who were very skeptical who were like like I've been doing it for so long like I I don't need you to tell me how to change my workflow why would I do that I I hear all the time from current you know people who you're about stacking yeah and then like what ended up happening was that as as these evangelists like made inroads and started to convince people like no no no it's actually speeding you up um people would adopt it see the value and and like become that evangelist themselves we like to joke that once you stack you don't go back um I I think why it worked so well is it did two things really it was like the perfect uh convergence of two forces one is that it gave individuals what they want so I think no developer likes being blocked right if you ask a developer like what is the feeling towards code review um or towards having to wait on code review it's almost always negative no one's like I love that feeling negative people are like it sucks it's the worst part and I think that um even if you have a more senior developer that's been doing that's had code review for 20 years if you ask them like why do you enjoy coding on personal projects as opposed to company project the like the overhead of collaboration becomes like quite clear there of like yeah cuz in like when I work with others things just slow down we need to review things we need to align on them they get a lot slower and I think what stacking did is it took away some of that overhead right where rather than having to wait on other people to review your code that you already thought was pretty good and you had already aligned on before you could just keep going and keep Building without needing to without needing to sort of wait for them the second side is on I guess it made it was great for three parties the second side was the reviewer the reviewer started to get smaller PRS right and whether you whether you know it or not when I send you a 10line PR you are so much happier than if I send you a 2,000 line so so many companies eventually when they write down Cod review guidelines you're always going to have small peers you know at Uber at some point we had like recommendations within the team for 50 or 70 lines whatever that is but you realize I mean there's jokes going around and I think it was viral on social media of the you know the 5,000 line preview looks good to me and then the 10 line which gets like 10 comments exactly right and I think people as we were saying before people buy out of it and and the third is I think it really gave the company something they needed which is that they started to see their or engineering organization speed up they started to see time to merge go down that's usually a metric that people measure and we can talk about engineering metric separately um and uh in the case that something went wrong it made it much easier to correct things right in the case that QA or even users like found something that wasn't working the way it should be it was much easier to then go to history and be like which one of these changes did it than when the changes were like monolithic feature monolithic feature monolithic feature yeah now you know you built this tooling and you are still a ## Graphite’s origin story [57:10] pretty small startup how did you convince companies to to you know like use a pretty small startup pretty unknown methodology you know rumors are meta uses it a lot because I assume it's not a really easy sell is it no it's not an easy sell at all um well so back to the company's Origins we didn't so I don't know if you know this we didn't actually start as a code review company uh so we actually started by building tools for mobile releases that's the background of the founders that's the area we wanted to like build in um I uh I was fortunate enough to have two of my teammates from meta join the team and I think one of the things that the three of us felt was like oh like the tooling out here I think they were both new grads when they had joined meta had been there for a handful of years call it like four and then when they left they had the same experience I did which was like oh like the TS out here is different and at the start we kind of were like well you know we came from we must be the weird ones and after about like a a few months we like h no like there there's actually something missing here and I think the uh the context that I now have that I wish IID had then was one most companies if you have more than 50 people you have at least one person who builds tooling on top of GitHub and I think that if you've worked at one of these companies you kind of just take that as a truth of like yeah I have the dev tools team devat Jin but these days Jenkins is GitHub actions so same difference exactly you have someone building tooling there to make your developers faster the second thing is that these companies so like meta and Google so GitHub released their po request page back in 2013 uh they haven't really made a ton of major edits to it since uh those companies uh had their changes then and have continued to iterate on their tooling for like the past decade and so when we say their tooling is like a decade more advanced that's a quite literal statement actually that's not a figurative one and so we realized that we were like hold on like there's missing tooling out here we tried a lot of the Alternatives as I said stacking was sort of that first paino and so we tried a lot of the sort of like open source CLI first Alternatives realized it it was ano it was like a approximation of what we wanted but it wasn't quite it um and then we built it uh and then what happened actually was uh a lot of our old co-workers many of whom had gone on to startups ended up asking us like hey like code review out here is is different like how do you all deal with it and we're like we we built that um let us know if you ever want us to tell you how to build it and they're like I I don't but like uh can I just use your thing and after we heard that for a few months we ended up pivoting back in I guess this would have been November of 2021 okay so quick pivot in the beginning of the just so Bic did you just feel a poll like okay like this is a good idea seems like there's a man for it uh I don't think it was uh the the other company IND was working quite well I think what we felt was we had all of our friends asking us for this thing that we had built and it wasn't the thing that we were selling and you can only listen to that for so long before you decide to say maybe maybe we got this one wrong maybe this is the thing it's it's just fascinating I guess it's hard to appreciate if if you're not working at a startup cuz you know my my imagination of joining a startup I I have friends that my brother was working at a startup but I didn't join a super early stage startup ever is you would imagine you're joining this is what we're doing this is what we're going to do but you know sounds like you need to be flexible enough because you might do something different that actually works and there's a bigger demand for it yeah we were fortunate enough that at the time that we pivoted we were just six people and so it was six people and uh three of us had felt the pain from Facebook so it was us telling people like no no no we've heard it like our our co-workers have heard it we' gotten other people at the company to buy into this workflow at that point and yeah it was certainly uh stressful but it was also uh it was also a thing we just had enough conviction of we're like this is the way the world will build software it's more a question of who's going to bring it into the world not if so we you previously mentioned injuring metrics what are injuring metrics you see these ## Engineering metrics that matter [01:01:20] fast moving companies measure and actually care about and and want to improve so I'll start with with uh I'll start with a statement that all engineering metrics are a proxy right so I think that me measuring engineering velocity is an art not a science and I think that the value of engineering metrics is to uh create some holistic metrics so that you can see uh major outliers and be like what's going on here um the kinds of the kinds of metrics which I see most people measure are number of poll requests is just a proxy for sort of like how is velocity doing uh time to merge of poll requests and then my favorite metric I've actually ever seen came out of uber um and so maybe you might have more context on this than I but it was basically uh time that a poll request is just spent waiting in review with no clear action and so the idea was like if an author puts up a PO request and it's just waiting on review you take that then the reviewer goes ahead request changes um and like it's uh that you then ignore that time uh you then put it up back up for review you start summing that time again and you end up with this composite of like this is the amount of time this for request just sat waiting on review without clear next steps and so this metric was introduced after I left Uber I I think I might have covered it in one of the the Deep di on how Uber did it but like this was a really big problem at Uber because we were distributed sites and like my team specifically had this problem where we would have to wait almost 24 hours to for some things to happen in the PO request cuz you know we had the we had strong code code ownership and I think Uber still might have it to this day I'm not when I was there we still had it so the the mobile platform team would own the platform code in San Francisco and Amsterdam you know at last thing last last thing at night you would make a change and you would have to head off and it would go to them they would rejected because of whatever reason you come back in the morning you know it's now night time in amster in in San Francisco you fix it quickly and then you have to wait another like eight hours and then stay late so like I'm glad that they start measure it because and this is one of the things where it's not really a big deal if you're a startup if you're in the same time zone if you're in the same location but and then you know we came up with creative workarounds like we we started to get like we onboarded someone to be an owner of Etc you can do a lot of things but it was really killing us like and I knew it because I I was the manager of the team and this was the number one complaint so we actually you know we we took it into our hands and we solved it but so many teams were doing this independently and then it kind of goes back to like you know like as as a Dev I didn't really like to hear too about strategy whatnot but there is a thing of like you know who owns what where how do you place your teams it's kind of easier when you have teams working on similar stuff close to each other in terms of time zone so so yeah and you know one other thing that Uber measured which I'm not sure how successful it was but it was an interesting thing is it they tried to measure Focus time on how much uninterrupted Focus blogs defs had uh I I think it's really Noble it's just really hard to measure but it's true that when like I was always happier like developing when I actually had time it wasn't fragmented with like you know the random meetings here here or there Etc yeah totally I I mean maker schedule manager schedule is super real I I think that as we were going back to the before though these problems which used to be more isolated to big companies I think as we start to see a higher volume of poll requests and more of the time is spent sort of like in code review not authoring poll requests uh they're going to become everyone's problems and I think the question is sort of like how ready are companies for that change yeah it I like this you know thinking because this this will happen so it's interesting how SM small companies might need to take some of the lessons from the the larger companies on how they solve that and prepare for it because it's it's going to come like I'm I'm kind of tempted to agree with you on this one it it'll be fascinating to see It'll be also fascinating to see if small companies speed up a lot more than let's say big companies where there's larger teams and and there's now more dependencies it'll be interesting one for us to I think small companies are certainly better in uh position to embrace the change where I think big companies will be slow I think small companies we've already seen them adopt a lot of AI tools faster um I think some of the like most advanced big companies are starting are like just trying to adopt them at the same pace of these small companies but they're definitely not trying to outperform them uh and I think we've seen this actually in quite in quite a few areas right like every time that you have a new technology that makes something uh more accessible easier the same way I think AI is making coding more accessible easier I think what ends up happening is that like practices that were previously only meant for like the the tippity top of the market start to distill their way down to everyone so you you you're a co-founder of a def tools company and I think this ## Learnings from building a company for developers [01:06:07] is a little bit of a dream of a lot of Engineers as you see this this pattern pretty much is you you work you join a company you work on a product if it's a big enough company there's a platform team a lot of Engineers gravitate to there because you can solve pretty cool problems in your case you actually did the you were on a def tools team and then uh you you you find the courage or find the right people to actually start a startup and what better thing to do as a Devon what you know you you already know how to do this is built for other developers what is something that you've you've kind of learned the the hard way of starting a de tools company because I I assume it's it's a bit trickier than than what you might what was different than than when you started out because it sounds like the dream right you kind of went out you actually managed to raise some funding and now you're going to build these cool tools for developers and you're a developer is great you know your customer is wonderful um I mean a ton of things uh I think that building a company can be very rewarding in a lot of ways I think there are a lot of challenges I think some of the things that you learn to deal with uh some of the things I think we've learned to deal with here are sort of the uh diversity of engineering workflows across even across uh Silicon Valley and non Silicon Valley style companies right of like what does code review mean to them what are they trying to get out of it what's the purpose of it and finding the uh the Middle Road to blend all of those uh to blend all of those uh I don't know needs desires Etc uh is difficult it's challenging it's something we build um I think one of the things that has most surprised me coming out of Facebook uh is is what we talked around earlier is is the tooling is the I think working within a company you probably have that team that is building really awesome tooling you know and like they really benefit from that when you come out here into the wild as I call it a lot less of that exists and you find yourself being like okay how can I set that up how can I get it back how can I get back to that um a company which I uh I'll give a shout out to is stat Sig um so Facebook had really wonderful sort of like um had really wonderful uh experimentation tooling feature flagging tooling and when I left immediately not immediately but like 6 months in I was like okay cool like we probably have our first like feature flag like how do we go ahead and turn this on and I think you just mentioned your at Uber you had wonderful tooling there too uh there's all of this stuff that's set up for you that you just take for granted and once you come out and have to build it you have to stly start to be like okay H how do we do this from scratch again you know or who can do this for me awesome so with that let's wrap up with some rapid questions I'll ## Rapid fire round [01:08:41] ask a question and you just shoot whatever come comes to your mind cool so first what is a tool that you saw inside Facebook that is now outside Facebook as well and that can be used and you think it's pretty cool I think stattic is still the answer I'm going give here it's just it's very cool having the feature flag system as it's used to and it's actually an amalgamation of a handful of different Facebook tools um our feature flagging system gatekeeper and then our experimentation feature uh deltoid and so it's cool to see both of those and QE2 I guess it's kind of cool to see that sort of like uh reborn outside of outside of Facebook that's awesome and this this is the beauty of like uh I think you know there there was this like several years where Dev tools starts got funded because I feel a lot of the things that were hidden inside these companies are now outside as well totally um I think also one of the things that's uh having worked at Dev tools within Facebook and out of Facebook I think one of the things that's so funny to me is that in Facebook building Dev tools is a lot easier you don't need to worry about things like marketing you don't need to worry about things like hiring and I think it takes a lot of uh courage and it's a very difficult path to build a tool outside of outside of one of these large companies and say no I'm going to figure out how to make this work for the market because again the engineering cultures are different the constraints are different the uh the other tools you're integrating with are different and so you have to start to rebuild all that and so what is your favorite programming language always an interesting question by far Love T I think typescript is one of the most beautiful languages and you're talking to a person that learned JavaScript when he was like 10 um and then the type system on top of typescript I think is is really clean and does a great job with the constraints they have it it was I remember when typescript was created and I I remember there was a bit of skepticism around a it was Microsoft B do we even need this because you know the dynamic typing system is kind of like this wonderful thing that allows you to do so many crazy things and yeah like I think everyone has turn well not everyone but like most devs have have slowly or or surely seen the benefits of what what what we get but with with typing and with a nice language my my controversial opinion is I'm glad Microsoft is want to do it because I think that Microsoft has some really wonderful compiler knowledge locked inside of it um and I similar had that ISM I remember at the time thinking like we have flow what what's the purpose of this um and I think that the the formal grammars and proofs that they've done around it are really really excellent the like you can go really deep on group Theory or category Theory just by looking at typescript um yeah I I there I frequently look at other untyped languages I'm looking at Ruby and I'm looking at Python and I I don't think they've gotten close to what typescript has done to JavaScript yet and then finally what's uh a book that that you read and would recommend fiction or non-fiction one or two books so I am an avid mystery reader so the mystery book I always recommend people is the last days of night it's a book around uh a Tesla Edison and wesing house and the race to invent the light bulb it's written as historical fiction I think it's phenomenal as one who currently Works in technology in New York there isn't a better book I can recommend to anyone in that position um non-fiction um I read a lot of books these days I've been actually reading a lot more prodct books um the Timeless way of building is a really interesting book about architecture the beginning is is um a little abstract to get through but once you get to the sort of Latter half it talks around the importance of uh having the people who ultimately use uh tools or buildings be the ones who build those tools or buildings uh because that's the only way that you create a self- sustaining cycle I guess it kind of resonates with with developers developer tools yeah which see seems kind of obvious cuz who else would program the tools that we used on developers it's one of these weird things but yeah there's an mcer painting to be made about that well this this was this is really interesting I'm glad we were able to go a bit deeper into stacking than we previously touched on on on that uh ## Closing [01:12:41] article as well and thank you for your time cool of course thank you this was always very fun thanks a lot to Thomas for all the details on metas tooling stag diffs and on tooling Trends you can find Thomas on social media as list in the show notes below for more deep Dives on tooling at meta and more details on sag diffs check out the pragmatic engineer deep Dives Linked In the show notes below if you enjoy the podcast please do subscribe on your favorite podcast platform and on YouTube this helps more people discover the podcast speci special thank you if you leave a rating thanks and see you in the next one