WTF is happening at xAI | Sulaiman Ghori (YouTube Video Transcript)

(00:00:00) Your YouTube transcript will appear here (00:00:00) Tyler took this bet with Elon like get a (00:00:02) Cybert truck tonight if you can get a (00:00:04) training run on these GPUs in 24 hours (00:00:06) and we were training that night. (00:00:07) >> Did he get the Cyber Truck? (00:00:08) >> Yeah, he got Cyber Truck. (00:00:09) >> My first day they just gave me a laptop (00:00:11) and a badge and I was like okay now (00:00:13) what? I don't even have a team. I've not (00:00:15) been told what to do. Ask Rock was (00:00:16) spinning up at the time our integrations (00:00:18) with X. They're like can you help? And I (00:00:19) was like yes. (00:00:20) >> What's the most fun thing about working (00:00:22) there? (00:00:22) >> No one tells me no. If I have a good (00:00:24) idea I can usually go and implement it (00:00:26) that same day and show it to Elon or (00:00:28) whoever and we got an answer. We did the (00:00:29) math. Right now we're I think at about (00:00:31) $2.5 million per commit to the main (00:00:33) refill and I did five today. So (00:00:35) >> you added like 12 and a half million of (00:00:36) value. (00:00:37) >> The levers are extremely strong. (00:00:41) >> Today I have the pleasure of sitting (00:00:42) down with Sully Kongori and he is one of (00:00:44) the engineers at XAI. I've been kind of (00:00:47) fascinated by XAI since like 2023 when (00:00:50) like Elon first started. I think it's (00:00:52) like one of the fastest growing (00:00:53) companies of all time. Can you just talk (00:00:56) about like what the [ __ ] is happening at (00:00:58) XAI? (00:00:59) >> Yeah. Um, we don't have really due (00:01:03) dates. It's always yesterday. (00:01:04) >> Um, (00:01:06) there's no blockers for anything like at (00:01:09) least nothing artificial. Uh, the whole (00:01:12) Elon thing about going down to the root, (00:01:14) uh, the fundamental, whatever the (00:01:16) physical thing is, we get there pretty (00:01:17) quick if we can, as quick as we can, (00:01:20) which is funny in software. It's not (00:01:22) really like a thing that you think about (00:01:23) is the physics too much, but we do try (00:01:26) quite a bit and we're not really fully a (00:01:28) software company given all the (00:01:29) infrastructure pull down. Um, (00:01:31) >> it's kind of hardware at this point. (00:01:32) >> Yeah, (00:01:32) >> it's like hardware constrained. (00:01:33) >> It's the probably our biggest edge is is (00:01:36) is the hardware because nobody else is (00:01:38) even close on on the deployment there. (00:01:39) Um, [clears throat] (00:01:41) although the talent density on software (00:01:42) is like incredible. I've never been (00:01:44) anywhere like that. It's it's really (00:01:45) cool (00:01:46) >> for Elon. he is very good at figuring (00:01:49) out like what the bottlenecks will be (00:01:50) even like a couple months or even years (00:01:51) in the future and then trying to work (00:01:53) backwards from that and make sure that (00:01:54) like he's in a really good position. Um (00:01:57) how does that work dayto-day with just (00:01:58) normal people like at XAI and like (00:02:01) adopting that kind of mental framework? (00:02:02) Usually when we spin something up new (00:02:04) very quickly either one of us or he (00:02:06) comes up with this uh metric that's (00:02:09) usually very core to (00:02:12) either the the financial or the physical (00:02:14) return or both sometimes. Um and so (00:02:18) everything is just focused on driving (00:02:20) that that metric. Um there's never (00:02:24) like a fundamental limitation to it or (00:02:26) like whatever the fundamental limitation (00:02:27) is it better be rooted deep down and not (00:02:30) something artificial. Um and there is a (00:02:32) lot of (00:02:34) uh perceived limitations um especially (00:02:37) in the software world coming from like (00:02:39) especially in the last 10 years of like (00:02:40) web dev and all these kinds of things. (00:02:42) People just assume or accept (00:02:44) >> certain limitations especially when it (00:02:45) comes to speed and latency (00:02:47) >> and they're not true. um you can get rid (00:02:49) of a lot of overhead. Like there's a lot (00:02:51) of stupid stuff in in the stack and if (00:02:54) you can knock out a lot of that, you can (00:02:56) usually two to 8x most anything at least (00:03:00) anything invented relatively recently. (00:03:03) Uh some stuff not so much, but yeah. (00:03:05) >> When was the last time that you (00:03:07) experienced this where there's some like (00:03:08) conventional wisdom that says that (00:03:10) there's this is the timeline and then (00:03:11) you guys just were able to completely (00:03:13) shred that? Um most recently it's our (00:03:16) model iterations on on macro hard. Um so (00:03:20) we're working on some novel (00:03:21) architectures actually multiple at the (00:03:22) same time and uh we're coming out with (00:03:26) new like iterations like daily sometimes (00:03:29) multiple times a day which is from (00:03:31) pre-train um in some cases uh which is (00:03:34) not something you ordinarily really see (00:03:36) but it comes from well a we have a (00:03:39) pretty great supercomput team and (00:03:41) they've knocked out a lot of the typical (00:03:43) barriers it takes to train a lot of this (00:03:44) stuff even with how variable our (00:03:47) hardware like the hardware is like it's (00:03:49) you know within a day of standing up a (00:03:52) rack you can usually be training (00:03:54) sometimes within the same day um even (00:03:56) within uh a few hours in some cases (00:03:59) >> and this is like not normal like (00:04:00) normally the timelines are like days or (00:04:02) or weeks (00:04:02) >> it takes a lot well in most cases at (00:04:05) least yeah in the last 10 years you (00:04:06) abstract this away and let Amazon or (00:04:08) Google take care of this um and so (00:04:11) whatever their capacity is is what their (00:04:12) capacity is but that's not like you (00:04:15) can't have that be the case and when an (00:04:16) AI now. So, the only solution is to die (00:04:21) or uh or build it yourself. (00:04:23) >> Can you tell me about like how what your (00:04:26) experience was like joining why you (00:04:28) joined and then kind of what the like (00:04:29) onboarding process was for the first (00:04:31) like couple weeks? (00:04:32) >> Yeah. So, um I was working on my own (00:04:35) startup when I moved to the Bay. Um and (00:04:38) actually during that time, Greg Yang, (00:04:40) one of the co-founders of XA, had (00:04:41) reached out. He's great at recruiting as (00:04:42) it turns out. Um (00:04:43) >> what did what did he say? Uh, so I got (00:04:46) an email and I thought it was spam (00:04:47) because it was I was getting a lot of (00:04:48) these like, you know, emails to founders (00:04:49) at the time of like, hey, you want to (00:04:51) chat or like I like what you're doing. (00:04:52) You want to chat, whatever. I was going (00:04:53) to mark it as spam to like delete it. (00:04:55) And I saw the domain x.ai. I was like, (00:04:57) oh, wait a second. I know these guys. (00:05:00) And they just uh I think it was probably (00:05:03) eight months in at that point. Um, and (00:05:06) so I was like, okay, yeah, let's chat. (00:05:07) And so we chatted a bunch of times. Um (00:05:09) then uh I wanted an aqua hire but uh I (00:05:13) think we were too early at the time and (00:05:15) that company kind of went mostly because (00:05:17) it was fairly obvious that you can't (00:05:19) build macro hard with like a million (00:05:20) dollars. Um but the uh idea was sound. (00:05:25) So I spent the next like six, seven (00:05:26) months wasting all my money um building (00:05:28) like aerospace projects and working on (00:05:30) uh an aerospace astro mining concept. (00:05:34) Um, that also I realized like probably (00:05:36) wouldn't work, but it was worth a try. (00:05:39) And so I emailed Greg again like uh hey (00:05:41) can uh like you want to chat again? He's (00:05:43) like yeah sure you want to interview (00:05:44) tomorrow. I was like okay. And um uh I (00:05:47) apparently did well and I moved on (00:05:49) Monday and I started uh then and it was (00:05:51) really great. Um nobody told me what to (00:05:54) do. So like my first day they just gave (00:05:56) me a laptop and a badge and I was like (00:05:58) okay. Um and I was like okay now what? (00:06:01) And so I went to go find Greg cuz I was (00:06:04) like, I don't I don't even have a team. (00:06:05) I've not been told what to do. Like Greg (00:06:08) just brought me on cuz I think he liked (00:06:10) what I was doing previously and it was (00:06:12) related to what the long term was for (00:06:13) macroard which wasn't really even a (00:06:15) project at the time. And I ended up (00:06:17) working on actually uh Ascrock was (00:06:20) spinning up at the time where our (00:06:21) integrations with X and so they're like (00:06:23) can you help and I was like yes I can (00:06:25) help. And so my first week was working (00:06:28) uh with the one guy. I found out very (00:06:31) quickly like everything that we built (00:06:34) like I could sit and I could stand up (00:06:36) from my desk which I didn't even have a (00:06:37) desk assigned to me. I just sat at (00:06:39) random people's desks that weren't there (00:06:40) that day. Um and I could point to (00:06:42) whoever built that thing at XAI um like (00:06:45) from my desk. It was very very very (00:06:47) cool. (00:06:47) >> And there was like almost no people (00:06:49) working there at this point. Just like a (00:06:50) couple hundred, right? (00:06:51) >> Uh yeah, about a hundred or so on the (00:06:53) engineering staff. And then I don't know (00:06:55) what the uh uh infra buildout team (00:06:57) looked like at that time. And it's kind (00:06:59) of hard to tell because some people move (00:07:01) up the ladder from like the actual (00:07:03) building and construction crew onto our (00:07:04) payroll. But um it was pretty small at (00:07:08) the time like much much like an order of (00:07:09) magnitude smaller than the other labs. (00:07:11) Um and we had still just done Grock 3. (00:07:14) Um (00:07:15) >> yeah, which yeah (00:07:18) was pretty cool. One of the things that (00:07:20) I kind of love um is how fast XAI went (00:07:23) from being founded. I remember Elon (00:07:25) initially saying like we're not even (00:07:27) sure if this can be a success with you (00:07:30) know people having you know a multi-year (00:07:33) advantage on on speed and like timing (00:07:35) and then you guys got done with the (00:07:38) first like Colossus data center in like (00:07:40) 122 days. Um and that was just like (00:07:42) unheard of and Jensen's out here saying (00:07:45) singing the praises of XAI and Elon. uh (00:07:48) what kind of culture did that allow to (00:07:51) be formed? (00:07:52) >> It definitely enabled like us on on (00:07:56) model and product to kind of assume we (00:07:58) would have the resources to do what we (00:08:00) needed to do. (00:08:01) >> Um and that's definitely the case. Like (00:08:04) we're not super duper resource (00:08:06) constrained. Like we've still found a (00:08:08) way to push up against that wall. (00:08:09) >> Um but that's just we have 20 different (00:08:11) things going at the same time. Like more (00:08:13) than that, like many more things than (00:08:16) that. there's a absurd amount of of runs (00:08:18) and training and all that stuff going on (00:08:20) at the same time in parallel usually by (00:08:22) like a few a handful of people. Um which (00:08:26) is how we're able to iterate very (00:08:27) quickly on on model and product side. Um (00:08:30) and utilization has definitely been very (00:08:32) high. The the speed allows us definitely (00:08:34) to I guess think more long term. Um, so (00:08:38) I think Grock 4 or five really what it (00:08:41) was was already planned out and and (00:08:45) designed in terms of size and what we (00:08:47) expected um way early like before I (00:08:50) joined. I joined around Gro 3. (00:08:53) >> So it's like thinking at least a year in (00:08:55) advance. you can yeah you can think much (00:08:57) more in advance and assume that those (00:08:59) estimates will be hit um just because (00:09:02) everyone's like pretty great and (00:09:03) reliable (00:09:04) >> which frees you up a lot in terms of (00:09:07) like what your limitations are I guess (00:09:09) so for us for example the assumed (00:09:11) minimum latency (00:09:13) was about three times higher than it (00:09:15) actually needed to be and the buildout (00:09:17) allowed for that basically um (00:09:19) >> what do you mean by that (00:09:22) >> so the one of the novel architectures (00:09:24) we're working on um is not really (00:09:29) possible unless you scale up your (00:09:33) experiment rate because it's it's not (00:09:36) building on any existing body of work. (00:09:37) You need a new pre pre-training body and (00:09:39) you need also uh a new data set but (00:09:41) that's not really constrained by the (00:09:42) resources like the physical uh (00:09:44) infrastructure resources mostly. Um (00:09:48) although there's the uh the Tesla (00:09:51) computer thing which I think maybe we'll (00:09:53) get into maybe not but um uh so actually (00:09:57) this one's public. So one thing that (00:09:59) we're thinking about is okay like we're (00:10:00) we're building this human emulator with (00:10:02) macro hard. Um (00:10:04) how do we deploy it? Because you (00:10:06) actually need like if we want to deploy (00:10:08) 1 million human emulators we need 1 (00:10:10) million computers. Um how do we do that? (00:10:13) and the answer showed up two days later (00:10:16) in the form of a Tesla computer because (00:10:18) those things are actually very capital (00:10:20) efficient as it turns out. And we can (00:10:22) run um potentially like our our model (00:10:25) and the like full computer that a human (00:10:28) would otherwise work at on the Tesla (00:10:30) computer for much cheaper than you would (00:10:32) in on on a VM on AWS or Oracle or (00:10:35) whatever or even just buying hardware (00:10:38) from Nvidia. that car computer is (00:10:40) actually much more capital efficient and (00:10:41) so it enables us to assume that we can (00:10:45) deploy much much faster at a much higher (00:10:47) scale. Um and so we've adjusted our we (00:10:50) adjusted our expectations for that (00:10:51) basically. (00:10:51) >> Are you basically able to just bootstrap (00:10:52) off of the like car network? (00:10:55) >> So that's one of the one of the (00:10:56) potential uh solutions basically. Yeah. (00:11:00) So like okay well we want 1 million VMs. (00:11:02) Um there's like 4 million uh Tesla cars (00:11:05) in North America alone. Um, and like (00:11:08) let's say 2/3 or half of them have (00:11:10) hardware 4. Um, and like somewhere (00:11:14) between 70 80% 80% of the time they're (00:11:17) sitting there idle probably charging. We (00:11:19) can just potentially pay and they have, (00:11:21) you know, networking, they have cooling, (00:11:22) they have power. Um, we can just pay pay (00:11:25) owners to lease time off their car and (00:11:27) let us run um like a human emulator uh (00:11:30) digital Optimus on right on it. and uh (00:11:33) they get you know their lease paid for (00:11:35) and we get uh a full human emulator we (00:11:38) can put to work. Um and that's something (00:11:40) without any buildout requirement. It's a (00:11:43) purely software implementation that's (00:11:46) required. Yeah. (00:11:46) >> The the asset is sitting there and you (00:11:48) can just go and use it. (00:11:49) >> Yeah. Amazing. What for the human (00:11:51) emulators uh in macro hard what is the (00:11:54) like purpose of that of scaling up you (00:11:56) know millions of many humans? Um, I mean (00:12:00) the basic con concept is very simple, (00:12:02) right? With with Optimus, you're uh (00:12:04) taking any physical task a human can do (00:12:07) and allowing a robot to do it (00:12:09) automatically at a fraction of the cost (00:12:10) at 20, you know, with 24/7 uptime. Um, (00:12:13) we're doing the same with anything that (00:12:15) a human does digitally. So any anything (00:12:17) where they need to digitally input uh a (00:12:19) keyboard and mouse inputs, which is (00:12:21) usually what humans do, um, and look at (00:12:23) a screen back and make decisions, (00:12:25) >> uh, we just emulate what the human is (00:12:27) doing. uh directly. So no adoption from (00:12:30) any software is required at all. Um we (00:12:33) can deploy in any situation in which a (00:12:35) human is in potentially currently. Um (00:12:38) >> interesting. (00:12:38) >> What is what is that actually going to (00:12:39) look like uh for rolling it out? (00:12:42) >> Um (00:12:44) I don't think we've detailed our plans (00:12:45) publicly yet specifically on uh on how (00:12:49) we'll roll out. It'll be slowly at first (00:12:51) and then very quickly basically like uh (00:12:55) like the difference for us given that (00:12:57) infrastructure buildout already has (00:12:59) happened or we can go on the Tesla (00:13:01) network or we can build out our own data (00:13:03) center Tesla computers actually. Um (00:13:06) the difference for us from from going (00:13:08) from 1,000 human emulators to a million (00:13:10) is actually not very big. It's not it's (00:13:13) not the biggest part of the challenge. (00:13:14) Elon, I know one of the things that he (00:13:16) does best is he basically just goes from (00:13:18) fire to fire on whatever the company is (00:13:20) and just kind of like puts it out and (00:13:21) unfucks whatever problem is exists. (00:13:24) >> Uh what has that been like? What when (00:13:26) have you like seen some problem exist (00:13:28) and just had it unfucked very rapidly do (00:13:30) this kind of process? (00:13:32) >> Um definitely on the infra build out (00:13:34) this is the biggest. Um on model side (00:13:37) we've been like we've had hiccups but (00:13:40) >> it's more or less been smooth but on (00:13:42) model side especially cuz there's a lot (00:13:43) of uh I mean infra side there's a lot of (00:13:46) very specific (00:13:48) uh operations that each of these (00:13:52) basically AS6 these GPUs are are built (00:13:54) for and when we roll out new products (00:13:58) like when we pick up new products from (00:13:59) Nvidia or whoever um not everything (00:14:02) works so in some of the meetings that we (00:14:06) had with him uh early last year. Uh he (00:14:09) would hear these and he would make a (00:14:11) phone call and the software team would (00:14:13) deliver a patch the next day and we (00:14:15) would work like side by side until that (00:14:16) was resolved. Um and then we could run a (00:14:18) model uh or a train training run uh on (00:14:21) the hardware uh very very quickly where (00:14:23) otherwise it would have taken weeks of (00:14:24) back and forth. So those kind of (00:14:27) blockers are usually very quickly (00:14:29) resolved with one phone call um or just (00:14:32) us bringing it up to him or him just (00:14:33) offering like frequently when uh a (00:14:36) meeting is ending or there's a lull in (00:14:38) in the conversation he'll be like okay (00:14:40) how can I help how can I make this (00:14:41) faster whatever and someone will come up (00:14:43) with with an answer (00:14:44) >> I know you guys are doing many different (00:14:46) products in parallel and I get that it's (00:14:48) kind of like you have to do that but (00:14:51) also it's sometimes in most (00:14:53) organizations it's like very difficult (00:14:54) to stay focused on a single thing and (00:14:56) like a single objective. How does that (00:14:58) kind of work uh for just executing on (00:15:01) multiple different fronts at the same (00:15:02) time? (00:15:03) >> Very frequently we actually uh and this (00:15:06) is increasing with scale. We don't have (00:15:08) a full picture until like the all hands (00:15:10) or we just chat with people what (00:15:12) everyone is doing and how far everyone (00:15:14) is on these different projects. Like for (00:15:17) example on on on when we did our our our (00:15:19) voice model and our voice deployment (00:15:22) >> um we actually had a lot of the work (00:15:24) built for extremely low latency uh (00:15:27) extreme low latency end like uh packets (00:15:30) to be sent to the client. it was already (00:15:32) built out and um it was a matter of (00:15:36) flipping the right switches and the (00:15:38) right configs basically to cut our (00:15:39) latency pretty significantly um like 2 (00:15:43) 3x uh and end. Um this is actually the (00:15:47) case a lot of the time is there is a (00:15:49) stupid thing that uh exists somewhere in (00:15:53) the software or the hardware and someone (00:15:55) has come up with a solution um and you (00:15:58) find it when you go to look for it in in (00:16:01) our codebase somewhere or you ask around (00:16:03) and someone's like oh yeah this XYZ (00:16:05) person has done this you should talk to (00:16:06) them and they will hook you up. Um (00:16:09) there's not a lot of time spent syncing (00:16:11) up with anyone or asking for permission (00:16:13) or (00:16:14) um waiting for anyone at all. Like the (00:16:17) answer is like when you propose someone (00:16:19) someone says a good idea. Like usually (00:16:21) you propose something and the the answer (00:16:23) is either no that's dumb or why isn't it (00:16:25) done already? [laughter and gasps] (00:16:26) Like (00:16:28) um and then you go and do it and then (00:16:30) it's done. With Elon companies, you can (00:16:32) kind of just ask for responsibility and (00:16:34) then you basically just live by the (00:16:35) sword, die by the sword, and if you get (00:16:37) things done, then you can just ask for (00:16:38) more responsibility and you can keep on (00:16:40) doing that or you're just like out. (00:16:41) What's been your experience like with (00:16:43) that? (00:16:43) >> Very much so. Yeah, like um I've jumped (00:16:47) around a lot of different projects and (00:16:48) mostly just because someone asked for my (00:16:49) help and I kept helping and then I ended (00:16:53) up owning some of the stack or a lot of (00:16:55) the stack. Um and this is the case for (00:16:57) everyone like this is just how it is. um (00:16:59) if you have any particular experience or (00:17:02) um can iterate on something very quickly (00:17:05) within days you own that component. Um (00:17:08) yeah there's no formal anything I think (00:17:10) officially on our HR software I I'm on (00:17:13) voice and iOS or something and our (00:17:17) security software thinks I still work on (00:17:20) RX integration and um (00:17:23) >> which never updated. (00:17:24) >> Yeah. No, no one ever updates this stuff (00:17:25) like um it's kind of ridiculous. And is (00:17:28) has has your like journey at the company (00:17:31) kind of been you show up there's not (00:17:33) exactly like a clear direction of what (00:17:35) you're going to work on and then you (00:17:36) just start working on stuff and then you (00:17:38) just kind of like hop from project to (00:17:40) project by whoever asks for your help. (00:17:42) >> There's a bit Yeah, there's quite a bit (00:17:44) of like overlap and flowing. (00:17:46) >> Um so like after onboarding I'm usually (00:17:49) on two or three projects at once. Um, (00:17:52) and whichever one is most pressing or I (00:17:54) can help the most on ends up taking (00:17:56) majority of my time and then that kind (00:17:58) of overlaps and flows in like a (00:18:00) waterfall way. (00:18:01) >> What's been the journey from like the (00:18:03) starting to to now? Like what what (00:18:05) projects have you worked on? (00:18:07) >> Yeah, so specifically I started um I (00:18:10) first worked on like ASRock uh and our (00:18:13) integration there and I worked with our (00:18:14) backend team a bit on like reliability (00:18:16) and scaling up because we were scaling (00:18:18) up a lot at that time. Uh and then after (00:18:21) that I took on solo building up our our (00:18:24) desktop suite. Um and took that went to (00:18:29) internal completion. Uh and then I got (00:18:32) asked for help on our imagine roll out (00:18:34) and iOS which yeah our iOS team is small (00:18:38) for like how many people use it. Like (00:18:41) it's ridiculous. You won't guess the (00:18:42) number. (00:18:44) Um (00:18:45) >> like five people for three. It was three (00:18:49) and I was the third person at the time (00:18:51) when we were rolling that out. It was (00:18:52) like it was ridiculous and everyone's (00:18:54) like really really good. Um yeah, this (00:18:57) is the first place where I've had to (00:18:58) work very hard to keep up really (00:19:00) >> with like the the speed and the talent. (00:19:03) What was the first uh experience that (00:19:05) you had where you thought to yourself (00:19:07) like you're actually being kind of used (00:19:10) to your full, you know, potential? And (00:19:12) >> I think that imagine roll out was (00:19:13) definitely like it was a really good (00:19:15) push cuz like we had this 24-hour (00:19:17) iteration cycle. Um you all would get (00:19:19) feedback every night on whatever we were (00:19:21) doing. Um and yeah, we we would push out (00:19:24) that night. Uh in the morning we would (00:19:26) have all the feedback. We would (00:19:27) immediately knock out all the bugs. um (00:19:29) implement the new stuff that that people (00:19:30) were asking for. Whatever model had come (00:19:33) up with, we implemented that too. Like (00:19:35) it was a very very fast cycle and it was (00:19:38) uh I think it was the longest like (00:19:40) continuous stretch of me being in the (00:19:42) office like every day. (00:19:44) >> What was that like at the time? (00:19:45) >> It was like two or three months. (00:19:46) >> Two or three times. Yeah. Yeah. Okay. (00:19:48) >> Um yeah, like there weren't weekends for (00:19:50) a while, which was uh it was good to (00:19:52) know that I could do that and I was (00:19:53) pretty happy doing that. Um, (00:19:56) and after that I got pulled onto Macro (00:19:59) hard product which was just one other (00:20:01) person at the time. So it was the two of (00:20:02) us uh for a while and I've been on that (00:20:04) since uh since that project off (00:20:06) basically. (00:20:07) >> I don't know how much you like know (00:20:08) about this but uh the like Colossus (00:20:11) build and all the ridiculous stuff that (00:20:15) the like early XAI team had to do to (00:20:17) turn on Colossus and like get power and (00:20:20) all the necessary inputs to making that (00:20:22) work. And even today, I think like it's (00:20:23) just bottlenecks across the entire (00:20:25) thing. You just want more you you want (00:20:27) more like uh chips and GPUs and all the (00:20:29) stuff working (00:20:30) >> and faster. Um (00:20:32) >> what was that like? (00:20:34) >> There's a lot of war stories um and a (00:20:36) lot of bets. Um (00:20:38) >> want to go into a few? (00:20:40) >> Yeah. So I think Tyler was took this bet (00:20:44) uh with Elon like uh one we were setting (00:20:47) up new racks I think of I forget what (00:20:49) which GPUs we were rolling out at that (00:20:51) time. Um, we took a bet. Uh, Elan's (00:20:54) like, "Okay, you get a cybert truck (00:20:55) tonight if you can get a training run on (00:20:56) these GPUs uh in 24 hours." (00:21:00) Uh, and we were training that night. Um, (00:21:02) >> did he get the cyber? (00:21:03) >> Yeah, he got [laughter] (00:21:05) >> I think it's Yeah, I see it from our (00:21:08) lunch window. (00:21:09) >> Mhm. (00:21:09) >> Cafeter. (00:21:10) Yeah, he's cool. Um uh you know what the (00:21:14) I so for power we actually have we have (00:21:16) to collaborate very tightly with the (00:21:19) like municipal uh and state power (00:21:22) companies uh because when load goes high (00:21:25) on their end we have to shut off and go (00:21:27) fully on the like 80 or maybe it's more (00:21:29) than that I think more more than that 80 (00:21:32) uh mobile generators we brought in on (00:21:34) trucks um and go fully on on those um (00:21:38) just so that we don't like impact power (00:21:40) uh anywhere. are like within and we have (00:21:42) to do that like seamlessly without (00:21:44) interrupting anyone's uh extremely (00:21:46) volatile training runs uh on extremely (00:21:48) volatile uh you know GPUs and and (00:21:50) hardware which scales up and down by (00:21:53) like megawws in milliseconds. It's it's (00:21:57) a lot. Um, (00:21:58) >> is that also part of the logic of like (00:22:00) basically putting massive battery packs (00:22:02) right next to the uh desenters cuz then (00:22:04) you can kind of like go up and down much (00:22:06) faster without (00:22:07) >> batteries can scale up a lot uh scale up (00:22:09) and down and uh balance that load a lot (00:22:12) faster. Um cuz with a generator you're (00:22:14) literally asking a physical thing to (00:22:16) speed up or or slow down like a spinning (00:22:19) spinning physical thing that's obviously (00:22:21) just going to take a certain amount of (00:22:22) time. the batteries can uh react to the (00:22:24) light much much faster and then yeah (00:22:26) it's like actually from the phys from (00:22:30) physical standpoint I think there's the (00:22:32) uh local capacitors the station like (00:22:35) data hall side capacitors the batteries (00:22:37) and then generators and then the public (00:22:40) municipalities although we might have (00:22:41) changed that infrastructure at this (00:22:42) point things very quickly especially on (00:22:45) the cooling side (00:22:46) >> do you have any other really good like (00:22:47) war stories that are just like uh I (00:22:50) don't know things that shouldn't have (00:22:51) been possible that became possible (00:22:53) Uh, so the the lease for the land itself (00:22:56) was actually technically temporary. It (00:22:58) was the fastest way to get the (00:23:00) permitting through and actually start (00:23:01) building things. Um, I assume that it'll (00:23:04) be permanent at some point, but yeah, (00:23:05) it's I think a very short-term lease at (00:23:07) the moment technically for all the data (00:23:08) centers. It's fastest way to get things (00:23:10) done. (00:23:10) >> And how do they how do they do that? Um (00:23:12) I think there's basically a special (00:23:13) exception uh within like the local and (00:23:15) state government says okay if you want (00:23:17) to just uh modify this ground (00:23:20) temporarily I think it's like for like (00:23:22) uh carnivals and [laughter] stuff you (00:23:24) can (00:23:24) >> Xi is actually just a carnival company (00:23:27) >> currently (00:23:29) [laughter] (00:23:29) >> and so that was the way to get done (00:23:32) quickly I mean it was done yeah 122 days (00:23:35) >> for like internal planning I know things (00:23:38) are just going to keep on scaling up (00:23:39) like crazy and Elon's talked about (00:23:41) energy being the biggest bottleneck and (00:23:43) then you know just being able to get (00:23:45) chips. Um how do you guys plan when it's (00:23:49) very difficult to like predict 12 to 24 (00:23:52) months in the future exactly what (00:23:53) projects you're going to be working on (00:23:54) or what their like resource requirements (00:23:56) are going to be. (00:23:57) >> We try we try very hard to work (00:23:59) backwards from like what's the highest (00:24:00) leveraged thing we can be doing and then (00:24:03) we determine the physical requirements (00:24:05) later. So like (00:24:06) >> if we want to get to 10 or hundred (00:24:08) billion in revenue by this date, uh what (00:24:11) are the highest leverage things we can (00:24:12) do like from an econ economic (00:24:15) perspective? How can we actually build (00:24:17) systems to do that? And then what does (00:24:19) it take on the physical and software (00:24:21) side to roll that out and and get it (00:24:23) done? Um just roll down roll backwards (00:24:25) the whole way. So we don't usually start (00:24:27) with the with the physical requirement. (00:24:31) That's usually actually at the end. Is (00:24:32) there like a SpaceXesque um like (00:24:35) algorithm for making things happen? (00:24:37) >> As in like the usual delete? (00:24:39) >> Yeah. (00:24:40) >> Yeah. I mean that's the case all the (00:24:42) time. Um and we do do the thing where (00:24:46) Yeah. We delete something and then add (00:24:47) it back later. Um (00:24:49) >> what was the like last time that you did (00:24:51) that? (00:24:51) >> Today. (00:24:52) >> Today. [laughter] (00:24:54) >> Um today. Yeah. So with macro hard we (00:24:56) deploy on um a lot of like physical (00:24:59) hardware that changes and um the testing (00:25:02) harness for that is hard. Um so we try (00:25:06) to minimize how much how many special (00:25:09) cases are downstream of where it needs (00:25:11) to be. And um for example like with (00:25:14) display scaling um we need to be able to (00:25:16) support displays that are you know 30 (00:25:19) years old as well as the latest like 5K (00:25:22) Apple whatever displays and that has to (00:25:24) happen on the same stack. Um, turns out (00:25:27) not all the systems are happy with that (00:25:28) at all times. Like you have to you have (00:25:31) to fiddle with the encoders at a certain (00:25:32) level. Like uh video encoders um was (00:25:35) [clears throat] the specific thing (00:25:35) basically we I didn't know but uh as it (00:25:39) turns out there are limits to the (00:25:41) maximum amount of pixels that certain (00:25:42) encoders can take. So we have to now (00:25:44) have I removed this special case for (00:25:46) multiple encoders and turns out we found (00:25:48) a problem at at plus 5K resolution and (00:25:50) so we added that back. What are the most (00:25:52) interesting things about XAI itself um (00:25:55) that you think like would be really good (00:25:57) stuff to talk about? (00:25:58) >> There's a lot of characters that work (00:25:59) there and also we're doing hiring in (00:26:02) like interesting ways I guess. Um like (00:26:05) things that I thought would be stupid (00:26:06) are okayed and we just do them and we (00:26:08) try them and it's like we we'll do a (00:26:10) hackathon and if we get five people in (00:26:12) as a result it's worth it. um because (00:26:15) just their like expected return on on (00:26:17) the company's like revenue or valuation (00:26:20) is higher than the cost of running this (00:26:22) hackathon for 500 people. Um like the (00:26:25) verhead value is actually very high (00:26:27) which is like funny. We did the math um (00:26:32) earlier this week. Uh right now we're (00:26:35) like I think at about $2.5 million per (00:26:37) commit is to to the main to the main (00:26:40) repo. Um and I did five today. So (00:26:43) >> you added like 12 and a half million of (00:26:45) value. [laughter] (00:26:46) >> Um (00:26:47) >> light day light days. (00:26:48) >> Exactly. It was a good day. [laughter] (00:26:51) Um it's funny things like that. Um like (00:26:55) the levers are are extremely strong. (00:26:59) Like you you can get a lot a lot done (00:27:01) with a lot less effort and time than you (00:27:04) used to be able to for sure just because (00:27:06) of who you work with, the internal (00:27:08) tooling that we built up. Um, and my (00:27:12) boss. (00:27:13) >> What's like an example of the type of (00:27:15) person that like wants to work here? Cuz (00:27:17) I know when when you're talking about (00:27:19) it, you kind of show up and the first (00:27:20) day you're just like, I want to work on (00:27:22) the weekends. I want to work on, you (00:27:23) know, during the night, all this stuff. (00:27:25) Uh, go all in on this. Um, what kind of (00:27:27) special characters are are working (00:27:29) there? (00:27:30) >> People are definitely very enthusiastic (00:27:31) when they come in. Like, um, very very (00:27:35) enthusiastic. (00:27:37) uh just like (00:27:38) >> like mission oriented. (00:27:40) >> Um there's I guess different types of (00:27:43) ambition for sure. Some people want to (00:27:44) move up like the leadership ladder and (00:27:46) own more in terms of a managerial like (00:27:49) how many people report to me sense. Some (00:27:51) people want to own huge parts of the (00:27:53) technical stack. So like right now we're (00:27:55) doing a big rebuild um of like our core (00:27:59) uh production APIs. It's being done by (00:28:02) one person with like 20 agents. Um, and (00:28:05) they're they're very good and they're (00:28:07) capable of doing it and um, like it's (00:28:10) working well. Uh, so you can own huge (00:28:14) chunks of the code base, no problem. (00:28:17) >> It's kind of like a X where like after (00:28:19) the acquisition they like had, you know, (00:28:23) much fewer people, but you just like (00:28:24) never had a lot of people in the first (00:28:26) place, so there's one person like owning (00:28:27) a huge part of the product. (00:28:29) >> Absolutely. for hiring. Um what's what (00:28:32) unusual practices outside of just (00:28:34) hackathons uh does XI do? (00:28:37) >> Uh so we're pushing very hard on Macro (00:28:39) hard. Like for two or three weeks I was (00:28:41) doing upwards of 20 interviews a week. (00:28:44) So that's like some of them are like (00:28:46) quick 15 minutes. Some of them are full (00:28:47) 1 hour technicals. So a lot of my time (00:28:50) uh is dedicated towards bringing in new (00:28:52) people and a lot of people are very (00:28:54) good. So it's it's actually very hard to (00:28:56) judge them. How do you (00:28:58) >> uh I have a very specific problem that I (00:29:01) have solved. I'm not going to reveal it (00:29:02) because then people will use it. But I (00:29:05) have solved a very specific computer (00:29:07) vision problem a few years ago for one (00:29:09) of my startups and I uh I give people (00:29:11) half an hour to try to implement the (00:29:13) solution. It's actually very very (00:29:14) simple. This deceptively simple (00:29:16) solution. People always overthink it. Um (00:29:19) and this is something I like to index (00:29:21) for on my team especially is like can (00:29:23) you not overthink it and come up with a (00:29:25) simple solution? Um it helps a lot (00:29:28) because we're deploying on such a wide (00:29:32) variety of like (00:29:34) uh on a wide variety of hardware as a (00:29:36) result of the wide variety of of (00:29:37) customers like literally 30 years 40 (00:29:39) years of uh different hardware, (00:29:41) different operating systems, everything (00:29:43) like that. You have to come up with (00:29:45) simple solutions or you're going to have (00:29:46) a 10 million line code base uh next (00:29:49) week. So you you this is like very (00:29:53) important. Um and especially now relying (00:29:56) more and more on on agents and and an AI (00:29:59) and and such for writing code. Um (00:30:03) an AI will happily train out 200 lines (00:30:06) when a 10line solution will do um and (00:30:08) probably do better. So you have to look (00:30:10) for that. Like I want people and I look (00:30:12) and actively hire for people who can (00:30:14) find the 10line solution first. Um, (00:30:18) we're totally fine with people using AI (00:30:19) to code things. Like you should you (00:30:21) should use that as a force multiplier, (00:30:23) but uh for now we're smarter. We'll see (00:30:27) next year. (00:30:27) >> What other like force multipliers do you (00:30:29) kind of like look for? (00:30:30) >> I like people who will challenge uh (00:30:34) challenge requirements and challenge me. (00:30:36) So often uh I got this from uh Chester (00:30:40) Zai German for he told me this and I (00:30:42) thought it was great. He throws in (00:30:43) usually um an incorrect requirement or (00:30:46) question or an impossible like uh line (00:30:49) in uh his challenges for people when (00:30:52) he's hiring like coding challenges and (00:30:54) he expects people to come back and say (00:30:57) like hey this is wrong this is not (00:30:58) possible you made a mistake and if he (00:31:01) doesn't then uh he doesn't hire them (00:31:03) same thing for me I picked that up it's (00:31:05) a great idea (00:31:06) >> the pace is insanely fast and like you (00:31:09) said you kind of have worked on a number (00:31:11) of different things How do you kind of (00:31:13) come up to speed on something as quickly (00:31:15) as possible when you're on a new task or (00:31:17) project? (00:31:17) >> It depends on what thing it is. If (00:31:19) there's a lot of code to read, (00:31:22) >> yeah, (00:31:22) >> read the code (00:31:24) >> by hand. Um like GD go to definition (00:31:27) over and over again and you'll find (00:31:29) things out very quickly. Actually, it's (00:31:31) not that hard. Um for most things, the (00:31:34) implementation is like less lines of (00:31:37) code than you would otherwise see, which (00:31:38) is nice. um not all the time, but in (00:31:40) most cases, if it's something that's in (00:31:42) very active development, this is not the (00:31:43) case. There's going to be 20 different (00:31:45) versions of it going at the same time, (00:31:46) and it's not obvious what is the current (00:31:48) path. So, you just got to talk to (00:31:49) people, and people are very open. Like, (00:31:51) this is actually one of the things I was (00:31:52) very surprised by, uh pleasantly (00:31:54) surprised by when I joined is I thought (00:31:55) people would be super smart and stuck (00:31:56) up, but no, people are just super smart (00:31:58) and very nice and helpful. Like, (00:31:59) everyone's on the same team, everyone's (00:32:01) rooting for each other, people are (00:32:02) willing to like help you out um and (00:32:04) answer your questions. So, which is good (00:32:06) because we don't like write a lot of (00:32:08) docs. We write things. We do things too (00:32:10) fast to write docs really. Um, actually, (00:32:14) yeah, we're trying to figure out some (00:32:15) systems on on my team to like (00:32:16) automatically generate docs as we like (00:32:18) build stuff. Um, and with Grock, which (00:32:23) is cool that we have unlimited access to (00:32:25) uh very smart AI because then we can try (00:32:27) a bunch of stupid things, see if it (00:32:29) works, which otherwise, you know, at a (00:32:31) startup would cost you maybe like$100 or (00:32:32) a million dollar 100k or a million (00:32:33) dollars in in credits or whatever. we do (00:32:35) it for free. So experimentation like (00:32:38) failure you can fail on a lot of things (00:32:39) and it uh a lot more things you (00:32:41) otherwise would um and as a result more (00:32:44) experiments are tried um more uh succeed (00:32:47) >> on the like experimentation side how are (00:32:50) you guys kind of like trying to maximize (00:32:51) for the number of experiments or like (00:32:53) good shots on goal uh that you can do. (00:32:55) >> There's often like uh a time time (00:32:58) constraint. We will frequently launch (00:33:01) multiple experiments especially on the (00:33:02) model side at the same time and in some (00:33:06) cases it's not even because of a time (00:33:07) constraint necessarily in terms of like (00:33:10) I need to try x amount of things in y (00:33:12) time it's in two weeks this prerequisite (00:33:17) will be ready either in the hardware or (00:33:19) in the training data or something but in (00:33:22) the meantime I need to deploy something (00:33:23) today (00:33:25) uh what can I do and so you run two (00:33:27) three experiments and you find out what (00:33:28) you can deploy day um and bring in (00:33:30) revenue or customer result whatever it (00:33:32) is today and then two weeks you switch (00:33:34) over um like that's something we do all (00:33:37) the time especially at macro hard. (00:33:39) >> Have you uh seen anything where a (00:33:40) timeline should have been much longer on (00:33:42) like a project that you were working on (00:33:44) um and somehow you guys were able to (00:33:46) kind of like bring that in by you know (00:33:47) weeks or months (00:33:48) >> all the time. (00:33:50) Every time uh all the time every time we (00:33:53) come come away from like an EL meeting (00:33:54) or something internal where um someone (00:33:59) pushes hard to get something done or (00:34:01) someone external who doesn't isn't (00:34:03) responsible for the thing asks for a (00:34:05) requirement ask for something to be done (00:34:07) in an what we originally think (00:34:08) unreasonable amount of time you know we (00:34:10) spend two minutes like thinking about it (00:34:12) complaining maybe a bit uh and then the (00:34:14) rest of the time is dedicated to getting (00:34:16) it done in that time. Um yeah, (00:34:19) frequently the estimated time to get (00:34:23) something done all the time to get (00:34:25) something done is based on some set of (00:34:28) assumptions. (00:34:29) >> And then once you get this timeline (00:34:32) that's like half or onetenth of what you (00:34:34) would have otherwise done. You look at (00:34:35) the assumptions say okay proportionally (00:34:37) how much is this impacting my my (00:34:39) timeline? And then you knock it out or (00:34:41) you change it and then suddenly you get (00:34:43) a 2x improvement in your timeline. You (00:34:45) do that a few times. you can meet (00:34:47) whatever requirement you really want. (00:34:49) Um, yeah, at a certain point you get to (00:34:51) the physical limitations, but you're (00:34:53) never there um from the start. (00:34:55) >> So, (00:34:56) >> I know for like full self-driving um and (00:34:59) same thing with the the rockets of (00:35:01) SpaceX, the Elon timeline was (00:35:04) significantly longer. like an Elon (00:35:07) [snorts] time might be a quarter or half (00:35:09) of what it actually eventually takes, (00:35:12) but then it also, you know, happens four (00:35:13) times faster because of the initial (00:35:15) timeline. Is it more or less like at XAI (00:35:19) because it's more I mean, I guess more (00:35:21) on the software side now. Um, but even (00:35:23) on the data center side, things seem to (00:35:25) be happening just way way way faster. (00:35:27) Uh, and they also seem to be happening (00:35:28) on like the same timeline as he's (00:35:30) roughly saying. He's like this is going (00:35:32) to happen roughly you know this number (00:35:33) of months in the future and then it (00:35:35) actually does. (00:35:35) >> I think he himself has calibrated his (00:35:37) timelines (00:35:38) >> like differently over (00:35:39) >> Yeah. now that he's deployed a number of (00:35:42) extremely like a wide variety of um (00:35:46) deployed hardware at scale. (00:35:48) >> So I think his own estimates for things (00:35:51) are definitely a lot better. And so uh (00:35:53) that's definely the case. I think he (00:35:55) also updates his timelines faster now (00:35:57) too like um sometimes daily. I think he (00:36:01) he he's talking with us and figures out (00:36:05) what the update on the timeline should (00:36:06) be based on various parameters and (00:36:08) sometimes they come from him too right (00:36:10) um especially on the infrastructure side (00:36:12) uh if a deal or um we can be put up in a (00:36:15) batch for for the production of a (00:36:17) certain chip um well we can save a month (00:36:20) or two maybe um maybe even more than (00:36:23) that depends on what the deployment is (00:36:25) specifically and then on the software (00:36:26) side it's the same he always says is (00:36:29) like you can always attempt to do (00:36:30) something, you know, in one month that (00:36:32) would otherwise take a year and you'll (00:36:34) probably get it done maybe in two. Um, (00:36:37) still a lot faster. (00:36:39) >> I remember in the like early days of uh (00:36:42) SpaceX there was this internal I think (00:36:45) Elon would say internally like every day (00:36:47) that we delay is like 10 million in loss (00:36:49) revenue and I have no idea what it would (00:36:51) be like for XAI like things are moving (00:36:54) so fast. It's like is there kind of an (00:36:56) internal thing in your head of every day (00:36:59) that we don't like push push hard or (00:37:01) make something happen um we're losing (00:37:04) out on x amount of value that could be (00:37:06) created. (00:37:07) >> Yeah, for for macroart specifically, we (00:37:09) do have a few pretty specific revenue (00:37:12) targets. I can't delete the number (00:37:14) specifically, but um like in my head (00:37:16) whenever something gets delayed or (00:37:18) accelerated, I can pretty quickly (00:37:20) calculate how much money we just made or (00:37:22) lost. um (00:37:23) >> just wild swings. You just (00:37:25) >> Yeah, I mean the numbers are huge (00:37:28) [laughter] (00:37:29) uh just because the expected return is (00:37:31) so huge and um the timeline is so fast. (00:37:34) So a few days is actually (00:37:36) proportionately fairly large compared to (00:37:39) how much you would you would otherwise (00:37:41) expect the revenue to be. (00:37:42) >> Elon's like famous for making really (00:37:44) really big bets pretty quickly. uh like (00:37:46) what's the biggest decision that's been (00:37:47) made in a single meeting where like huge (00:37:49) huge amounts of uh capital or time or (00:37:52) commitment were done? (00:37:54) >> Um I think one of them was certainly the (00:37:59) decision to go with a model that would (00:38:01) be at least 1.5 times faster than a (00:38:04) human for (00:38:05) >> macro hard looking like significantly (00:38:08) faster than that. 8x maybe maybe more. (00:38:11) Um the like for other human emulator (00:38:16) type attempts in the other labs the (00:38:18) approach has been let's do more (00:38:19) reasoning and build a bigger model. (00:38:22) We've like that decision put us in (00:38:24) totally the opposite track of what (00:38:26) everyone else is doing. And everything (00:38:29) that we're doing really is downstream of (00:38:30) that like a well not everything but a (00:38:32) pretty much everything. Um it impacted (00:38:35) and it was very early on uh that this (00:38:38) was decided. It was sort of expected (00:38:40) also. um that this is the move, (00:38:42) especially given the analog to full (00:38:44) self-driving. Um (00:38:47) no one's going to wait around uh 10 (00:38:49) minutes for the computer to do something (00:38:50) that I could have done in five, but if (00:38:52) it can be done in 10 seconds, well, I'd (00:38:54) be happy to pay whatever amount of money (00:38:56) for that. Um it's just obvious really. (00:39:00) So normally like us engineers would you (00:39:03) know if it's would push back and say oh (00:39:06) you know here's the 20 different reasons (00:39:08) uh that it needs to be this way um but (00:39:10) if a decision is made and you work (00:39:12) backwards (00:39:13) then life finds a way. (00:39:16) >> I remember Elon saying uh I think it was (00:39:18) at the like Y cominator uh he was doing (00:39:21) a like Q&A with Gary Tan and Gary talked (00:39:24) about like AI researchers and he was (00:39:26) like no they're just all AI engineers (00:39:28) now. Yeah, we did the this was someone (00:39:32) said that in one of the meetings um we (00:39:35) did with him uh talking about recruiting (00:39:37) like here was here's the job (00:39:38) descriptions or something like that and (00:39:40) like for 10 minutes he just goes on (00:39:42) engineers just engineers doesn't matter (00:39:44) good engineers engineers just someone (00:39:46) who's fundamentally a problem solver (00:39:48) doesn't matter if they did like this you (00:39:49) know XY thing and this infrastructure or (00:39:51) this you know particular architecture or (00:39:54) whatever engineers (00:39:56) >> why is it so important why is that (00:39:57) definition so important (00:39:58) Um (00:40:00) it's keeps things broad. It means that (00:40:02) people can come in to us from like a (00:40:06) extremely wide variety of places and (00:40:10) this has been the case. I mean, there is (00:40:12) I think less so in the AI world, but I (00:40:15) think there's a lot of SpaceX stories (00:40:17) where people came in from strange walks (00:40:20) of life that would not have otherwise (00:40:21) seemed to be the case and then ended up (00:40:23) doing huge things at SpaceX in the (00:40:26) engineering world as a result. So, (00:40:28) keeping it broad means that those people (00:40:31) can have a path to us and uh and help us (00:40:36) accelerate. For you personally, what's (00:40:38) the most like fun thing about working (00:40:40) there dayto-day? (00:40:41) >> No one tells me no. (00:40:42) >> No one tells me no. (00:40:43) >> No one tells me no. Um yeah, if I like (00:40:47) have a good idea, I can usually go and (00:40:49) implement it that same day and show it (00:40:50) off and we'll see if uh if it makes (00:40:53) sense. We we'll we'll we'll run whatever (00:40:55) eval or um show it to a customer or show (00:40:59) it to Elon or whoever and we'll get an (00:41:02) answer usually that same day uh as to (00:41:04) whether or not that was the right move. (00:41:05) There's no deliberation. There's no (00:41:07) waiting for any bureaucracy. Uh I like (00:41:09) that a lot. I was expecting to sacrifice (00:41:11) some amount of this coming from (00:41:13) extremely small startups to a larger (00:41:15) company. Like I guess joining at 100 (00:41:18) people, I [laughter] mean to me it was (00:41:20) like a 10x leap of anywhere else I've (00:41:22) been. But uh I guess relatively to loan (00:41:25) companies is pretty small and it does (00:41:26) feel very small. Um there's not a lot of (00:41:29) overhead in anything. (00:41:30) >> Did you have any other like big (00:41:31) assumptions going in that proved like (00:41:33) completely wrong? I thought there would (00:41:36) be more top down. Um, and there's some, (00:41:39) but not really that much. (00:41:40) >> Um, especially because of how many (00:41:42) there's basically only three layers of (00:41:45) management. There's um like IC's uh (00:41:48) there's the co-founders and some of the (00:41:50) new managers and then Elon and that's (00:41:52) it. And so because there's so many (00:41:54) reports to the managers now, um (00:41:58) nothing really comes from them top down (00:42:00) like we'll usually come up with a (00:42:01) solution. They're okay. Elon okay is (00:42:05) we're good if there's feedback then we (00:42:07) update but (00:42:08) >> it's a lot more bottom up than I (00:42:09) expected (00:42:10) >> like trying to be designed so that (00:42:12) everyone is like building things and (00:42:13) there there was like fewer manager (00:42:15) managers and mo more just like builders (00:42:18) >> uh there's yeah when I joined I think (00:42:20) every manager also wrote code um and I (00:42:23) think largely today they still do um not (00:42:26) as much now that some of them have you (00:42:29) know 100 plus people reporting to them (00:42:31) But (00:42:33) everyone's an engineer. I remember (00:42:34) actually on my first week um I sat down (00:42:37) for dinner and this guy sits next to me (00:42:38) and I asked, "Hey, you know, like what (00:42:40) what team you on? How you going? How you (00:42:41) doing? I I just joined." And he tells (00:42:43) me, "Oh, I'm on on sales uh and like (00:42:46) enterprise deals." And I was like, "Oh, (00:42:48) I don't want to talk to this guy. He's a (00:42:49) sales guy." And then he starts telling (00:42:51) me about this model he's training on the (00:42:54) he's an engineer, too. The sales team is (00:42:56) an engineer. Are all engineers. Everyone (00:42:58) is an engineer. Uh I think at the time (00:43:00) there was probably less than like eight (00:43:01) people who were not engineers at the (00:43:03) company um in some capacity and even (00:43:06) then like yeah it was really cool. (00:43:09) Everyone (00:43:11) everyone contributes to the machine. (00:43:13) >> Is it a little bit more like you have a (00:43:16) single person working on some project (00:43:17) and they just you know if you're an (00:43:19) engineer and you're working on the thing (00:43:20) you can have a much closer relationship (00:43:22) to the customer and like understanding (00:43:23) their problem and then rapidly like (00:43:24) implementing solutions and stuff. (00:43:26) >> Yeah. (00:43:27) >> Yeah. um the less layers you have, the (00:43:29) less information is lost. Um there's (00:43:31) less compression basically. Uh because (00:43:34) you have to communicate less times and (00:43:37) language is lossy compared to what's (00:43:39) going on in your brain. Um so if you (00:43:42) have to go from customer's brain to (00:43:45) words to, you know, salesperson's brain (00:43:48) to words to manager, every layer, you're (00:43:51) losing (00:43:52) >> a huge amount of information. Yeah. And (00:43:54) so if you can cut as many layers as (00:43:57) possible, then you've only got one (00:43:59) compression step of the customer telling (00:44:01) you what to do or what they want and (00:44:03) what or what their experience is or (00:44:05) whatever and you as the engineer can (00:44:06) solve it directly. (00:44:07) >> Is there anything like specific that (00:44:09) you've never heard of or seen at any (00:44:11) other company that XAI does where that (00:44:14) allows things to just happen way faster? (00:44:16) The fuzziness definitely between teams (00:44:20) and what everyone is responsible for is (00:44:23) definitely not what I expected and I (00:44:24) don't think exists nearly as much in any (00:44:27) large company or even remotely similar (00:44:30) similarly sized company. Like if I need (00:44:33) to fix something on our VM (00:44:34) infrastructure, (00:44:36) uh I will do it. I will show it to the (00:44:37) guy who owns that and they will be like (00:44:39) okay and it's merged immediately and (00:44:41) deployed. like uh there's not a lot of (00:44:45) strict regiment. (00:44:47) >> Mhm. (00:44:47) >> Um like everyone is allowed to update (00:44:51) everything and there's some checks for (00:44:53) dangerous things but um largely you're (00:44:56) trusted to do the right thing and do it (00:44:58) right. Uh which is really cool. I (00:45:01) remember when uh Elon was still like (00:45:03) really working on Doge, there was at one (00:45:05) point I think they deleted um like Ebola (00:45:08) prevention or something and then and (00:45:10) then they rapidly like reput that back (00:45:12) in what things have been like deleted (00:45:14) because of this rapid process of trying (00:45:16) to figure out, you know, what doesn't (00:45:17) need to be done. Um and then like (00:45:19) reimplemented. (00:45:20) >> There's very rarely anything like (00:45:22) irreversibly destructive. I'm actually (00:45:24) not really aware of anything where (00:45:26) something was irreversibly destroyed, (00:45:28) but like I said, yeah, frequently (00:45:30) something will be deleted or removed or (00:45:31) something like that and someone will be (00:45:33) like, "Hey, I needed that." uh you know (00:45:36) an hour or two and then you go and roll (00:45:38) back um or you know sometimes it can be (00:45:41) months uh where you know someone's (00:45:43) building this project and they're (00:45:45) depending on I don't know some piece of (00:45:46) infrastructure something like that and (00:45:49) turns out we rebuilt that thing three (00:45:51) times by the time you go and and deploy (00:45:53) and need it and so you update and and uh (00:45:57) go that way. (00:45:58) >> Do you think it's helpful to have like (00:45:59) so few people working there on on the (00:46:01) engineering team? (00:46:02) >> Yeah, definitely. um the more people you (00:46:05) have doing a like I I definitely say (00:46:07) like a a job for one person done done by (00:46:10) two will take twice as long. Um and it (00:46:13) applies for every skill I think. Uh and (00:46:17) especially now that you don't need to (00:46:20) physically write as much code as you did (00:46:23) previously. You can be more of the (00:46:25) decision maker and the architect. (00:46:27) Everyone can be an architect. Um you (00:46:29) just don't need as many hands. Uh so one (00:46:32) brain can do a lot more. (00:46:33) >> You tried starting multiple companies (00:46:36) and you were doing a whole bunch of (00:46:37) different projects prior to this. What (00:46:39) about working here and like what about (00:46:41) the mission the culture resonated? Why (00:46:43) did you why did you decide to work on (00:46:45) this? Uh, I've definitely always been (00:46:47) very Elon fil like I always uh he's been (00:46:50) a big personal hero of mine. Uh (00:46:51) especially growing up uh you know seen (00:46:53) the Falcon landings, the first ones and (00:46:55) um I went out to uh launch five of (00:46:59) Starship which was so worth it. It was (00:47:01) the first one they caught. It was it was (00:47:03) really cool. It was definitely the (00:47:05) coolest thing I've ever seen. Um so (00:47:07) being part of uh anything even remotely (00:47:10) related to that sounds awesome to me. (00:47:12) Um, (00:47:13) >> is there a reason why you chose like (00:47:14) this company instead of SpaceX or Tesla (00:47:16) or (00:47:17) >> Yeah, I'm definitely like an (00:47:18) entrepreneur by heart and um, uh, Xi is (00:47:20) definitely the smallest company uh, the (00:47:22) newest of all of them. It I think my (00:47:25) assumption is and this is largely proven (00:47:27) true I think uh, is where you can have (00:47:29) the most leverage and change as an (00:47:31) individual person um, because (00:47:34) proportionately you're a much larger (00:47:36) percentage of the company um, than you (00:47:39) would be at these other companies. Not (00:47:40) to say that like they're not doing cool (00:47:43) things or everyone's not as important, (00:47:45) but the Yeah. Just the proportional (00:47:48) change (00:47:48) >> kind of to to decision is like way (00:47:50) higher. (00:47:50) >> Yeah. (00:47:51) >> Yeah. (00:47:52) >> Uh not even to decision but to (00:47:54) implementation to seeing the results (00:47:56) like it's very quick. And um I guess (00:48:01) another assumption that I thought would (00:48:02) be the case but it's wrong uh that I had (00:48:05) was that uh I would be faster on my own (00:48:07) you know to build XYZ thing or try XYZ (00:48:11) experiment. I'm actually usually faster (00:48:13) at XAI just because I have (00:48:16) uh a groundwork and a team who's (00:48:19) probably already done a lot of the steps (00:48:20) that I would otherwise have to do by (00:48:21) hand. Um and there's yeah no one saying (00:48:24) no. You mentioned like it's kind of a (00:48:27) fuzzy blurred line between people (00:48:29) working on different people working on (00:48:31) different things. Has there been any (00:48:33) ability for you to kind of go to other (00:48:34) people in the organization and just ask (00:48:36) for help (00:48:37) >> all the time? (00:48:38) >> What does that look like? (00:48:39) >> Um I walk up to their desk and I say, (00:48:42) "Hey, here's my question. Um what are (00:48:45) you working on right now? Can I support (00:48:46) any of that? And can you help me with (00:48:48) this?" Uh that's it. Everyone's in the (00:48:51) same building. So, uh yeah, actually (00:48:55) funny enough, we um uh we started (00:48:58) testing some of our uh human emulators (00:49:01) internally within the company as as (00:49:03) employees. And um in some cases like (00:49:07) like we didn't really tell anyone about (00:49:08) this. And so in some cases there'll be (00:49:11) someone like someone doing some work and (00:49:14) someone is like, "Hey, can you help me (00:49:15) with this thing?" Or like, "Can you do (00:49:16) this thing?" And the virtual employee is (00:49:19) like, "Yeah, sure. Come to this desk. (00:49:20) Come to my desk." and they go there and (00:49:21) there's nothing there. (00:49:22) >> It's like the the claw situation where (00:49:24) it's like we're going to show up and uh (00:49:26) I think when they first rolled out their (00:49:28) vending machine, it was like, "I'm going (00:49:30) to see you tomorrow." And then (00:49:31) [laughter] it, you know, it's obviously (00:49:32) like a piece of code. (00:49:34) >> Yeah, exactly. And so, uh, multiple (00:49:36) times I've gotten a ping saying like, (00:49:37) "Hey, this guy on the org chart reports (00:49:39) to you. Is he like not in today or (00:49:41) something?" [laughter] (00:49:42) >> It's just like an emulation (00:49:44) >> and it's a it's an AI. (00:49:46) Uh, it's a virtual employee. Um but (00:49:49) yeah, generally we all expect to be like (00:49:50) in the same building and reachable to (00:49:52) each other. Um so uh it goes like always (00:49:56) and uh I can ask for help. I people ask (00:50:00) me for help all the time. (00:50:01) >> What have been the biggest like blunders (00:50:03) that have happened? Hm. (00:50:06) So, um, with the human emulators, with (00:50:08) the customers that we're working with, (00:50:11) um, when we try to understand, like we (00:50:14) always try to understand what the job (00:50:15) that they're doing is and all the facets (00:50:17) of it. Um, frequently we'll, you know, (00:50:20) talk to them, we'll interview, we'll (00:50:21) even watch them. Well, actually, we'll (00:50:23) do the watching at the last step. So, (00:50:24) we'll talk to them, we'll interview, (00:50:24) they'll give you either write up or (00:50:26) we'll just meet up with them and and (00:50:28) write notes as to how they do their job. (00:50:31) And then um like a week later we'll look (00:50:34) at the uh mistakes that the virtual (00:50:36) employee is making and realize like well (00:50:40) it's always making mistakes in these (00:50:41) places in these specific cases. What's (00:50:42) going on? And we go watch the human (00:50:43) doing the same thing and there's like 20 (00:50:45) different steps that are missing that (00:50:46) they just totally left out and we go to (00:50:48) them and they're like oh yeah we do that (00:50:50) like I forgot to tell you. My bad. It it (00:50:53) happens all the time. Um a lot of things (00:50:56) people like I guess assume (00:50:59) automatically. it's all for granted in (00:51:00) their head. Totally on autopilot. The (00:51:02) same way that you um can like be driving (00:51:05) for an hour and not remember a single (00:51:07) second of it and not be paying attention (00:51:08) can be totally in your own world. Um (00:51:10) this is the same for every thing that a (00:51:12) human does uh repeatedly. And that's (00:51:15) what we're trying to solve basically is (00:51:17) all the uh dumb stuff that humans do (00:51:20) repetitively right now that they don't (00:51:21) need to. (00:51:22) >> Um trying to solve for that case. (00:51:24) Exactly. (00:51:24) >> How do you decide like which which thing (00:51:26) to go after? What's like the in your (00:51:28) head when you're thinking about that, (00:51:30) what are the biggest things outside of (00:51:31) driving that humans do all the time that (00:51:33) they just don't need to do? (00:51:35) >> Um, (00:51:36) anything repetitive on a computer. So (00:51:38) like customer support is a big one. um (00:51:40) where it's just taking in free form (00:51:43) input from arbitrary customer in (00:51:45) arbitrary form factor and uh translating (00:51:49) that into a standard workflow that is (00:51:52) purpose-built for like uh an AI to take (00:51:56) care of that so that human could go and (00:51:58) do something more creative and uh use (00:52:00) their brain like in a more effective (00:52:02) way. um totally the same like it's it's (00:52:06) a total parallel to what happened uh in (00:52:08) the coding world like okay I don't need (00:52:11) to write the same uh you know (00:52:13) implementation 20 different times (00:52:14) anymore uh I can describe it in like (00:52:16) three words and it's done um it's a huge (00:52:21) compression step uh and this is the same (00:52:24) thing basically but for arbitrary uh (00:52:26) digital workflows (00:52:27) >> on the human emulator side you run into (00:52:29) this problem of humans not existing and (00:52:31) then like someone says come to my desk (00:52:33) and the person doesn't exist. Is there (00:52:34) any other thing that's been kind of (00:52:36) surprising on rolling that out (00:52:37) internally? (00:52:38) >> Surprisingly, we've been able to (00:52:41) generalized to more cases than we (00:52:42) thought. We test (00:52:44) >> and we're pleasantly surprised a lot of (00:52:46) times. Um like just today we we gave (00:52:49) Elon a few cases where we did not train (00:52:52) on this task at all, but it did it (00:52:54) flawlessly, like perfectly like way (00:52:56) better than we would have expected. Um (00:52:58) yeah, the the generalization is better (00:53:00) than we expected for sure. and we're (00:53:02) still at a very early stage, so it's (00:53:03) only going to get better. Um, (00:53:06) and it's again the same parallels to (00:53:08) full self-driving where there's stuff (00:53:10) not in the training data that the car (00:53:13) does react to perfectly um due to (00:53:16) generalization of a otherwise very very (00:53:19) small model like it's a matter of like u (00:53:23) basically weight efficiency. (00:53:24) >> For the Elon meetings that you've been (00:53:26) in, like what does that actually look (00:53:28) like? Um, they're pretty simple, (00:53:31) honestly. Um, and, uh, I've been lucky (00:53:34) that most of the ones I've been in have (00:53:36) gone mostly pretty smoothly. Um, uh, (00:53:40) yeah, there's always, (00:53:40) >> what does smooth look like? (00:53:42) >> Smooth. Smooth is, uh, limited feedback (00:53:45) or thumbs up. Um, that means like, okay, (00:53:48) you're going in the right direction. (00:53:50) Keep going. Uh, I'll hear updates next (00:53:52) week. Uh, or whatever it is. Um if (00:53:56) there's feedback or a total reversal of (00:53:58) direction as a request then we messed up (00:54:01) somewhere. Um (00:54:03) then the question is where? So that's (00:54:07) usually we don't even have time to (00:54:08) identify that. Um that's something you (00:54:10) just build up implicitly as a muscle as (00:54:12) you go on. Um and sometimes assumptions (00:54:15) also change um (00:54:17) based on new information. That always (00:54:19) happens in every case. So when it comes (00:54:20) from the top down, it's a little (00:54:21) chaotic, but (00:54:22) >> I know with like SpaceX, the cost for (00:54:25) parts and building things is super super (00:54:27) important. Uh cuz, you know, everything (00:54:29) basically costs a [ __ ] ton of money and (00:54:31) time to to do, right? Um (00:54:34) >> for this sort of thing, I imagine it's a (00:54:36) little bit less focused on, you know, (00:54:38) he's not like necessarily drilling down (00:54:40) on do you understand every part of every (00:54:43) process. Um what is what does it look (00:54:45) like when he's kind of giving feedback? (00:54:48) Um, usually (00:54:52) it's either at a very high level or at a (00:54:54) very low level. (00:54:55) >> It's not really often in between. Um, so (00:54:59) either on the high level it's like a (00:55:00) product direction or customer sense, you (00:55:02) know, focus on this segment (00:55:04) >> exclusively or don't do this thing at (00:55:07) all or whatever. Um and then at a low (00:55:10) level uh especially when it comes to uh (00:55:14) compute efficiency or latency, he'll (00:55:16) always have a specific uh suggestion uh (00:55:20) or let's try this. And he's open to (00:55:23) being like proven wrong, but it has to (00:55:25) be proof. It has to be like let's try it (00:55:27) and see what the results are. Uh it (00:55:29) can't be just someone's opinion. There (00:55:31) has to be an experiment done. Um which (00:55:33) has led to some surprising results (00:55:35) sometimes and we go with it. What have (00:55:37) been those? (00:55:38) >> Um, so the compute efficiency of going (00:55:40) with the small model has led to well a (00:55:44) lot of improvements that we wouldn't (00:55:45) have otherwise thought. Um, some of them (00:55:47) are secondary, some of them are primary. (00:55:49) The obvious ones are well obvious um (00:55:52) being able to go much much faster to (00:55:53) human but also uh as a result and Tesla (00:55:57) found this too with full self-driving (00:55:59) going with the smaller model they're (00:56:00) able to iterate much much faster. Um so (00:56:03) not only does the model uh react to (00:56:05) situations faster and um can be more I (00:56:08) guess tolerant of time frames um (00:56:13) you can also just deploy iterations much (00:56:15) faster if it was 4 weeks before maybe (00:56:17) it's one week now. Um so as like that (00:56:20) that actually goes back to the (00:56:22) experimentations why we can have 20 (00:56:23) different ones going in parallels was a (00:56:25) result of that particular decision um (00:56:28) early on in in the chain (00:56:29) >> and was the initial idea like go just do (00:56:32) big large models and then (00:56:34) >> sort of uh we definitely wanted to go (00:56:36) faster than everyone else um but the (00:56:39) question of how much faster was well the (00:56:42) answer to that was amplified basically (00:56:44) multiply by a lot (00:56:45) >> there's this uh like a lot of bias and (00:56:47) stuff in Wikipedia and Elon has been (00:56:49) like focused on kind of creating an (00:56:51) alternate version that's just kind of (00:56:52) like you know more truthful uh in (00:56:54) effect. Um how do you go about basically (00:56:58) cleaning up the internet in that way to (00:57:00) figure out what is truth? (00:57:01) >> It's a really hard problem. (00:57:03) >> Yeah, (00:57:03) >> it's very hard especially because um (00:57:08) the internet is not usually the ground (00:57:10) truth for whatever thing it is. So (00:57:13) wherever we can we try to drill down to (00:57:16) the fundamentals which is very hard like (00:57:19) I don't know what is the fundamentals (00:57:21) like in physics of the constitution (00:57:25) that's not really a question I think I (00:57:27) can answer or anyone could really (00:57:28) faithfully answer very well but you try (00:57:30) to do something like that um drill down (00:57:33) as close as you can and then build up (00:57:35) from that which is hard too because (00:57:37) there's not actually a big body of (00:57:38) [snorts] like (00:57:40) writing that does that. Um, (00:57:44) one of the few probably examples is like (00:57:46) James Burke um with his connection (00:57:49) series is where he'll take two totally (00:57:51) seemingly unrelated concepts and then (00:57:53) connect them um through physics and (00:57:55) inventions. Um it's really cool and (00:57:58) we're trying to do the same but uh it's (00:58:01) fairly novel. (00:58:02) >> How do you find better data? (00:58:04) >> Uh data is not the only thing that goes (00:58:05) into the results. (00:58:08) >> Yeah. like how you actually train on (00:58:11) that data and I know it's a pretty broad (00:58:14) term but um it is true like how you (00:58:17) actually evaluate against that data and (00:58:19) train against it and your different (00:58:21) methods for updating the weights do (00:58:22) matter a lot. um (00:58:26) you can try to faithfully recreate the (00:58:28) input or the the output given any (00:58:30) arbitrary input and well you can create (00:58:34) basically a horrible copy paste (00:58:35) mechanism if you want um which is a (00:58:38) classic problem in in ML um (00:58:42) there's a bit of an art to it to to (00:58:43) avoid that problem but the I guess we're (00:58:46) a few steps removed from that at this (00:58:47) point um we're not measuring (00:58:52) the fitness to any particular data set. (00:58:54) At this point, we're trying to measure (00:58:55) to an arbitrary output. So, it matters a (00:58:57) lot how you construct your E dolls. Um, (00:58:59) which is really hard for truthfulness (00:59:01) because then you need to know the truth, (00:59:03) which isn't always well, I mean, that's (00:59:06) really the problem we're trying to (00:59:06) solve, right? So, it's kind of chicken (00:59:08) egg. Um, (00:59:11) yeah, there's like a lot of different (00:59:12) approaches and a bunch of smart people (00:59:14) working on it. Um, if yeah, if anyone (00:59:17) has suggestions, please send them (00:59:18) through. There's like a lot of different (00:59:20) ways to look at it. So, (00:59:22) >> there's been like moments in time where (00:59:23) I've seen um Elon on X and someone has (00:59:26) said like this is obviously not right (00:59:28) and it's like some Grock output and he's (00:59:30) like we're going to fix this and then (00:59:32) you know 12 hours later, 24 hours later (00:59:33) he's like all right it's fixed. (00:59:35) >> When that happens like what happens (00:59:36) internally. (00:59:37) >> Uh he shows us what went wrong and then (00:59:42) quickly whoever um is awake at the time (00:59:47) it will uh start up a thread to go and (00:59:49) solve it. uh usually individually pull (00:59:51) in a few few others if need be um and (00:59:55) then give a postmortem on what happened (00:59:56) and everyone will understand then what (00:59:59) uh what went wrong and how to avoid it (01:00:00) in the future. Uh, ideally, (01:00:02) >> yeah, the like generally making mistakes (01:00:05) once is okay, but making the same (01:00:07) mistake twice is big problem. (01:00:09) >> Throughout SpaceX's history, there's (01:00:10) been a number of and same thing with (01:00:12) Tesla, there's been a bunch of these (01:00:13) like surges where randomly Elon will (01:00:16) like come in at midnight and say, you (01:00:17) know, like everyone that can come in, (01:00:19) like send out a companywide email and (01:00:20) say like come in, we need to be working. (01:00:22) That sort of thing. Um, has there any (01:00:24) been anything like that? (01:00:26) >> It's more for the big models that that (01:00:27) that happens more than anything. Um, for (01:00:31) Macro Heart specifically, we've been (01:00:32) operating in in a war room for 4 (01:00:35) [laughter] months. (01:00:36) So, so we've kind of always been on that (01:00:38) on that push. (01:00:39) >> Do you guys have like a sign on the door (01:00:40) that says war room? (01:00:41) >> Yeah. (01:00:42) >> Amazing. (01:00:43) >> Actually, well, yeah, we we outgrew the (01:00:45) original war room. Um, and so we moved (01:00:48) everything out and uh I'm told like (01:00:51) walks in to the war room and it's (01:00:52) totally empty and he's like, "Where is (01:00:54) everyone? What?" and he walks over to (01:00:56) where we are now, which is just the gym, (01:00:58) which we cleared out and put everyone in (01:01:00) now, and then conducts his impromptu (01:01:03) questions of what's going on. That was a (01:01:06) long night. [laughter] (01:01:08) >> What is it like on on one of those (01:01:09) nights where a lot of things kind of get (01:01:12) shaken up and and moved forward or like (01:01:14) there's there's one of these searches. (01:01:15) What does that feel like? (01:01:17) >> Um, I think actually I saw this from one (01:01:19) of the co co-founders uh of XA posted (01:01:22) this recently. um Igor uh who was great (01:01:26) to work with. I liked him a lot. It was (01:01:28) actually really cool to work with him. (01:01:28) Side tangent um because his work on um (01:01:32) on Starcraft AI uh way back like (01:01:36) >> I guess 10 years ago now almost was one (01:01:38) of the first like cool ML work that I (01:01:41) tried to replicate myself in high (01:01:42) school. Uh which was very hard. Um it (01:01:45) was really cool. So it was really cool (01:01:46) to work with him. Like I totally never (01:01:49) thought I would get the chance to. Um, (01:01:51) but anyway, uh, I saw him, uh, post this (01:01:54) thing a few days ago where he's like, (01:01:55) "Okay, there there are some, uh, you (01:01:58) know, months where, uh, only a few days (01:02:01) go by and then there's some nights where (01:02:03) months happen." And that was like one of (01:02:06) them for sure. Um, (01:02:08) months might be an exaggeration. I think (01:02:09) we would have gotten to the technical (01:02:10) result we would have in a few weeks (01:02:12) anyway, but doing it in one night was a (01:02:14) huge push and it was a long night. (01:02:18) Has there been any moments where the (01:02:20) company just didn't leave the office for (01:02:22) like 5 days or like a week? (01:02:24) >> Yeah, the surges for the models usually (01:02:27) results in a lot of people staying in (01:02:29) overnight. Um, (01:02:30) >> and you mentioned there's like five or (01:02:31) six pods that people can sleep in and (01:02:33) they like toggle out. (01:02:35) >> Yeah. Yeah. There's some there's some (01:02:36) sleeping pods and we have some bunk beds (01:02:37) now, too. Um, which are less less nice, (01:02:40) >> but they exist. Um, and then when the (01:02:42) tent picture came out, everyone kept (01:02:44) sending that to me and I was like, (01:02:46) honestly, yeah, we have tents, but I've (01:02:48) never seen that many out at once. (01:02:51) [laughter] (01:02:51) Um, so yeah, (01:02:54) >> I know you worked on a bunch of (01:02:55) different projects as a kid. And I think (01:02:57) I don't know if this was their first (01:02:58) one, but it was like fidget spinners and (01:02:59) and and making fidget spinners. Um, I (01:03:01) don't think it was in your garage, but (01:03:02) maybe it was like in your room. (01:03:04) >> Yeah. What kind of stuff like that (01:03:06) tinkering mindset? How much of that have (01:03:08) you kind of taken to this? Uh quite a (01:03:11) bit. Quite a bit. Yeah. Um so I learned (01:03:14) programming when I was quite young. Um (01:03:16) my dad got me a book when I was like 11 (01:03:18) and I liked it a lot. Um well I liked it (01:03:21) a bit but I really started to like it (01:03:22) when I realized you can make money from (01:03:24) it. (01:03:24) >> And so um I I met some people online who (01:03:27) were basically writing scripts for games (01:03:29) as hacks and would sell them online um (01:03:31) for small amounts of money. But you know (01:03:33) making a couple hundred bucks online was (01:03:35) huge for me. Um, (01:03:36) >> I think the first time that you like (01:03:37) have someone give you money, it's the (01:03:39) strangest feeling. (01:03:40) >> Crazy. Yeah. I remember having to ask my (01:03:42) dad for like a PayPal like custody (01:03:44) account or whatever and uh getting the (01:03:46) money in and it was like the coolest (01:03:48) thing of all time ever for me. Um, yeah, (01:03:52) it was really big. And so uh I did that (01:03:55) for um (01:03:58) like a couple months and saved up enough (01:04:00) money to at the time I was really (01:04:02) interested in uh added manufacturing (01:04:03) like 3D printers. RepRap was the big (01:04:06) thing then. So that was kind of where (01:04:08) what kicked off the modern 3D printing (01:04:10) revolution. Uh RepRap was like this (01:04:12) >> built your own, right? (01:04:13) >> Yeah, you had to. That was the only (01:04:14) option. (01:04:15) >> Um RepRap is literally just a bunch of (01:04:19) university students basically um who (01:04:21) said like let's see if we can build a (01:04:23) machine that can build almost all the (01:04:25) components for itself. Um which was that (01:04:27) why it was called RepRap. And uh they (01:04:30) basically built in a variety of (01:04:32) universities these rooms where you start (01:04:35) with one printer um and then it prints (01:04:38) the parts for the next printer and you (01:04:39) go all the way up and you scale up and (01:04:40) there's lots of problems as it turns out (01:04:43) and that's what they were solving and (01:04:45) eventually kicked off like the the (01:04:46) modern 3D printing resolution. Um, but I (01:04:48) was very obsessed with it and so I took (01:04:50) one of their parts list and bought (01:04:51) everything from Alibaba and a month (01:04:52) later things came in and I assembled it (01:04:54) all one night which went poorly actually (01:04:56) when I was uh unbundling the copper (01:04:58) cable for the power supply. Um, which (01:05:00) was a very sketchy power supply and did (01:05:02) catch fire in the end. Um, the all the (01:05:05) copper windings came like loose and (01:05:07) frayed everywhere and one went like 2 in (01:05:09) into my thumb. Um, (01:05:10) >> did you just can't your thumb just (01:05:12) doesn't work or did you go to the (01:05:14) hospital or something? (01:05:15) >> No. So, it was a school night and it was (01:05:16) like 3:00 a.m. cuz I wasn't very good at (01:05:18) building things at 13. Um, and I spent (01:05:22) like an hour in the bathroom trying to (01:05:23) pull it out with tweezers and it just (01:05:25) wasn't it was like it was bad. So, I (01:05:27) just cut it off and I was like, eh. (01:05:29) [laughter] (01:05:30) And so, bit by bit over the next few (01:05:31) weeks it came out and I would snip it (01:05:33) off in the mornings. Um, it was fun. (01:05:35) [laughter] (01:05:36) Um, yeah. Uh, but I got the printer (01:05:39) assembled. Um, and uh around that time, (01:05:42) yeah, the fidget spinner craze was going (01:05:43) off. So, I bought 1 th00and skateboard (01:05:45) bearings from China and basically set up (01:05:46) a little factory uh in my bedroom where (01:05:49) every two hours at night I would wake up (01:05:51) and I would clear the print bed, start a (01:05:52) new print of fidget spinners and I would (01:05:54) sell them online and then um before (01:05:56) school [clears throat] I had a little (01:05:57) assembly line in my garage where I would (01:05:59) uh put in the bearings, spray paint, dry (01:06:01) and then run around to all the other bus (01:06:02) stops of the other schools um sell them (01:06:04) to my distributors which were just uh (01:06:06) other kids of other schools, sell all (01:06:08) day at school, come back, collect from (01:06:09) my distributors and then um sell online, (01:06:12) ship uh built a little healthy business (01:06:15) and uh after 2 months they ended up (01:06:17) getting shut down by the county. Um (01:06:19) their official quoted reason was that (01:06:20) the uh companies that sell the school (01:06:23) food have technically an exclusive (01:06:25) license to sell anything in school (01:06:28) property. But I think they just didn't (01:06:30) like that I was distracting everyone and (01:06:32) making money doing it. Um but it taught (01:06:35) me a good like healthy disrespect for (01:06:36) authority. I think (01:06:37) >> that that has kind of been like a (01:06:39) constant theme. what what does that (01:06:40) actually how is that materialized in (01:06:43) your life with like the healthy (01:06:44) disrespect for authority like what and (01:06:46) you even mentioned um institutions like (01:06:49) you don't like necessarily trust (01:06:50) institutions um (01:06:52) >> how did you kind of come to that and (01:06:54) what does what does that look like (01:06:56) >> um (01:06:58) I I've always known from very young like (01:07:00) I I want an unconventional outcome (01:07:03) >> and so (01:07:05) going through a conventional path would (01:07:08) pretty much necessarily not lead you to (01:07:10) an unconventional outcome. So I grew (01:07:12) opposed to any form of convention and (01:07:16) institutions necessarily enforce (01:07:18) convention. Um I think creativity and (01:07:22) interesting outcomes come mostly from (01:07:24) free-spirited individuals um in almost (01:07:28) every case if not all of them. Um, I (01:07:31) guess it's a bit of a like high (01:07:33) highminded way of saying it, but yeah, (01:07:35) like individuals are (01:07:37) the most creative you can get. And so (01:07:39) staying true to that is the way to go. (01:07:40) >> I do love uh John Carlson's idea of like (01:07:43) everything is so hard to build and so (01:07:46) hard to make, especially, you know, put (01:07:48) into the real world that if you look (01:07:50) around, it's basically like the world is (01:07:51) just filled with some, you know, (01:07:53) people's passion projects. (01:07:54) >> Yeah. It's a total miracle. Um there's a (01:07:56) story behind every little thing. Um way (01:07:59) more than you would think. I remember (01:08:01) reading about the um I think it was YKK (01:08:03) zippers. Apparently every good zipper (01:08:05) like there's two or three companies in (01:08:07) the world that make zippers which are (01:08:08) actually pretty little little miracles. (01:08:10) They're very cheap but also mechanically (01:08:12) comp like relatively complicated for how (01:08:14) much you pay for them. And there's only (01:08:16) a few companies that are capable of (01:08:18) building or have have set up to build (01:08:20) them. Um and it's it's Yeah. Yeah. (01:08:22) basically this one Japanese guy's (01:08:24) passion project over 40 years uh to (01:08:26) figure out how to do this properly. Um (01:08:27) and this is the case for pretty much (01:08:29) everything. Um anything very specific (01:08:32) and at scale is probably only done by a (01:08:34) few companies or a few people in the (01:08:35) world. Um so yeah, I mean you hear about (01:08:38) it every so often, right? Like some (01:08:40) company in Germany, arbitrary company in (01:08:42) Germany shuts down and Volkswagen has to (01:08:43) halt all their lines or something like (01:08:45) that. Um happens all the same. It was a (01:08:48) big thing in co (01:08:49) >> right before we met you had made a (01:08:52) liquid fuel I think rocket engine. (01:08:54) >> Um it was like a very small thing. I saw (01:08:55) it upstairs. Um but you said we were (01:08:58) talking before this that you did it in (01:09:00) like 24 hours just on a whim. Um how did (01:09:03) that happen? (01:09:04) >> Yeah. Um so it was a project over like (01:09:07) roughly four weeks. Um and I started by (01:09:10) literally just buying a bunch of (01:09:11) textbooks. um and trying to figure out (01:09:14) like what are the design principles (01:09:16) behind a rocket engine like how do I (01:09:18) design it? There's not like um you it's (01:09:21) totally different from learning software (01:09:22) where you can just go on GitHub and (01:09:23) download people's code and modify it. (01:09:25) There's no file for a rocket engine. You (01:09:27) have to learn how to like what are the (01:09:29) material properties, what's the chemical (01:09:31) properties, how do you actually machine (01:09:33) it, um how do you design the parameters (01:09:35) and know what to expect from in terms of (01:09:37) thrust output and how do you not over (01:09:39) pressure the engine and all these kinds (01:09:41) of things. Um, how did you design the (01:09:42) injector which is uh the injector was (01:09:44) very hard. That was probably 50% of the (01:09:46) time. (01:09:47) >> Was that the hardest thing? (01:09:48) >> Yeah, the injector was very hard and it (01:09:50) was like the biggest flaw in the end. (01:09:51) Um, so yeah, I spent like 3 4 weeks (01:09:54) doing this and uh expedited a bunch of (01:09:56) parts from China like CNC and all that (01:09:57) stuff. Um, and uh it was right before (01:10:00) Thanksgiving. I was going to go fly back (01:10:02) to the east coast and visit my family (01:10:04) and I was like, "Okay, either I fire it (01:10:05) like build it and fire it tonight. It (01:10:07) was all just a bunch of parts at that (01:10:09) time. uh or I uh do this in two weeks (01:10:12) and I'm like I'm not going to do this in (01:10:13) two weeks. I'm going to do this right (01:10:14) now. So uh I drank a lot of coffee in (01:10:18) the morning and then spend the whole day (01:10:20) like hacking away at at uh aluminum (01:10:22) extrusions and built out the test frame (01:10:24) and then the the engine itself and uh (01:10:26) let it off that night. Um yeah, which (01:10:30) had a lot of um (01:10:33) we'll say (01:10:35) uh concessions made to make it happen (01:10:37) that night. [laughter] (01:10:38) Um, (01:10:39) >> I did find it absolutely hilarious that (01:10:41) you like you said, were you like a (01:10:43) couple feet away? (01:10:44) >> Yeah. So, I designed it like I wasn't (01:10:47) stupid. I designed it so that I could (01:10:48) remotely fire it, but um I didn't the (01:10:51) power supply hadn't come in yet to to (01:10:53) remotely power the computer that was on (01:10:55) board. So, I had to use a USB cable from (01:10:57) my laptop to power the onboard computer. (01:10:59) And I didn't have a long enough USB (01:11:00) cable. Uh, the longest one I had was (01:11:03) like 6 foot. So, I had to stand right (01:11:05) next to it and light it up. And I was (01:11:08) like, there's like a 30% chance that (01:11:09) this thing explodes or or launches fire (01:11:12) everywhere. And actually, um I don't (01:11:14) know if it shows in the video. I think (01:11:15) it does show in the video, but my jacket (01:11:16) did catch fire [laughter] (01:11:18) because I I wasn't that great at (01:11:20) designing the injector and it did create (01:11:22) a lot of over pressure events, which (01:11:23) meant there was a lot of basically uh (01:11:24) unburnt fuel spewing out, which was (01:11:28) ethanol. And so that's liquid and just (01:11:30) landed some landed on my on my jacket (01:11:33) and caught fire. Um so yeah, that's a (01:11:35) trophy still, the burnt jacket.