U.S. Defense Strategist on How AI Drone Warfare Could Spiral Out of Control (YouTube Video Transcript)

(00:00:00) Your YouTube transcript will appear here (00:00:00) How do you end a war that's happening at (00:00:03) superhuman speeds? If our competitors go (00:00:06) to terminators and their decisions are (00:00:09) bad, but they're faster, how would we (00:00:11) respond? There's this incentive towards (00:00:13) faster reaction times and decision-m. (00:00:15) They have to go faster to keep up. I (00:00:17) think we have a really interesting (00:00:18) example in financial markets, stock (00:00:21) trading, where humans can't possibly (00:00:23) intervene in milliseconds. And then (00:00:24) we've seen examples like flash crashes. (00:00:26) Could we have something like a flash war (00:00:29) where interactions are so fast that they (00:00:31) escalate in ways that humans really (00:00:34) struggle to control? The ugly reality is (00:00:37) likely to be that politically people (00:00:40) will have to suffer and die for wars to (00:00:42) end. (00:00:47) Today I'm speaking with Paul Sharie. (00:00:49) Paul is a former US Army Ranger who (00:00:51) served in Iraq and Afghanistan, the (00:00:54) current vice president and director of (00:00:55) studies at the Center for a New American (00:00:57) Security and the award-winning author of (00:01:00) two books, Army of None, Autonomous (00:01:02) Weapons and the Future of War, and Four (00:01:04) Battlegrounds: Power in the Age of (00:01:06) Artificial Intelligence. He also worked (00:01:08) at the Pentagon where he led the team (00:01:10) that wrote the US military's first (00:01:12) policy on autonomous weapons. Thanks for (00:01:14) coming on the podcast, Paul. (00:01:16) >> Thanks for having me. Very excited for (00:01:17) the discussion. you expect AI and (00:01:20) automation to transform the nature of (00:01:22) war. Uh can you talk concretely about (00:01:24) what that will look like? (00:01:26) >> So I think we're already starting to see (00:01:29) artificial intelligence play an (00:01:30) important role on modern battlefields. I (00:01:33) think over time we're going to see AI (00:01:35) take on more of the cognitive dimensions (00:01:38) of warfare (00:01:40) and get to a place where and this might (00:01:42) take several decades. the speed and (00:01:45) tempo of war could start to really push (00:01:49) at the boundaries, maybe even exceed (00:01:51) them of what humans can do. So you can (00:01:53) envision a world in the future where you (00:01:55) sort of get this tipping point, what (00:01:58) some Chinese scholars have called a (00:02:00) battlefield singularity, an idea that (00:02:04) the speed and tempo of war outpaces (00:02:06) human control and war at a large scale (00:02:09) shifts to really a domain of machines (00:02:12) and machines making decisions. (00:02:15) >> Okay. Yeah. the the idea of a (00:02:16) battlefield singularity um is extremely (00:02:21) extremely interesting to me. Um but I (00:02:23) want to step back briefly and just kind (00:02:25) of try to understand how exactly AI will (00:02:28) be integrated into uh kind of weapon (00:02:30) systems. Um and I guess yeah then how (00:02:33) it'll affect how wars are fought and (00:02:34) won. Um, so if I understand correctly, (00:02:38) uh, autonomous weapon systems are kind (00:02:40) of complete weapons platforms that kind (00:02:43) of once activated, um, can select and (00:02:46) engage targets, uh, without further (00:02:48) human intervention. Um, can you give a (00:02:50) couple examples of the kind of most (00:02:53) advanced autonomous weapon systems that (00:02:55) are being developed today? (00:02:56) >> Sure. So conceptually an autonomous (00:02:59) weapon is really one where the weapon (00:03:01) itself is making its own decisions on (00:03:03) the battlefield about whom to kill. I (00:03:05) think that I'll make an analogy for us (00:03:07) to cars. (00:03:10) Conceptually the idea of a self-driving (00:03:12) car is pretty straightforward and you (00:03:14) can imagine cars that don't even have a (00:03:16) steering wheel. Certainly people have (00:03:18) have designed them um where the car is (00:03:20) totally driving itself. Now in practice (00:03:22) as we see the technology evolving is (00:03:24) these sort of incremental movements you (00:03:27) have um sort of incremental advances in (00:03:31) autonomy automation in cars like (00:03:33) intelligent cruise control automatic (00:03:36) lane keeping automatic braking automatic (00:03:39) self-parking (00:03:40) um you know Tesla that has more kind of (00:03:43) incremental self-driving features (00:03:46) but you can you can see sort of this (00:03:47) path towards a completely self-driving (00:03:50) are we're seeing a very similar thing (00:03:52) inside militaries where militaries are (00:03:54) incrementally automating different tasks (00:03:57) that are used in finding targets in (00:04:01) processing those targets in presenting (00:04:03) that information to decision makers in (00:04:06) missiles and drones that would be able (00:04:08) to carry out attacks. What is can you (00:04:11) give a few examples of like as it (00:04:14) advances maybe quite significantly like (00:04:18) what are longer term visions for how AI (00:04:21) and autonomy um could could kind of make (00:04:24) really big differences to how these (00:04:26) weapon systems work and I guess there (00:04:28) I'm sure there are like loads of (00:04:29) different places these things will be (00:04:31) integrated so I'm maybe I'm most curious (00:04:32) about the places that um could most (00:04:35) drastically change uh how how wars are (00:04:38) on one. (00:04:40) >> Yeah. So I think one major paradigm (00:04:43) shift that um could occur, I think it's (00:04:46) probably eventually likely to occur over (00:04:47) the next several decades is towards (00:04:50) swarming warfare where you could have (00:04:53) you can imagine large numbers of (00:04:55) autonomous drones on the air, at sea, (00:04:58) undersea, on land that areworked that (00:05:02) are working together cooperatively and (00:05:05) autonomously adapting their behavior on (00:05:07) the battlefield to respond to events. (00:05:10) So, right now we're seeing massive (00:05:13) numbers of drones deployed in Ukraine. (00:05:16) Um, you know, certainly tens of (00:05:17) thousands of drones on the front lines, (00:05:19) but those drones are not only remotely (00:05:22) controlled for the most part, they're (00:05:23) not really working cooperatively in any (00:05:26) way. So even if humans had the ability (00:05:29) autonomously for the drone to go out and (00:05:31) find its own target, having 10,000 (00:05:33) drones that are independently finding (00:05:35) targets is very different than 10,000 (00:05:38) drones that are working cooperatively (00:05:39) together. And you could have much more (00:05:42) dramatic effects in the battlefield by (00:05:45) having swarms that are able to uh (00:05:49) simultaneously attack from multiple (00:05:51) directions, have self-healing (00:05:53) communications networks, self-healing (00:05:55) minefields. Uh the ability to to react (00:05:59) to what humans are doing to what the (00:06:01) enemy is doing in real time and at a (00:06:03) faster not only speed but also scale of (00:06:06) coordination than is possible with (00:06:08) humans. And this is I think the the real (00:06:10) dramatic change here is not actually in (00:06:12) the physical technology. I mean drones (00:06:15) are are interesting. They could do neat (00:06:16) things but it's in the sort of cognitive (00:06:18) dimension and in particular here of what (00:06:21) the military would call command and (00:06:22) control. So militaries today are (00:06:24) organized in this very hierarchical (00:06:25) fashion. You have teams and squads and (00:06:29) platoon companies and battalions and you (00:06:31) have these um them organized (00:06:34) predominantly because of the limitations (00:06:36) of human cognition. So if you put a (00:06:38) human commander in charge of 10,000 (00:06:41) soldiers and just like they were (00:06:44) directly issuing orders to each 10,000 (00:06:46) like that would be totally impractical. (00:06:47) There's no way to do that. That's not (00:06:48) how um militaries are organized. That's (00:06:51) not how corporations are organized. Um (00:06:54) if you look at sports, it's really kind (00:06:56) of interesting that um you know sort of (00:06:59) a lot of team sports have somewhere (00:07:00) between maybe five to a dozen or so (00:07:04) players on the field. Now imagine a game (00:07:06) of soccer um where you had a 100 players (00:07:11) on each side and 50 balls, (00:07:14) right? You'd have to have a completely (00:07:15) different way of organizing that. But (00:07:17) robots or swarms could do that (00:07:19) differently. They could perfectly (00:07:20) coordinate their behavior and ensure (00:07:24) that they're optimally using those (00:07:27) resources to, you know, hit the soccer (00:07:30) balls, go after the enemy targets, (00:07:32) whatever it is. And so I think that (00:07:33) that's a a potentially really dramatic (00:07:35) shift in how militaries fight in the (00:07:36) future. (00:07:37) >> There is this vision of a possible (00:07:40) future that as militaries integrate (00:07:42) artificial intelligence and autonomy (00:07:44) more fully across the force that we (00:07:47) might reach some tipping point where the (00:07:50) pace of combat action is just too fast (00:07:53) for humans to respond and humans have to (00:07:55) be completely out of the loop. And I (00:07:58) think what's scary about that possible (00:08:00) vision is that humans are then no longer (00:08:03) in control of violence and warfare. And (00:08:07) that raises moral questions, but it also (00:08:09) raises just really fundamental questions (00:08:11) of how do you control escalation in (00:08:13) wartime. How do you end a war that's (00:08:16) happening at superhuman speeds? And we (00:08:19) don't have good answers for that. And I (00:08:21) think maintaining human control over (00:08:23) warfare is absolutely essential to (00:08:27) making sure that we can navigate this (00:08:29) transition towards more powerful AI in a (00:08:32) safe way. (00:08:33) >> Okay. So just to (00:08:36) make sure uh we get kind of a concrete (00:08:39) picture of what um this kind of (00:08:41) battlefield singularity or sometimes (00:08:43) called hyperwar would look like. Um can (00:08:46) you kind of describe Yeah. What does it (00:08:48) look and feel like? what kinds of (00:08:50) weapons, how have they been automated? (00:08:52) What do conflict engagements look like? (00:08:54) You know, are there are there any humans (00:08:56) in the loop at any level? (00:08:58) >> So, let's start with where we are today. (00:09:00) >> And I want to kind of paint a picture (00:09:02) for how that might grow over time. So (00:09:04) since at least the 1980s, countries have (00:09:07) had automated air and missile defense (00:09:09) systems that can shoot down incoming (00:09:12) threats when the speed of these incoming (00:09:15) missiles or rockets or artillery or (00:09:17) aircraft are just too fast for humans to (00:09:20) respond. So for example, a US Navy (00:09:23) warship has an automated mode on the air (00:09:27) and missile defense system that can be (00:09:28) activated where there might be missiles (00:09:31) coming in and there's just there's just (00:09:33) so little time for humans to respond and (00:09:36) you might have multiple threats coming (00:09:37) from different directions that then the (00:09:39) machine once it's activated by people (00:09:42) can automatically sense all these (00:09:44) threats and shoot them down. (00:09:45) >> Now we've had these systems around for (00:09:48) decades. They haven't really been widely (00:09:50) used in conflicts in these automated (00:09:52) modes and there have been a couple (00:09:55) examples of accidents. There was a (00:09:57) fratricside a couple fratricside (00:09:58) incidents in 2003 with the Patriot air (00:10:01) and missile defense system. (00:10:04) But that's something that we have some (00:10:05) experience with that there is this sort (00:10:08) of very narrow domain today of machine (00:10:12) control over warfare where machines (00:10:15) really humans just can't be in the loop (00:10:17) in this in this area. I think what I (00:10:21) would envision is that this domain of (00:10:24) machine warfare grows over time and that (00:10:27) several decades from now we end up in a (00:10:29) world where something like that exists (00:10:31) at a much larger scale along the entire (00:10:34) front where there are swarms of (00:10:37) thousands of drones on both sides and (00:10:40) they're dynamic and responding to enemy (00:10:42) behavior and there are missiles being (00:10:45) launched and striking targets and there (00:10:48) are AI systems identifying new targets (00:10:50) which are moving and mobile and humans (00:10:52) can't possibly be in the loop to respond (00:10:54) to that enough. It's too slow. And (00:10:57) humans are are maybe observing this (00:11:00) action. Maybe you could think the way (00:11:02) like a coach might on the sidelines, (00:11:04) >> right? And humans could have some degree (00:11:07) of direction of okay, I'm going to um (00:11:10) you know change the higher level (00:11:12) guidance for these systems or I might (00:11:16) try to add new parameters to sort of the (00:11:20) operating systems. But humans can't (00:11:21) really in real time intervene. And I (00:11:24) think we have a a really interesting (00:11:26) example of this exact kind of behavior (00:11:30) in financial markets, stock trading, (00:11:32) where there's this whole new domain of (00:11:34) highfrequency trading um where humans (00:11:37) can't possibly intervene in milliseconds (00:11:39) that these algorithms are responding (00:11:42) and then we've seen examples like flash (00:11:43) crashes that come with that. And so I (00:11:46) think the scary (00:11:48) sort of analogy there would be well well (00:11:52) could we have something like a flash war (00:11:54) where the interactions are so fast that (00:11:57) they escalate in ways that humans really (00:12:00) struggle to control. Um and I think (00:12:04) that's a a really scary proposition. How (00:12:06) do you find ways to stop that? that in (00:12:09) financial markets they put in place (00:12:10) circuit breakers that can take traiting (00:12:14) offline if um we see movements that are (00:12:17) too volatile. There's no good way to do (00:12:19) that in warfare. There's no referee to (00:12:20) call timeout. (00:12:21) >> And I think that's you know how do you (00:12:23) maintain human control over warfare (00:12:24) that's happening at superhuman speeds. (00:12:27) >> Can you talk about the incentives that (00:12:29) lead to this kind of uh inevitable speed (00:12:32) up and taking of humans out of the loop? (00:12:34) um given that my sense is that currently (00:12:37) uh the defense department and other (00:12:39) others who kind of are going to be in (00:12:41) charge of these decisions do not want to (00:12:43) take humans out of the loop. Um so why (00:12:45) why does that thing seem like a likely (00:12:47) thing that's going to happen and kind of (00:12:49) how does that drive things faster? (00:12:51) >> So I think this yeah that's a great (00:12:53) question. And I think this pushpull (00:12:56) um is very common in these types of (00:12:59) major revolutions and military affairs (00:13:02) where you have old institutions and ways (00:13:05) of fighting that are not necessarily (00:13:07) super enthusiastic about the new way of (00:13:09) fighting. um you know the cavalry for (00:13:12) example wasn't particularly enthusiastic (00:13:14) about tanks right and and (00:13:17) right now certainly within the US (00:13:19) military there's a strong belief that (00:13:22) humans should remain quote in the loop (00:13:24) that's not actually official US policy (00:13:26) but certainly when you hear US military (00:13:29) senior military officers talk about it (00:13:30) they'll talk about it that way that they (00:13:31) want humans in control and I think (00:13:33) because there's just a healthy (00:13:34) skepticism for all the reasons that (00:13:37) everyone's interacted with AI could (00:13:39) understand about these systems that like (00:13:41) well you know sometimes they get it (00:13:43) wrong and there's value in humans making (00:13:46) these decisions. I think the ultimate (00:13:48) arbiter on the is what works on the (00:13:51) battlefield and so that is what will (00:13:54) drive how militaries change. Militaries (00:13:57) tend to be often very conservative with (00:14:00) these um types of changes in part (00:14:03) because you never really know what's (00:14:05) going to work until militaries fight a (00:14:07) war. (00:14:08) >> Right. Okay. And so the idea is there is (00:14:10) this conservatism um and maybe maybe it (00:14:13) takes decades but eventually (00:14:16) uh the technology I mean it's my my (00:14:21) suspicion that the technology just is (00:14:23) very likely to improve enough that um (00:14:27) you're really disadvantaging yourself if (00:14:30) you don't use it. Does that sound right (00:14:32) to you? (00:14:35) I think that there's there's a (00:14:36) trajectory towards greater automation (00:14:39) and greater speed and tempo of war. Um I (00:14:43) do think that militaries have choices (00:14:45) about exactly how they implement that (00:14:47) technology. And the sort of important (00:14:51) thing for militaries, this is actually (00:14:53) true of most military technical (00:14:54) revolutions, what matters most is not (00:14:56) actually um getting the technology first (00:14:59) or even having the best technology in (00:15:01) some sense. It's figuring out the best (00:15:03) ways of using it. It's figuring out like (00:15:05) what do I do with a tank? What do I do (00:15:07) with an airplane for example? And um I (00:15:12) think you know there are there's value (00:15:14) to human cognition. Um there are lots of (00:15:18) types of cognitive problems that at (00:15:20) least today are very challenging for AI. (00:15:24) And even if AI is cognitively better, (00:15:26) there's probably value in keeping humans (00:15:28) in control of warfare. Um the question (00:15:31) is sort of how to maintain that balance (00:15:33) in the best possible way. Um and I think (00:15:37) that that's going to be a really (00:15:38) important question in the next several (00:15:40) decades. (00:15:41) >> But do you think that that value is (00:15:45) really likely to persist long enough (00:15:47) that at some point at least one country (00:15:50) decides that taking the human out of the (00:15:53) loop is strategic and then does better. (00:15:58) And if they do better, that creates this (00:16:01) pressure for their adversaries to take (00:16:03) them out of the loop. (00:16:04) >> So, former Deputy Secretary of Defense (00:16:06) Bob Work, who was really a pioneer in (00:16:08) bringing artificial intelligence into (00:16:10) the US military, has this quote of, "If (00:16:13) our competitors go to terminators (00:16:16) and their decisions are bad, but they're (00:16:19) faster, how would we respond?" which is (00:16:22) kind of a colorful way to for a senior (00:16:24) leader to be talking about throwing the (00:16:25) terminators. But I think it does (00:16:27) highlight this really difficult problem (00:16:29) of this potential for an arms race in (00:16:33) speed in militaries that there's this (00:16:37) incentive towards faster reaction times (00:16:39) and decision-m that might pressure (00:16:41) militaries to do the same. Even if they (00:16:43) don't want to, if they're not (00:16:44) comfortable with that, they have to go (00:16:45) faster to keep up. similar to what we've (00:16:48) seen in financial markets with high (00:16:50) frequency trading. And that could lead (00:16:53) to a sort of dangerous situation where (00:16:55) you have this dangerous arms race in (00:16:58) speed in the military. And I've heard (00:17:00) some people argue we ought to have some (00:17:02) limits on that. How do you know put a (00:17:04) speed limit on warfare? Seems like an (00:17:06) appealing idea. I don't know how you do (00:17:07) that in practice to sort of try to put (00:17:10) brakes on this tendency which is I think (00:17:13) a big risk as militaries are adopting AI (00:17:15) and autonomy. (00:17:16) >> Yeah. interesting. (00:17:17) >> I think for autonomous weapons, yes, the (00:17:20) answer is going to be that there are (00:17:21) strong competitive pressures to take (00:17:22) humans out of the loop, at least in (00:17:24) certain kinds of scenarios. I guess what (00:17:26) I would say when I certainly talk to to (00:17:28) folks in the US Defense Department, um, (00:17:31) is that there's still a lot of value (00:17:34) that humans bring and that these (00:17:36) machines can make mistakes. And so we (00:17:37) want to, (00:17:38) >> you know, let's all sort of throw out (00:17:40) human cognition. Um, and even if we (00:17:44) think our systems work great in testing, (00:17:45) like we don't know what the enemy is (00:17:46) going to do. We don't know how our (00:17:47) systems might interact with the enemy. (00:17:50) And so, particularly in (00:17:53) um, you know, there would be some (00:17:55) battlefield applications where it it (00:17:57) might make sense to say, okay, using (00:17:59) autonomous weapons in this area is fine. (00:18:01) I think so. But certainly in um (00:18:05) controlling escalation, I think human (00:18:07) control is really really important. If (00:18:08) you imagine something like the Cuban (00:18:10) missile crisis, And if we add a bunch of (00:18:12) autonomy, (00:18:14) I'm not sure that makes it safer, right? (00:18:17) To me, that makes sort of the whole (00:18:19) situation (00:18:21) much more brittle and more likely to (00:18:24) escalate in dangerous ways because you (00:18:27) can give humans this you can give humans (00:18:30) really ambiguous guidance. You can tell (00:18:31) humans things like, "Look, you're (00:18:34) allowed to use force to defend yourself, (00:18:36) but don't start a war." (00:18:38) And a human can understand that, right? (00:18:40) And they might be like, well, I don't (00:18:41) really know what that means in practice. (00:18:43) But we're basically saying, look, I (00:18:44) trust you that you're pretty smart and (00:18:46) like figure it out as best you can based (00:18:48) on the situation that you're in. And (00:18:50) humans can handle that. And I don't know (00:18:52) that AI systems, even as a starter, (00:18:54) won't necessarily understand the (00:18:56) consequences of those actions in the (00:18:57) same way. Maybe they will. Um, but I (00:19:00) this is a place where I am a little bit (00:19:01) conservative. (00:19:03) >> Okay. I I would love to talk about um AI (00:19:06) in kind of the nuclear command and (00:19:09) control space. (00:19:10) >> Things aren't scary enough. (00:19:12) >> Exactly. Exactly. Let's just escalate it (00:19:14) a bit further. Um so nuclear uh command (00:19:18) control and communications is this (00:19:20) bucket that includes kind of sensors and (00:19:23) analysts command nodes communications (00:19:25) links um and procedures used to detect (00:19:28) nuclear attacks um and then decide how (00:19:30) to respond and then order and execute (00:19:32) that response. So it's this huge kind of (00:19:34) bucket that all relates to detecting and (00:19:38) responding to potential nuclear threats. (00:19:41) um what is the argument for building AI (00:19:44) into these systems? (00:19:47) >> So I think the argument for integrating (00:19:49) AI or automation into nuclear operations (00:19:52) is that it's absolutely crucial to for (00:19:55) these systems to operate at a very very (00:19:57) high degree of performance and (00:19:59) reliability. And so within the nuclear (00:20:01) space there's this concept of the always (00:20:04) never dilemma. And what this means is (00:20:07) that you always want nuclear weapons to (00:20:10) be used if there is an authorized order (00:20:14) from the president or whoever the (00:20:16) command authority is for these systems (00:20:19) to have a breakdown there undermines (00:20:22) nuclear deterrence. And so if your (00:20:24) adversary knows that there's holes in (00:20:26) your system and like if the president (00:20:28) says use nuclear weapons, maybe maybe it (00:20:31) doesn't happen or maybe they don't (00:20:32) launch, that might create incentives for (00:20:35) an adversary to engage in risky (00:20:37) behavior. (00:20:39) But on the other hand, you never want (00:20:41) nuclear weapons to be used when there's (00:20:43) not an authorized signal for launch, (00:20:46) either by accident or by some rogue (00:20:49) actor doing that. Now, that's really (00:20:51) hard from an engine either technical or (00:20:54) sort of a social engineering standpoint. (00:20:56) You can imagine that there's lots of (00:20:57) safeguards that you can put in place (00:20:59) against an accident or an accidental (00:21:01) launch or unauthorized launch occurring (00:21:02) that then make it just harder for an (00:21:05) authorized launch to occur. Um, and I (00:21:08) think the idea is that you could use AI (00:21:10) to sharpen this distinction if you do it (00:21:13) right to um make things both more (00:21:17) reliable when needed and safer and on (00:21:22) the detection side that you could use AI (00:21:25) to have greater visibility on what (00:21:28) adversaries are doing and buy more time (00:21:31) for decision makers. like a fundamental (00:21:33) problem in the nuclear space is (00:21:35) >> let's say you get one we've had these (00:21:37) incidents um both in the United States (00:21:39) and and in the uh Soviet Union and and (00:21:42) Russia after uh the fall of the the (00:21:45) Soviet Union where there'll be an alarm (00:21:49) that goes off that looks like a missile (00:21:52) launch (00:21:54) and right now decision makers have (00:21:56) minutes to make a decision and the (00:21:58) problem is is that even though um both (00:22:02) the US and Russia have invested in their (00:22:04) nuclear arsenals in ways that ought to (00:22:06) allow them to survive a first strike and (00:22:07) still retaliate that you could really be (00:22:11) severely disadvantaged by a major major (00:22:13) first strike. And so there are (00:22:14) incentives to what's called launch on (00:22:16) warning. So you have this warning that (00:22:17) missiles are inbound. You need to launch (00:22:18) your missiles before they get taken out (00:22:20) in their silos. Um and there's not much (00:22:23) time. So if you could buy automate if (00:22:25) you could use automation to buy more (00:22:26) time for decision makers to speed up the (00:22:28) process and then make that information (00:22:30) more reliable that would be really (00:22:32) valuable. So I think there is like a (00:22:34) legitimate case for using AI and to be (00:22:37) fair like there are lots of places where (00:22:38) automation is already used today (00:22:41) in actually nuclear operations. Um but (00:22:44) boy there are ways you can get it wrong (00:22:46) too and that's what's scary. (00:22:47) >> Yeah. So I find these arguments in favor (00:22:52) um of more automation and more AI uh in (00:22:55) kind of nuclear command control and (00:22:56) communications very alluring. Um but I (00:22:58) have the sense that you are keen to (00:23:00) proceed with caution. Um can you talk (00:23:03) through one or two of the failure modes (00:23:05) that worry you most? (00:23:07) Maybe a good place to start is with (00:23:09) Stannis La Petrov (00:23:12) because it's this really clear example (00:23:15) of what how things could go wrong in a (00:23:18) really bad way. So (00:23:21) Stanos Petro is this lieutenant colonel (00:23:23) in the Soviet military. He's a watch (00:23:26) officer on duty on a night in September (00:23:30) in 1983. (00:23:32) and he's sitting in this bunker outside (00:23:34) Moscow and the system says that there's (00:23:37) a missile launch from the United States. (00:23:39) United States has launched according to (00:23:40) the system a nuclear tipped (00:23:42) intercontinental ballistic missile (00:23:43) against the Soviet Union. (00:23:46) And then the system says there's another (00:23:48) launch and another and another five (00:23:50) missiles total inbound. And there's this (00:23:53) big Petro describes that there's this (00:23:57) big sort of backlit red screen and it's (00:24:00) saying um missile launch (00:24:03) um and and he has really not much time (00:24:07) to decide what am I going to do here. (00:24:08) Now he knows a couple things that are (00:24:12) sort of outside the details of the the (00:24:14) system itself. one, he knows that the (00:24:18) Soviet Union had just deployed a new (00:24:20) satellite-based early warning system, (00:24:22) and he knows a lot of technology like (00:24:24) just doesn't work. And so, he doesn't (00:24:26) necessarily trust the system from the (00:24:28) get-go. He also thinks that launching (00:24:30) five missiles just doesn't make sense. (00:24:32) Like, if the US were to launch a massive (00:24:34) strike, you'd launch all the missiles, (00:24:37) why like poke the bear? It just doesn't (00:24:39) it doesn't strategically make any sense (00:24:41) when he thinks about what the US might (00:24:42) do. So he calls the early warning radars (00:24:47) which as the missiles are allegedly (00:24:49) coming should be able to see the (00:24:50) missiles coming over the horizon. They (00:24:52) said we don't see anything. So Petrov (00:24:55) says later that he thought it was 50/50 (00:24:58) whether this was legit, but he had a (00:25:00) funny feeling in his gut. Didn't make (00:25:02) sense. So he tells his superiors (00:25:04) systems malfunctioning. (00:25:08) And the scary thing about this is what (00:25:11) would an automated system have done (00:25:14) whatever it was programmed to do. And it (00:25:17) certainly wouldn't have known (00:25:19) necessarily the ability to kind of step (00:25:21) outside that situation and and say, (00:25:24) well, you know, this is a new system. (00:25:27) Should I trust it or this type of attack (00:25:29) just doesn't make any sense? And then (00:25:31) come to that conclusion that he did and (00:25:33) it certainly wouldn't have known the (00:25:35) stakes, right? Which which Petrov (00:25:37) understood, right? That boy, like if we (00:25:40) get this wrong, like a lot of people are (00:25:41) going to die. And so I think that's kind (00:25:43) of the the worrying this really stark (00:25:48) illustrative example of the stakes if we (00:25:52) get this wrong. (00:25:54) >> Yeah, I'm I'm completely with you on the (00:25:56) stakes. Um and I've found it incredibly (00:26:00) incredibly unsettling to learn about all (00:26:02) of the near misses that we've had. Um, (00:26:05) and yeah, I guess so, so I guess I'm (00:26:09) with you on that, but um, and I don't (00:26:12) want to be naive and fall prey to kind (00:26:15) of magical thinking about how great AI (00:26:19) could be in theory. Um, but it does feel (00:26:22) to me like (00:26:25) an automated system with quite (00:26:27) hard-coded (00:26:28) uh programming might respond to that (00:26:31) situation (00:26:33) in in a different way to Petrov in a way (00:26:36) that is catastrophic. Um, but when I (00:26:41) chat to LLMs, it it does seem like (00:26:44) they're able to one, you know, be (00:26:47) programmed uh not just programmed, but (00:26:49) but trained to be flexibly conservative. (00:26:53) Um, also to have a bunch of context (00:26:56) including uh what the stakes are and how (00:27:02) how strikes are likely to play out such (00:27:04) that they would have some sense of like (00:27:06) the ability to reason about why one (00:27:09) might see five warheads coming or not. (00:27:12) Um, is it naive to think that (00:27:17) AI systems might actually be able to (00:27:20) have enough context that they'd have (00:27:22) made similarly good judgments to prov? (00:27:26) >> I mean, they might maybe, (00:27:29) >> but like how reliably, I think, would be (00:27:31) the question, right? And so um the the a (00:27:35) couple challenges I think in using AI in (00:27:38) this and I think there are legitimate (00:27:39) ways to use AI in nuclear operations for (00:27:41) example using image classifiers to track (00:27:44) mobile missiles uh missile launchers for (00:27:46) example great great potential use case. (00:27:48) >> Okay. Um but (00:27:51) like the problem with some system making (00:27:54) some determination is in particular like (00:27:57) what is the training data that you use (00:28:02) for a surprise attack like we don't (00:28:05) actually know what that looks like and (00:28:08) we know that AI systems are going to (00:28:09) perform often perform quite badly when (00:28:13) pushed outside the bounds of their (00:28:15) training data and so that if you prevent (00:28:17) it with a novel situation like maybe you (00:28:18) get something good, maybe you don't. Um (00:28:22) and and so there's this really (00:28:24) interesting example from the 80s where (00:28:26) the Soviets put this created this (00:28:27) intelligence system called V Ryan that (00:28:30) was designed to um predict the (00:28:34) likelihood of a surprise US attack. And (00:28:37) what it was designed to do was collect (00:28:39) data on all of these things that um the (00:28:43) Soviets thought might be indicators that (00:28:44) the US is preparing for a surprise (00:28:47) attack on the Soviet Union. So things (00:28:49) like the US stockpiling blood in blood (00:28:51) banks, the locations of senior US (00:28:54) political and military leaders, and so (00:28:57) you could see indicators of maybe like, (00:28:58) okay, it looks like maybe they're (00:29:00) they're getting ready for something. (00:29:01) >> Yeah, (00:29:02) >> that sounds actually like a really (00:29:04) interesting use of automation. Yeah. (00:29:06) >> What happened in practice was KGB agents (00:29:09) were basically incentivized (00:29:11) to generate reports and feed data into (00:29:14) the system and the data that was coming (00:29:16) in was just bogus (00:29:18) >> because people were judged based on like (00:29:20) going out and getting information and (00:29:22) bringing it in. And so the whole thing (00:29:23) was sort of trained on just bad data (00:29:26) >> and um or relied on bad data rather. And (00:29:30) so I think that's you know an example of (00:29:32) some of the flaws of these systems. you (00:29:33) could imagine um some AI intelligence (00:29:37) system that's looking at all of these (00:29:40) different indicators, right? Um troop (00:29:44) activity, uh the locations of senior (00:29:46) political leaders and um you know where (00:29:51) we see their nuclear submarines and (00:29:54) bombers and mobile missile launchers (00:29:56) being moved and sort of comes to some (00:30:00) judgment. Okay, this is this is my (00:30:02) probability. (00:30:03) One of the problems is like how do we (00:30:05) verify that that's accurate, right? Like (00:30:08) we can verify that a lot of other AI (00:30:10) things are good and are performing well (00:30:12) because we can test them in their actual (00:30:13) operating environment. Um we can look at (00:30:16) image classifiers and we can get to like (00:30:17) ground truth of what is the thing is it (00:30:19) accurate. We could take self-driving (00:30:20) cars and drive them running in the (00:30:21) operating environment. In this case, we (00:30:23) wouldn't have any great way to measure (00:30:24) the baseline like if just like is it is (00:30:27) it good at this at all? And then of (00:30:29) course a lot of AI systems are super (00:30:30) opaque. So let's say your AI system (00:30:33) says, "Okay, I think there's a 70% (00:30:35) probability that there's an attack. (00:30:37) Why?" And maybe it can tell you (00:30:40) something, but that doesn't necessarily (00:30:43) mean that the story it's telling you is (00:30:45) accurately reflective of the underlying (00:30:47) cognitive processes inside that neural (00:30:49) network, of course. And so I think it's (00:30:52) a (00:30:54) it seems like a really dicey way to use (00:30:57) AI. I do think that over time militaries (00:31:00) are going to start to integrate and (00:31:01) intelligence communities AI in this (00:31:04) fashion. I think they're likely to be (00:31:05) very conservative though. (00:31:06) >> Um which is probably not a bad thing in (00:31:08) this case. (00:31:09) >> Yeah. (00:31:10) Yeah. Okay. So I think I'm I am moved by (00:31:14) the fact that um you won't be able to (00:31:18) test these in real world situations. (00:31:20) Maybe at best you'll have like war games (00:31:22) where um people are making moves that (00:31:27) they would think they would in the real (00:31:29) world and you can you can train the (00:31:32) system on those outcomes. Um but you (00:31:34) cannot train them on real world examples (00:31:37) of nuclear exchanges and escalations. (00:31:40) Um, and then I also just buy that. (00:31:43) Yeah, I guess when I'm like in (00:31:46) particular the prov situation, (00:31:49) I can imagine a model reasoning well (00:31:51) about that. But there are so many many (00:31:54) many ways um that all of these variables (00:31:57) could come together and maybe it is just (00:32:00) pretty pretty dicey um to expect them to (00:32:03) perform well in all of those. I guess (00:32:08) do you have a take on (00:32:11) like it feels like it's important that (00:32:13) we're comparing these systems to humans (00:32:16) and humans are also fallible. Um but is (00:32:18) your is your kind of overall take like (00:32:21) yes humans are fallible but they just (00:32:23) will be better at this for a very long (00:32:25) time maybe indefinitely. (00:32:28) >> I do think there's there's certainly (00:32:29) value in in human judgment. Um I I think (00:32:33) maybe one thing I would say is I think a (00:32:35) little bit depends on how militaries or (00:32:38) intelligence services might employ AI in (00:32:42) different kinds of high-risisk (00:32:43) scenarios. So in this case for example, (00:32:45) okay, you're trying to figure out is are (00:32:47) these indicators of a true attack or a (00:32:49) false attack. I think an use case that I (00:32:52) would not think is wise would be to have (00:32:55) this AI predictive system that sort of (00:32:56) predicts the likelihood of an attack. I (00:32:58) just think that for all the reasons it's (00:33:00) not a good idea. Now, here's a use case (00:33:01) that might be might be valuable. You (00:33:04) could have an AI system that's fed all (00:33:06) this information and the question you (00:33:08) ask it and the thing that you've trained (00:33:09) it to do is make the case for me why (00:33:14) this is a false alarm. (00:33:16) >> We have a just like crazy crazy number (00:33:19) of false alarms. It's a terrifying (00:33:21) somebody puts the wrong tape into the (00:33:22) system and you know like a lot of scary (00:33:25) things and that could be an interesting (00:33:29) use case of you're in this moment and (00:33:31) humans are like not sure how to respond (00:33:33) and you turn to the AI system and the AI (00:33:36) system is like well have you thought (00:33:37) about these factors and it's oh like (00:33:39) those are things to run down let's check (00:33:41) is it a training tape in the system is (00:33:43) it you know um sounding rocket that was (00:33:46) launched and it's not actually an attack (00:33:48) is the system malfunctioning In the (00:33:49) Petro case, the satellite system that (00:33:54) the Soviets deployed was detecting (00:33:57) sunlight glinting off of clouds (00:34:00) and not and and sort of registering that (00:34:02) as the flash from a nuclear launch. So (00:34:04) like that would be an interesting use (00:34:06) case. Um and sort of you would then bias (00:34:10) the system intentionally in a way to try (00:34:13) to help you identify false alarms. Um, (00:34:16) but I I think there's value in (00:34:20) retaining humans for some of these types (00:34:22) of decisions in particular because (00:34:24) humans understand the moral stakes. (00:34:28) So like there's kind of this more (00:34:30) fundamental question of what are the (00:34:32) types of things that AI is likely to be (00:34:34) good at? What are the types of things (00:34:35) that humans are likely to be good at? (00:34:37) And I think things where it makes sense (00:34:39) to use AI would be situations where we (00:34:40) have we either have or can create good (00:34:43) data on what performance looks like and (00:34:46) we have clear metrics of better (00:34:48) performance. Self-driving cars is a (00:34:49) great example of this. It's a hard hard (00:34:51) problem but we can correct a ton of data (00:34:54) and then we can run those we can create (00:34:56) synthetic data by running simulations to (00:34:58) to amplify that data that we get and we (00:35:00) can test the cars in the real world (00:35:02) operating environment and there's a (00:35:04) clear metric of performance and don't (00:35:05) get into accidents. There are other (00:35:07) situations where it's just not as clear (00:35:11) um or we don't have good data or what (00:35:14) depends on the right answer might often (00:35:16) depend on context and judgment. So, to (00:35:19) give a military example, um, let's say (00:35:22) that we have a drone looking at a person (00:35:24) and they're standing in a dark alleyway (00:35:26) and they're holding some object in their (00:35:27) hands. What is the thing? Well, that's a (00:35:30) thing where AI could be really helpful. (00:35:31) Are they holding a rifle or holding a (00:35:33) rake in their hands? What are they (00:35:34) doing? We could build up a database of (00:35:37) images under different lighting (00:35:39) conditions and angles and probably get (00:35:41) to the place where AI is better than (00:35:43) humans. Now, let's say we identify the (00:35:45) object holding a rifle. Is that person (00:35:48) an enemy combatant? (00:35:50) That's like a super hard problem because (00:35:52) it depends on context and judgment. (00:35:54) Like, well, you know, what is enemy (00:35:57) behavior in this area and civilians? (00:35:58) Maybe civilians carry rifles in this (00:36:00) area for protection against thieves and (00:36:02) bandits. Maybe that's super normal. Um, (00:36:04) maybe the person carrying a shovel, but (00:36:07) we just saw them planting an IED. (00:36:08) They're an enemy combatant. Like, those (00:36:10) are that's harder. And you could imagine (00:36:14) maybe (00:36:15) reasoning systems that are plugged in (00:36:17) that are synthesizing all this (00:36:18) information and coming to judgments. Um (00:36:21) but I would be cautious in those um for (00:36:26) all the flaws that you have in in LLMs. (00:36:29) Those are use cases that I think I would (00:36:32) just be very cautious about. (00:36:34) >> Okay, setting that down for now. Um I'm (00:36:37) interested in how this all affects um (00:36:39) kind of nuclear stability and (00:36:41) deterrence. Um (00:36:44) so I guess we already do have some (00:36:45) automation um but we'll probably be (00:36:48) building in more um in in different (00:36:51) parts of the kind of nuclear command (00:36:53) control and communications um system. (00:36:57) How do you do you think there will be (00:36:59) kind of one effect uh on kind of the (00:37:02) game theory of of deterrence? Um or do (00:37:05) you think it's just like they're going (00:37:07) to be a million and it depends on (00:37:08) exactly how countries incorporate these? (00:37:12) Um I guess another option is like (00:37:14) countries don't know how their (00:37:16) adversaries are incorporating these and (00:37:18) they'll be making guesses. Uh what does (00:37:21) all of this do to like (00:37:25) nuclear stability? (00:37:27) I think there's a couple use cases that (00:37:29) you could see that seem likely that (00:37:31) might increase stability. One would be (00:37:34) using at a just a better visibility on (00:37:36) what other countries are doing and so it (00:37:40) makes it harder to launch some kind of (00:37:42) surprise attack. Or if you get a signal (00:37:45) of a launch, you have a lot of other (00:37:47) information that you can use to verify (00:37:49) it that can allow you to have sort of (00:37:52) multiple different looks at this (00:37:54) problem. Um, that's one use case that (00:37:56) would be valuable. Another one is if (00:37:59) militaries just get better at using (00:38:01) automation to make their operations and (00:38:03) responses more reliable. And that has a (00:38:06) stabilizing effect because maybe other (00:38:08) adversaries are less willing to (00:38:11) contemplate doing some kind of attack, (00:38:13) right, of like, well, you know, we we (00:38:15) really know that they've shortened the (00:38:16) response time and they're able to get (00:38:18) better information and, you know, we're (00:38:20) not going to be able to disarm them (00:38:21) through a first strike. Here's a couple (00:38:23) use cases that might be concerning and I (00:38:26) don't really know what the net effect of (00:38:27) this kind of would net out to be. One is (00:38:30) this sort of fear that um AI enables so (00:38:36) much transparency (00:38:38) coupled with precision guided weapons (00:38:40) that it might enable a first strike to (00:38:43) disarm opponents. Um, I think this is (00:38:46) like super unlikely because for this to (00:38:49) work, you have to basically get all of (00:38:52) the weapons or almost all of them and (00:38:54) then have adequate missile defenses to (00:38:55) soak up like one or two that get (00:38:57) through. I think it's extremely hard to (00:38:59) pull that off in practice. It's not (00:39:01) enough to say, well, we know these are (00:39:02) their launching sites. You need to know (00:39:05) where are the mobile missiles in real (00:39:07) time like for all of those systems. (00:39:12) >> Right. (00:39:12) >> Right. and and (00:39:13) >> submarines and trucks with nuclear (00:39:16) weapons on the backs of them driving (00:39:17) around countries. (00:39:18) >> Yeah, (00:39:19) >> I think it's just Yeah, it's like a (00:39:20) really high bar to achieve and to get to (00:39:23) a confidence level that somebody would (00:39:24) feel comfortable executing for a strike. (00:39:27) Now, the more realistic worrying (00:39:28) possibility is that even though an (00:39:31) attacker might feel like, yeah, this is (00:39:32) not plausible, a defender might feel (00:39:34) vulnerable (00:39:35) >> as a result. (00:39:36) >> Yep. (00:39:37) >> And then they say, well, we got to build (00:39:39) more weapons. We got to build more more (00:39:41) silos. We've seen China engage in this (00:39:42) massive um buildup of their nuclear (00:39:45) weapons and I do think part of that is (00:39:48) likely China reacting to US (00:39:51) technological advances uh drones, (00:39:54) satellites, you know, other systems that (00:39:57) that make China feel like they're (00:39:59) vulnerable and they need to increase (00:40:01) their nuclear stockpile. Um and so that (00:40:05) that actually could be destabilizing (00:40:07) either because it um in an immediate (00:40:10) situation it it creates sort of this (00:40:12) incentive to maybe launch because you (00:40:16) feel like your weapons are super (00:40:18) vulnerable or on a longer timeline it (00:40:21) creates sort of an arms race instability (00:40:23) dynamic where countries feel like we got (00:40:25) to build more of these weapons and then (00:40:28) others say we got to build more too and (00:40:30) particularly as we're moving into a (00:40:32) >> there's sort of this nasty combination (00:40:33) right now in the nuclear space of we're (00:40:36) moving towards one a tripolar nuclear (00:40:38) world where the balance really isn't (00:40:40) just between the US and Russia, it's the (00:40:42) US, Russia and China. China builds up (00:40:44) their stockpile (00:40:45) >> and so like US analysts are looking at (00:40:46) this and like now we need to have enough (00:40:48) weapons to to counter not just Russia (00:40:49) but Russia and China and then so the US (00:40:52) builds increases its arsenal that others (00:40:54) are going to increase theirs and sort of (00:40:55) you could you could get this kind of (00:40:57) more complicated arms race dynamics (00:40:59) coupled with emerging technologies that (00:41:02) maybe create vulnerabilities or (00:41:03) uncertainty whether that's AI space (00:41:06) systems cyber systems that maybe create (00:41:10) uncertainty among political leaders (00:41:12) about how safe their arsenal is and then (00:41:14) coupled with sort of really I think (00:41:16) disturbingly increased political (00:41:18) salience of nuclear weapons that Putin (00:41:21) in particular has done a lot of nuclear (00:41:23) saber rattling in the last couple years (00:41:25) in the context of Ukraine. Um, and I (00:41:28) think the US has done a decent job of (00:41:30) responding in a measured way to and not (00:41:32) overreacting to, but Putin is sort of (00:41:34) putting nuclear weapons on the table in (00:41:37) a sort of diplomatic sense of sort of (00:41:40) waving this nuclear stick around, (00:41:42) >> right, (00:41:42) >> in a way that is is new and different (00:41:44) and that might change how policy makers (00:41:46) think about the relevance of these (00:41:48) weapons and maybe even make them feel (00:41:50) more usable to some countries in some (00:41:52) types of conflicts. And I think that (00:41:53) that combination is a little bit scary. (00:41:56) >> Yep. Yep. Yeah, that makes sense. Um I (00:41:59) guess if you could um kind of choose one (00:42:05) internationally agreed upon kind of (00:42:07) binding constraint on automation and AI (00:42:12) in the context of nuclear command, (00:42:13) control and communications tomorrow, say (00:42:17) um yeah. Yeah. What would that be? What (00:42:19) would what would make you feel safest? (00:42:22) I would love to see nuclear powers agree (00:42:25) to maintaining human control over (00:42:27) decisions relating to using nuclear (00:42:29) weapons. (00:42:30) >> That seems like a really low bar to (00:42:32) clear, right? Like we all ought to be (00:42:34) able to agree that, you know, regardless (00:42:37) of what we do with autonomous weapons or (00:42:38) other things that humans should maintain (00:42:40) control over decisions relating to (00:42:42) nuclear weapons and the US, that's (00:42:45) actually official defense department (00:42:46) policy that the US put out several years (00:42:48) ago in the nuclear posture review. it (00:42:51) was agreed to between Biden and she (00:42:55) between the US and China um during (00:42:57) during talks in the Biden (00:42:58) administration. Uh France and the United (00:43:01) Kingdom have had similar sort of public (00:43:03) statements. So there's sort of some (00:43:05) really interesting important foundations (00:43:07) there particularly I think the USChina (00:43:08) agreement is really significant. I'd (00:43:11) love to see that expanded to include (00:43:12) other nuclear powers ideally to get a (00:43:15) statement from the the P5. Uh so that (00:43:18) would be the existing countries plus (00:43:19) Russia. Um and to to maybe deepen that a (00:43:24) little bit to have a little more clarity (00:43:27) like what does that mean internally? (00:43:31) Okay. Like how do we implement this kind (00:43:32) of guidance? What's sort of inbounds and (00:43:34) what's out? And are there things that (00:43:38) countries could do that would be (00:43:41) credible asurances between other (00:43:44) nations? like what could we do that (00:43:46) might credibly cuz statements are one (00:43:48) thing like what are you actually doing (00:43:50) um that that might assure others that (00:43:53) we're actually following through on that (00:43:54) and I'm not totally sure what that looks (00:43:56) like but I think that that would be a (00:43:58) really important direction to go and (00:44:02) there's already a sort of early (00:44:03) foundation there to build upon. (00:44:05) >> Yeah. Yeah. I mean, my first thought (00:44:08) hearing this is (00:44:11) verifiable commitments seem so so (00:44:13) important and how do you have verifiable (00:44:17) commitments to keeping humans aloop in (00:44:20) the loop in these decisions? Are there (00:44:23) proposals for technical solutions to (00:44:25) this? Um, or is this just an open (00:44:29) problem that people should think more (00:44:30) about? I think it's hard to get to a (00:44:33) place where there's clear verifiable (00:44:34) commitment because I think it's (00:44:37) impossible, but I don't I don't have an (00:44:38) answer here, I guess, because um you (00:44:41) know the the answer is embedded in your (00:44:43) software and your operations in a way (00:44:45) that's very different than like counting (00:44:47) missile silos. We can see the missile (00:44:49) silos. We can count them. We know how (00:44:50) many you have or submarines. Like hard (00:44:52) to hide. You can hide a submarine (00:44:53) underwater where it is. You don't hide (00:44:56) the existence of a submarine. Like it (00:44:58) has to come to port. Um, and so that (00:45:01) seems really tricky and to allow the (00:45:04) like countries are not going to allow (00:45:06) other nations into their nuclear (00:45:10) operations. There's just like way too (00:45:11) much vulnerability there. And it's a (00:45:13) consistent problem at arms control is (00:45:14) how do you sort of manage this (00:45:17) verifiability kind of vulnerability (00:45:19) paradox of like how do I show you what (00:45:22) I'm doing in a way that verifies that (00:45:26) you could see what I'm doing but doesn't (00:45:28) make me vulnerable because I'm giving (00:45:29) away too much information. Um I think (00:45:32) it's also worth taking a step back and (00:45:34) saying like where do we need (00:45:35) verifiability (00:45:36) in arms control agreements? Some arms (00:45:40) control agreements have um like (00:45:42) inspection procedures. Many of them do (00:45:45) not because there's just the ability to (00:45:47) externally observe what others are (00:45:49) doing. Again, I think in this case (00:45:51) that's going to be really hard. (00:45:53) Another (00:45:55) another sort of way to look at this is (00:45:57) you don't necessarily it'd be nice to (00:45:59) have some credible assurances. You don't (00:46:00) necessarily need that if you're going to (00:46:02) do it anyways. So for the United States (00:46:04) for example, if this is defense (00:46:06) department policy because we think it's (00:46:07) a good idea, (00:46:09) then encouraging other countries to (00:46:11) adopt that same policy is in US (00:46:14) interests regardless of exactly what (00:46:17) they're doing because we're going to do (00:46:19) it anyways. It's not like oh like (00:46:21) another country decided to automate (00:46:23) nuclear launch so it seems like a good (00:46:24) idea. Um it's if we think it's unwise (00:46:27) we're not going to do it anyways. And so (00:46:30) um you know you do have some dynamic (00:46:32) there are some examples of other weapons (00:46:34) like that like biological weapons where (00:46:38) the US has signed a biological weapons (00:46:40) uh treaty and has for sworn them and (00:46:43) like sort of even if other countries (00:46:45) have covert programs and it's probably (00:46:47) true that some other countries might on (00:46:48) a small scale it's not something still (00:46:51) the US wouldn't want to do because the (00:46:53) risk of blowback is simply too high. (00:46:56) >> Yeah. Okay. That is that is slightly (00:46:58) reassuring. Um I guess on your kind of (00:47:02) best guess of how things go over the (00:47:04) next few decades, can you kind of paint (00:47:08) a picture of what you think wars will (00:47:11) look like and which parts might be (00:47:14) really fast and then where humans will (00:47:17) uh kind of stay in the loop and maybe (00:47:19) slow things down. (00:47:21) >> Sure. So I think over the next say 5 to (00:47:23) 10 years we'll see militaries (00:47:24) increasingly um build more drones of (00:47:29) different types them in larger numbers (00:47:31) they'll have the incremental more (00:47:33) autonomy we might see some isolated (00:47:35) examples I think it's likely of (00:47:37) autonomous weapons being deployed on the (00:47:39) battlefield but I don't think they'll be (00:47:40) integrated into military operations in (00:47:42) large numbers I think we'll certainly (00:47:43) see more mature AI technology like image (00:47:46) classifiers um used in a in a wide (00:47:49) variety of context by militaries (00:47:51) to understand the battlefield. I think (00:47:53) you'll see a lot of what you might (00:47:54) really just call business process (00:47:56) automation of people sort of taking (00:47:58) things that humans are now doing in (00:48:00) Excel spreadsheets or passing along (00:48:03) communications and information more (00:48:05) manually gets automated um and that (00:48:08) speeds up sort of the the tempo of war (00:48:10) in terms of people's ability to sort of (00:48:12) compress their decision-making time. So, (00:48:14) for example, one of the things that the (00:48:15) US Army is quite keen on is if they get (00:48:18) some intelligence, let's say satellite (00:48:20) imagery of where an enemy target is, um, (00:48:24) they want to be able to sort of shorten (00:48:26) the time it takes to put sort of useful (00:48:30) targeting information in the hands of (00:48:31) artillery systems that are on the ground (00:48:34) so they can actually carry out a strike. (00:48:36) And right now, that's measured in hours. (00:48:38) And you might see that compressed to (00:48:41) minutes as that gets compressed over (00:48:43) time. Um, but I think you'll still see (00:48:45) humans in still engaged in a lot of (00:48:48) operations. Um, over a longer timeline, (00:48:51) maybe 15 to 20 years, we might see some (00:48:54) integration of swarms at small scale for (00:48:57) sort of tactical purposes, maybe (00:48:59) swarming robots being used to do (00:49:01) reconnaissance over an area or strike (00:49:04) targets that humans have approved. Um, (00:49:07) we might see autonomous weapons uh (00:49:09) become more widely integrated into (00:49:11) military operations, (00:49:13) more AI being used to generate courses (00:49:16) of action um to support decision-m I (00:49:20) think we'll probably see over that time (00:49:22) frame (00:49:23) more AI involved in intelligence (00:49:25) processing and so that so that people (00:49:27) aren't just looking at an image of an (00:49:30) object that's been found by you know an (00:49:32) AI system for example but actually the (00:49:35) intelligence reports are synthe (00:49:36) synthesized and analyzed in ways um by (00:49:40) AI before they're given to commanders. (00:49:42) And so commanders are they're still (00:49:44) humans making decisions, but they're (00:49:45) increasingly relying on information (00:49:48) that's mediated by artificial (00:49:49) intelligence, which introduces a lot of (00:49:51) weird vulnerabilities of like are there (00:49:53) biases in the system and has the enemy (00:49:55) found ways to manipulate it? Um, but I (00:49:58) think you'll see the value in that over (00:50:01) time and then maybe over a time span of (00:50:03) 30 or 40 years something approaching (00:50:06) more like some of these discussions (00:50:08) about battlefield singularity or hyper (00:50:10) war where militaries have really fully (00:50:13) integrated AI and we see at a much (00:50:17) larger scale operations being condone in (00:50:20) ways that that start to maybe exceed (00:50:22) humans abilities to stay in the loop in (00:50:25) terms of actually managing tact tactics (00:50:27) on the ground. I would say the exception (00:50:29) to everything I outline is probably (00:50:31) what's happening in cyerspace (00:50:35) >> where I think automation will happen (00:50:36) much much much faster and we'll see (00:50:41) cyber attackers and defenders sort of be (00:50:43) forced into a position where they have (00:50:45) to seed control to machines on a much (00:50:48) faster time scale because of just the um (00:50:51) both the ability of machines to operate (00:50:53) in you know what is sort of their native (00:50:54) environment right with the machines (00:50:56) they're just much more capable (00:50:58) um on the internet than they are out of (00:50:59) the out of the real world at least at (00:51:01) the moment. And just that the tempo of (00:51:04) operations in cyerspace being one that's (00:51:07) closer to things like financial trading (00:51:09) where you sort of have to the (00:51:10) competitive pressures to pull humans out (00:51:12) of the loop are going to happen just (00:51:13) much sooner. (00:51:14) >> Yep. Yeah. Okay. I I want to come back (00:51:17) to cyber war um for sure. Before we do (00:51:20) that, um, so you've got these kind of (00:51:24) you've got a couple of different (00:51:25) timelines for (00:51:27) different kind of scales of increases in (00:51:29) the tempo of war. Um, so five, uh, it (00:51:32) sounds like, yeah, next next fiveish (00:51:34) years is is maybe a is maybe a slight (00:51:37) increase in tempo. Um, then you get a (00:51:39) bigger one and then at some point maybe (00:51:41) um, 40 or 50 years from now, uh, maybe (00:51:44) you do start seeing something like hyper (00:51:46) war. Um, really concretely, how fast uh (00:51:50) are we talking about? So, in in a hyper (00:51:53) war scenario, (00:51:56) are wars fought and won in the span of (00:52:00) hours or days or weeks or months? (00:52:03) >> Yeah, that's a great question. So, there (00:52:05) are still very real physical constraints (00:52:08) on moving physical objects around that (00:52:12) still will apply to robots and to AI. So (00:52:15) again, cyerspace being an exception (00:52:17) where you could have um very very fast (00:52:20) operations. I mean there have been um (00:52:22) you know botn nets for example that have (00:52:23) spread like very very quickly um cyber (00:52:26) attacks that have taken down um you know (00:52:29) infrastructure for example cyber attacks (00:52:31) from Russia against Ukraine that have (00:52:33) taken down Ukrainian um digital (00:52:35) infrastructure in some case like a (00:52:37) matter of seconds. So, so you could have (00:52:39) si conflict unfold in the cyerspace in a (00:52:41) matter of minutes. But in you know the (00:52:43) the sort of the physical world it takes (00:52:46) time for missiles to fly long distance. (00:52:50) It takes time for aircraft to fly. Like (00:52:52) AI is not going to fundamentally change (00:52:55) those physical constraints. Now maybe in (00:52:57) the long run like people say well AI is (00:52:58) going to label better materials and blah (00:53:00) blah blah blah but like okay maybe (00:53:03) incrementally still. Um but I think you (00:53:06) could look at like the the pace of (00:53:08) advancement of um you know aircraft (00:53:12) propulsion it's not growing (00:53:14) exponentially right it's very (00:53:16) incremental improvement um and so that I (00:53:20) think will will mean you know for things (00:53:22) like um a a missile salvo (00:53:28) would unfold over a period of hours it (00:53:32) might be that the scale of that missile (00:53:35) salvo is much larger. (00:53:37) >> Um now depending on the range it might (00:53:39) be that you know it's 20 minutes for (00:53:40) missiles to come in and to attack. Um (00:53:43) but you know a back and forth of a (00:53:45) missile change might take several hours (00:53:46) and maybe within 10 hours like it's all (00:53:48) done and and and sort of the dust (00:53:51) settles and one side has a dominant (00:53:53) advantage in terms of at least that (00:53:54) initial missile salvo. What would be (00:53:56) different would be maybe the number of (00:53:58) missiles and drones that are coming in (00:54:00) much larger than say the raids that (00:54:02) we're seeing uh Russia launch against (00:54:05) Ukraine right now for example um to (00:54:07) maybe thousands of of drones and (00:54:10) missiles all at once and they might be (00:54:12) much more cooperative and they're (00:54:13) they're dynamic and responding in ways (00:54:15) that force defenders to basically (00:54:17) automate their defenses. M um but you (00:54:20) know something like a like a ground (00:54:21) invasion (00:54:23) we've you know would still take days and (00:54:26) at minimum for a lightning invasion a (00:54:29) couple weeks to unfold. Um so I think (00:54:32) that like that's just there are some of (00:54:34) these physical constraints that are very (00:54:37) real that would still exist even in the (00:54:39) world of AIdriven warfare. (00:54:41) >> Yep. Um yeah that that makes sense. is (00:54:45) the can you explain why (00:54:49) one of the changes there that you expect (00:54:52) is is kind of bigger simultaneous (00:54:56) attacks. (00:54:57) >> So there are incentives already, right? (00:54:59) So in an initial attack to have a (00:55:01) massive salvo to sort of hit as many (00:55:04) targets as you can at once. If you look (00:55:06) at for example like the um you know US (00:55:09) shock and awe aerial campaign against um (00:55:13) Saddam in 2003 (00:55:15) um when the US military sort of games (00:55:18) out a potential warf fight against Thai (00:55:20) uh China over Taiwan. One of the things (00:55:22) that the US expects was that China would (00:55:25) launch massive mil missile sables (00:55:27) against US air bases in the region to (00:55:30) crater runways to blow up fuel depot to (00:55:33) target aircraft. And if you can sort of (00:55:35) make these initial attacks severe (00:55:39) enough, quick enough, you can really (00:55:40) degrade the other side's ability to (00:55:43) respond, right? If you can, you know, (00:55:44) cradle the runways before the aircraft (00:55:46) got off the ground, like great. That's a (00:55:48) big win. So, um, I think what's (00:55:50) different about AI autonomy is a whole (00:55:52) bunch of factors that favor scale. (00:55:55) >> So, let's talk about the way drones are (00:55:57) used in Ukraine, for example. So let's (00:55:59) say you got about 10 order of magnitude (00:56:01) about 10,000 drones operating right now (00:56:03) on the front lines. Well, you need (00:56:05) 10,000 people to fly those drones. You (00:56:07) need a lot of operators. Now, there are (00:56:10) advantages to having the human not in (00:56:12) the drone. You can make the drone much (00:56:13) smaller. You can make it cheaper. You (00:56:15) can make it disposable. Um if you lose (00:56:17) the drone, you don't lose the pilot. (00:56:19) Pilots can gain a lot of experience over (00:56:21) time. So pilots who maybe are not very (00:56:23) good at first and they might have died (00:56:24) in their first mission in a crude (00:56:26) aircraft, (00:56:27) >> they can wreck the first 20 drones or (00:56:30) more before they really sort of figure (00:56:32) it out and like it's fine. It's like (00:56:34) really cheap. So there there are (00:56:35) advantages there. Um but you're still (00:56:37) limited in this sort of onetoone ratio (00:56:40) between pilots and drones. Now autonomy (00:56:43) totally breaks up the dynamic. Now you (00:56:45) could have one person launch a swarm of (00:56:48) a 100 drones and the person just says go (00:56:51) here and perform this operation and the (00:56:53) drones do that autonomously. (00:56:55) Um that has major advantages allowing (00:56:59) militaries to just exercise command and (00:57:01) control over larger numbers of forces (00:57:04) and therefore to start fielding them in (00:57:07) ways that are more incentivized. Um and (00:57:11) so you know if you (00:57:14) if you if you have the ability to now (00:57:16) basically control an unlimited number of (00:57:20) drones now that sort of changes your (00:57:22) paradigm of like okay well maybe we (00:57:23) could just like let's crank out a lot of (00:57:25) these things. Um there's also other ways (00:57:27) in which like as over time (00:57:30) so let's talk about marginal costs for (00:57:32) drones. I go like really really dirty (00:57:34) for a second. Okay. So, um, as you feel (00:57:38) larger numbers, obviously you get better (00:57:40) economies of scale, but there's still (00:57:42) some marginal cost in producing that (00:57:43) additional drone. Drones are pretty (00:57:45) cheap, but you've got to make the (00:57:46) physical hardware. Now, for the (00:57:48) software, it's different, right? The (00:57:51) software scales differently than (00:57:53) hardware. So, (00:57:54) >> it might cost a lot to develop software, (00:57:57) but once you do, it's basically costless (00:57:59) to replicate software. And so this is (00:58:01) why you see like totally different (00:58:03) economics in um handheld devices and (00:58:05) smartphones for example. Like a phone (00:58:08) costs a good chunk of change, several (00:58:09) hundred even though there's huge (00:58:12) economies of scale. There's like 7 (00:58:13) billion smartphones. Um but the software (00:58:16) is totally different once I've got the (00:58:18) the hardware. I could download apps for (00:58:20) free. Um and anybody can download an (00:58:23) app, right? Once you have that kind of (00:58:24) hardware in place. So similarly you (00:58:26) could see that as more of the cognitive (00:58:30) abilities of the drone gets from the (00:58:32) human which humans don't scale right (00:58:34) I've got to train pilots their scarce (00:58:36) resources um to to embedded in the (00:58:40) software itself that software scales (00:58:42) very easily and the sort of all of those (00:58:46) economies favor scale but there are (00:58:49) still again I just want to say like very (00:58:50) physical real physical constraints in (00:58:53) production like who's going to make all (00:58:55) these things in um the ability to (00:58:58) transport it to the front and get it (00:59:00) there in the logistics of doing (00:59:03) maintenance for example. Um but I think (00:59:06) the the dynamics will benefit scale in (00:59:09) pretty dramatic ways. (00:59:11) >> Yep. Um yeah, that that's fascinating. (00:59:14) um does how so (00:59:18) automated weapons are likely cheaper um (00:59:23) for a bunch of the reasons you've just (00:59:25) said. Um it sounds like the scale goes (00:59:27) up. So maybe that's um kind of the cost (00:59:30) of an individual attack goes up a bit in (00:59:33) that sense. Um but how kind of on net uh (00:59:36) does the cost of war change uh as we as (00:59:39) we get more automated weapons? And I (00:59:40) guess yeah, I'm interested in both kind (00:59:42) of financial costs um but also in human (00:59:45) casualties. Um intuitively it seems like (00:59:47) you'd get uh way fewer human casualties. (00:59:50) Um but curious if that's right. (00:59:52) >> I do think that over time AI and (00:59:55) automation will allow militaries to sort (00:59:57) of do more with less in terms of (00:59:59) personnel. And certainly for some (01:00:01) militaries like the US military (01:00:04) personnel costs are very very high. (01:00:06) >> Um that's not true for the Russian (01:00:08) military for example, right? a totally (01:00:10) different personnel model for how (01:00:12) they're how they're thinking about (01:00:13) people. Um but so that could allow (01:00:16) militaries to do more with less. The (01:00:18) sort of question of overall cost of (01:00:19) militaries maybe depends a lot on like (01:00:22) what is your your mental model for what (01:00:25) the price point is set for defense (01:00:27) spending and to what extent it's driven (01:00:28) by um defense needs versus other factors (01:00:33) exogenous to the defense department and (01:00:35) that ecosystem. how much Congress is (01:00:38) willing to spend domestic factors like (01:00:40) political issues. Um (01:00:42) I do think in terms of like (01:00:46) these changes are likely to relatively (01:00:49) benefit less capable actors and that is (01:00:52) a meaningful change in thinking about (01:00:54) cost. So for right now like prior to (01:00:56) drones if you wanted air power you (01:00:59) wouldn't have to buy an airplane. (01:01:01) Airplanes are traditionally very very (01:01:03) expensive and certainly like a fighter (01:01:05) jet um you know costs you know maybe 50 (01:01:08) to 100 million dollars um order of (01:01:11) magnitude. Drones are super cheap. You (01:01:13) can buy you know small quadcopters for (01:01:16) you know maybe few hundred thousand (01:01:17) bucks and um it doesn't do the same (01:01:21) thing but it does give you air power. It (01:01:22) does give you the ability to recon (01:01:24) targets from the air find them to even (01:01:26) carry out like very small strikes. Um (01:01:28) and so sort of that sort of lowers the (01:01:31) the price of entry into air power and (01:01:35) because in particular AI seems to (01:01:37) proliferate very very rapidly and (01:01:39) software proliferates easily. I think (01:01:41) that this sort of relatively benefits (01:01:44) small-cale actors (01:01:46) um who who are advantaged by this. Now, (01:01:48) in terms of the human cost, I guess (01:01:50) there is this idea maybe of okay, well, (01:01:53) we have all these drones and then like (01:01:54) people aren't fighting and that's great. (01:01:58) I'm I'm going to start with a I don't (01:01:59) believe it's going to be true. (01:02:01) The cautionary tale to me would be the (01:02:03) invention of the Gatling gun, which the (01:02:05) inventor saw the horrific bloodshed (01:02:08) coming from the American Civil War and (01:02:10) thought, "Okay, what if I could have a (01:02:13) machine that could automate firing on (01:02:16) the front lines, which the Gatling gun (01:02:18) did, it was sort of a forerunner of the (01:02:20) machine gun, and automated the process (01:02:21) of firing, dramatically sped up um (01:02:24) firing rates on the battlefield." and (01:02:26) his vision was then there'd be fewer (01:02:28) soldiers on the battlefield, save lives. (01:02:31) The opposite happened. It dramatically (01:02:34) expanded the lethality of warfare and we (01:02:37) saw huge casualties in World War I, (01:02:41) trench warfare, when once machine guns (01:02:43) had matured and fully implemented into (01:02:45) warfare. And so, you know, just because (01:02:50) I I'm not sure that automation (01:02:51) necessarily (01:02:53) um like is going to pull people back in (01:02:55) part like I just don't buy this vision (01:02:57) of futures of robot armies fighting in (01:03:01) these bloodless battles. (01:03:02) >> I think humans will still be needed to (01:03:05) perform some cognitive tasks and some of (01:03:06) those are likely to be actually close on (01:03:08) the front lines because of challenges in (01:03:11) long range protected communications. (01:03:13) it's going to be easier to have short (01:03:14) range communications to control robots (01:03:16) from relatively nearby, (01:03:18) >> right? (01:03:19) >> Um but also I just think sort of the the (01:03:22) ugly reality is likely to be the (01:03:24) politically (01:03:26) unfortunately (01:03:28) people will have to suffer and die for (01:03:30) wars to end. I think that's the (01:03:32) practical reality and that's that's (01:03:36) tragic and that's kind of dark but I (01:03:38) think that's probably likely to be the (01:03:39) case. (01:03:40) >> Yeah. I think when I (01:03:43) imagine (01:03:46) kind of these these robot wars fighting, (01:03:48) I'm like, but then you're not losing (01:03:50) anything of value or like enough of (01:03:52) value. (01:03:52) >> Yeah, I think that's right. And that's (01:03:54) unfortunately the reality of a lot of (01:03:56) wars. Like if you look at um the war (01:03:58) between Russia and Ukraine, for example, (01:04:01) >> the front lines are relatively static. (01:04:03) They're not moving in a dramatic way. (01:04:05) The war has devolved into this war of (01:04:07) attrition where some of it's about the (01:04:10) economics of fielding artillery for (01:04:13) example and causing casualties but a lot (01:04:16) of it is simply a war of suffering of (01:04:19) who is willing to incur more costs for (01:04:23) longer. Who wants this more? And a lot (01:04:25) of wars unfortunately turn out that way. (01:04:28) Like it would be nice if militaries (01:04:30) could go off and fight a battle that (01:04:33) doesn't involve bloodshed or they could (01:04:35) just game all this out on their (01:04:37) computers that's clearly going to win. (01:04:39) And when there are dramatic changes that (01:04:41) does happen. We see politically that (01:04:44) when there are huge differences in (01:04:46) power, countries generally will exceed (01:04:49) to then what sort of you know stronger (01:04:51) nations want. um but not always and (01:04:54) sometimes small countries fight fiercely (01:04:58) and and hard for their independence and (01:05:00) that that will to fight. So like the (01:05:04) Russia's invasion of Ukraine is like a (01:05:06) really interesting example here. So part (01:05:09) of what happens is you can think of wars (01:05:14) as a failure of negotiations, a peaceful (01:05:17) negotiations that moves into sort of (01:05:19) negotiation through violence. And um one (01:05:22) of the challenges here is that on paper (01:05:24) people can sort of add up military (01:05:26) hardware, but we what's really hard to (01:05:28) measure is things like will to fight and (01:05:32) morale. (01:05:33) >> And um on paper, Russia should have (01:05:36) ended that war in 30 days. And a lot of (01:05:38) lot of military analysts, myself (01:05:40) included, like thought that's what was (01:05:42) going to happen. And what we saw was (01:05:45) that Ukraine is superior in all of these (01:05:48) intangible dimensions of war of morale, (01:05:54) will to fight, leadership, unit (01:05:56) cohesion, (01:05:57) um corruption in the ranks, like (01:06:01) that. That has huge effects on the (01:06:04) battlefield. always been the case that (01:06:06) these human factors matter a lot. (01:06:10) >> Napoleon talked about them counting uh 3 (01:06:13) to one I believe he said against (01:06:15) material factors in war. But these (01:06:17) things are these immaterial factors (01:06:18) human factors are hard to measure. Now (01:06:20) the interesting question is how does AI (01:06:22) change that? (01:06:23) >> Um Kenneth Payne has written some (01:06:26) phenomenal phenomenal work on this. He (01:06:29) wrote a book I warbot and some other (01:06:31) work sort of thinking through how AI (01:06:32) changes the psychology of war. So what (01:06:36) does will to fight mean when you have (01:06:39) drone swarms fighting that like they (01:06:43) never get tired? They never right. And (01:06:46) so so one of the arguments that he makes (01:06:48) for is an illustrative example here and (01:06:50) I'm not saying that I buy this but is an (01:06:52) interesting argument is often times this (01:06:55) will to fight has benefited defenders. (01:06:58) They're fighting for their homeland. um (01:07:01) aggressors maybe just they don't wipe (01:07:02) there as much very clearly in in Ukraine (01:07:05) in terms of the balance among Russian (01:07:07) and and Ukrainian troops when you have (01:07:10) AI fighting if you take that off the (01:07:14) table (01:07:15) that would seem like maybe that (01:07:18) relatively then takes away some of the (01:07:20) advantages of defenders sort of it's (01:07:22) more equal and that maybe in relative (01:07:24) terms then benefits attackers more I (01:07:26) don't know if that's valid but AI raises (01:07:28) some like just really interesting (01:07:29) questions about changes is in the (01:07:31) psychology war. (01:07:32) >> I want to go back to something you said (01:07:34) a few minutes ago. Um it sounds like we (01:07:39) this might um cause us to enter a world (01:07:42) where kind of smaller, poorer states um (01:07:45) or even non-state actors um can actually (01:07:47) threaten much larger militaries um by (01:07:50) leveraging kind of these cheap automated (01:07:52) weapon systems that are offense (01:07:53) dominant. Um how much is that going to (01:07:56) change kind of balance of power (01:07:58) dynamics? um is are are there going to (01:08:01) be more wars fought because it's cheaper (01:08:03) to start them um including by uh small (01:08:06) groups that don't have as many (01:08:08) resources? (01:08:10) >> Well, that's a good question. I think (01:08:13) like the I think the economics of it I (01:08:16) feel are valid that it relatively (01:08:18) benefits smaller groups more. I'll give (01:08:21) another example here. Um, Ukraine (01:08:23) basically neutralized Russia's Black Sea (01:08:25) fleet, sinking um several sinking and (01:08:28) damaging several warships worth um, you (01:08:31) know, hundreds of millions of dollars by (01:08:34) spending a few tens of millions of (01:08:36) dollars in small drone boats laden with (01:08:38) explosives (01:08:39) >> that could come in and sink a warship. (01:08:41) >> Um, and so I think we're going to see (01:08:43) those tactics copied more. Um, whether (01:08:45) that leads to more wars is tricky, (01:08:48) right? because it depends a lot on what (01:08:50) do you think the mechanics are that (01:08:53) drive wars um I think one mechanic can (01:08:58) be if there's a a disagreement between (01:09:02) actors about the relative balance of (01:09:04) power right and here's a place where um (01:09:08) I think you could argue AI and net kind (01:09:10) of does one or the other like it's the (01:09:12) arguments it's both ways right the (01:09:14) argument that AI might um make conflicts (01:09:17) more likely would be that um one it's (01:09:20) just a disruptive change and so there's (01:09:22) more uncertainty about how this is used (01:09:25) and who's an advantage here and some (01:09:26) countries might think we have the AI we (01:09:28) can win now in particular countries (01:09:31) might sort of feel overconfident about (01:09:34) AI because humans often sort of seem to (01:09:36) overestimate what AI can do in terms of (01:09:38) its abilities um we see this really (01:09:41) dramatically unfortunately with early um (01:09:44) implementation of like autopilot in (01:09:47) Teslas where there were a number of (01:09:48) fatal accidents, people sort of (01:09:50) overrusting the automation. Um, so that (01:09:52) could be one kind of risk. Another way (01:09:54) that AI might, you can imagine, make (01:09:56) wars more likely is that as more (01:09:58) military capability is embedded in (01:10:00) software and algorithms, it's much (01:10:02) harder to measure. Like you can measure (01:10:04) airplanes, you can measure ships, you (01:10:06) can measure tanks, and be, well, look, (01:10:07) they have three times as many aircraft (01:10:09) as we do um, and twice as many tanks, so (01:10:12) maybe we shouldn't fight a war with (01:10:13) them. But when it's like algorithms, (01:10:15) it's really hard to like, okay, how do (01:10:17) we know if our swarming algorithm is (01:10:20) better than their swarming algorithm? (01:10:22) >> You know, that's actually really tricky, (01:10:25) right? Other than like, well, we'll (01:10:26) fight them and find out. It's going to (01:10:27) be really hard. So, that might lead to (01:10:28) more uncertainty and disagreements. Um, (01:10:32) one way that AI might be more (01:10:33) stabilizing (01:10:35) is if it creates more transparency (01:10:39) and greater ability for um, countries to (01:10:43) just sort of like see what others are (01:10:45) doing and may make it harder to carry (01:10:48) out surprise attacks. And I think we've (01:10:50) actually got really solid evidence of (01:10:52) this. We saw this in the run-up to (01:10:54) Russia's invasion of Ukraine where the (01:10:56) US because of just greater intelligence (01:10:59) satellite imagery was able to have (01:11:01) really great visibility in what Russia (01:11:03) was doing and then share in a really I (01:11:05) think really impressive diplomatic move (01:11:07) share that intelligence declassify it (01:11:08) share it with European allies to get (01:11:10) Europeans on board that this was this (01:11:12) was something that was going to occur. (01:11:14) And so you could see AI just like makes (01:11:16) it really harder to mass forces for (01:11:19) surprise attack and that takes away some (01:11:21) of the incentives. We see a little bit (01:11:22) of this even tactically on the front (01:11:24) lines in Ukraine where despite the fact (01:11:26) that there's all these drones and the (01:11:28) drones are kind of hard to defend (01:11:29) against, the front lines are really (01:11:31) static for a lot of reasons. But drones (01:11:34) seem to be making the front lines more (01:11:37) static. And one of the things that we're (01:11:39) hearing from people on the front lines (01:11:40) is there's no way to mass forces for an (01:11:44) assault. They can see you because they (01:11:46) have drones overhead. they can see what (01:11:48) you're doing and so they know that (01:11:50) you're going to make an attack in this (01:11:51) area (01:11:52) >> and then they can defend against it (01:11:54) >> and so it contributes to this stasis and (01:11:57) so that (01:11:58) >> all of which is say like I don't know (01:12:00) you could see arguments on either side (01:12:02) and a lot of it depends upon how the (01:12:04) technology is implemented by countries (01:12:06) >> you've kind of described this (01:12:08) incremental (01:12:10) uh push toward more and more autonomy at (01:12:14) some point potentially leading to um (01:12:18) very very automated uh war. Um it's (01:12:21) maybe something like this hyperwar um (01:12:24) image. Um and you think that it could (01:12:27) take decades uh to get there. Why do you (01:12:30) think it'll take that long? Are the key (01:12:32) barriers more like technology or are the (01:12:35) key barriers more like deployment (01:12:37) because of people wanting humans in the (01:12:40) loop? (01:12:41) >> I think there's a a whole bunch of (01:12:43) barriers. I think the biggest one is (01:12:44) really about adoption. (01:12:46) by militaries and there's barriers to (01:12:50) adoption at like all of these stages. (01:12:52) There's there's barriers at the (01:12:54) conceptual stage. So sometimes just like (01:12:57) for militaries to conceive of a world of (01:13:00) all of these swarms of robots and drones (01:13:02) that are fully automated like that's (01:13:06) that's a big you have some you have some (01:13:07) independent thinkers that are writing (01:13:09) about these things. John Allen and Amir (01:13:11) Hussein sort of coined this term (01:13:13) hyperwar and and have written about it. (01:13:15) You have others, but it's it's going to (01:13:19) take time for that to be absorbed into (01:13:21) the bloodstream of the actual decision (01:13:22) makers inside militaries. There are (01:13:25) challenges in procurement and (01:13:26) acquisition. There's a lot of just lock (01:13:28) in into the system. It's hard for new (01:13:30) entrance to sort of fight their way (01:13:32) through bureaucratic red tape. And then (01:13:34) even when you deploy the technology, one (01:13:36) of the things that is stands out really (01:13:38) clearly in these past examples of (01:13:40) military technical revolutions (01:13:42) is figuring out how do you use this and (01:13:45) integrating it effectively into (01:13:46) operations is really hard. And so it's (01:13:49) sort of like the difference between (01:13:51) let's say a CEO is like we're going to (01:13:53) go all in on AI in this company and buys (01:13:57) you know enterprise subscriptions for (01:13:58) chat GPT or one of the MCL claw or (01:14:00) something Gemini one of the models for (01:14:02) everybody in the company okay here you (01:14:03) go have at it like that that doesn't (01:14:06) actually transform your business process (01:14:08) what transforms your business process (01:14:09) people figuring out what do I do with (01:14:10) this thing (01:14:11) >> and experimenting and everything else (01:14:13) like putting the technology in the hands (01:14:14) of people is (01:14:16) >> an important step but it's not the only (01:14:18) thing and that requires a lot of (01:14:20) experimentation, (01:14:22) requires a willingness to change your (01:14:24) way of fighting and sometimes in ways (01:14:26) that are deeply uncomfortable for (01:14:29) militaries. So (01:14:32) militaries put a lot of emphasis on (01:14:35) identity. (01:14:36) Um you know service members identify (01:14:40) with their service. (01:14:42) um you know they're soldiers or they're (01:14:45) sailors or airmen or marines. They (01:14:47) identify with their occupation often (01:14:49) times. So okay, a person's a pilot or (01:14:52) they're a a sapper that's a term for an (01:14:55) engineer in the in the army, right? Or (01:14:58) they're, you know, in the infantry, for (01:15:00) example, or they're they're in armor, (01:15:02) they're a tanker. Um and these (01:15:05) identities can be really important to (01:15:08) military. Some of these identities are (01:15:09) so important that they persist even (01:15:12) after the sort of actual occupation (01:15:15) evaporates. So for example, we call (01:15:17) people on Navy ships sailors. We no (01:15:20) longer have sails. Like they're not (01:15:21) climbing the mast and working the rings, (01:15:23) but we still call them sailors. We have (01:15:25) people in the US Army that we call (01:15:26) cavalry. They don't ride horses. They (01:15:29) don't even know people who wore horses. (01:15:30) Like that's so long ago. But that (01:15:32) identity of cavalry persists and they (01:15:34) have worse stsonins and they have boots (01:15:36) and the whole thing. So um but in some (01:15:40) cases that can hinder adoption and I (01:15:42) think one of the places we've seen this (01:15:43) most clearly is the US Air Force's (01:15:46) struggle with drones because of this (01:15:50) this sort of salience of the pilot as an (01:15:52) identity. And so when we have pilots (01:15:57) flying Air Force Reaper drones, (01:16:01) um, they're they're sitting in like a (01:16:03) cockpit on the ground wearing a flight (01:16:06) suit as though they were in a cockpit of (01:16:08) an aircraft. The army doesn't care about (01:16:11) pilots as an identity. That's not like (01:16:12) an important thing. You walk in the (01:16:13) army, you're like, "I'm a pilot." People (01:16:15) like, "All right, give me a cup of (01:16:16) coffee." They don't care, right? They (01:16:18) don't think about it that way. They have (01:16:19) enlisted personnel um flying them. they (01:16:22) were uh earlier to adopt automated (01:16:25) takeoff and landing. They um had been (01:16:28) more open to the idea of one person (01:16:30) controlling multiple aircraft. They um (01:16:35) um call their people operators instead (01:16:37) of pilots. They have more automation (01:16:39) inside their systems even though they're (01:16:41) basically using the same technology (01:16:42) built by actually the same company. Um, (01:16:44) believe it or not, what the army does (01:16:46) care about, this is wild to me, the army (01:16:49) cares about deploying overseas. (01:16:52) So, while the air force has their (01:16:55) pilots, okay, flying from bases in the (01:16:58) United States for their drones, the (01:17:00) army, at least during the wars in (01:17:02) Afghanistan, would forward deploy their (01:17:04) operators for their drones because (01:17:07) soldiers don't telecommute to war. (01:17:09) Soldiers need to be there, boots on the (01:17:10) ground. You feel like it totally doesn't (01:17:12) matter. (01:17:13) >> Yeah. fascinating. (01:17:14) >> But so these identities and so I think (01:17:16) that that that can be a hindrance to (01:17:18) adoption and sometimes a really big one. (01:17:20) >> So those it sounds like there's a lot of (01:17:23) um yeah it is these psychological (01:17:26) factors um rather than technological (01:17:28) ones or even super political ones. Um so (01:17:32) that's that's really interesting. Um I (01:17:35) want to talk through a few failure modes (01:17:38) of autonomous weapon systems. So you've (01:17:40) already kind of alluded to a few. Um, (01:17:42) one is automation bias. So, the idea (01:17:44) that humans tend to overrust machines. (01:17:47) Um, especially when the machines sound (01:17:48) confident, which I already relate to, (01:17:52) uh, given my experience with LLMs. Um, (01:17:54) it's just really hard not to be like, (01:17:57) "Yeah, this thing seems really smart. (01:17:59) It's probably telling me true things." (01:18:00) Um, another one that's come up is, um, (01:18:04) this kind of brittleleness. So these (01:18:06) systems can look superhuman in training (01:18:08) um but then kind of fail (01:18:09) catastrophically when the environment (01:18:11) changes or when someone actively tries (01:18:13) to fool them. Um there are I think loads (01:18:16) and loads of these failure modes. Um but (01:18:18) yeah to start kind of which which (01:18:21) failure mode uh worries you most? Oh, (01:18:24) all of (01:18:27) I mean um I think the the way that I (01:18:30) tend to think about this is that the (01:18:33) sort of operating parameters of these (01:18:36) systems, whether it's a rule-based (01:18:38) system or one that relies on machine (01:18:40) learning tend to be very brittle and (01:18:44) when you sort of push the system outside (01:18:46) the bounds of its operation, either it's (01:18:48) in a situation where there sort of we (01:18:49) didn't have rules for that, rules don't (01:18:51) apply or it's just not in the training (01:18:53) data. for a machine learning system, (01:18:54) they tend to fail quite badly and humans (01:18:58) don't think about them that way, right? (01:19:01) So that's not inherently a problem if (01:19:04) the people employing the systems (01:19:06) understand that limitation and they know (01:19:08) what the bounds of its operation or if (01:19:10) they know like oh it works in this (01:19:11) setting and it doesn't work in that then (01:19:13) fine we can compensate for that (01:19:15) limitation. The problem is that um (01:19:19) humans, you alluded to this with LLMs, (01:19:21) right, will often experience that an a (01:19:25) machine is good at one task and then (01:19:27) humans logically sort of transfer that (01:19:31) competence to some other very closely (01:19:34) related task. Hm. (01:19:35) >> So, uh I think you can see this in (01:19:38) examples of self-driving cars where uh (01:19:42) certainly early on people would see that (01:19:44) the car was effective and you know (01:19:47) driving and and and staying in lanes and (01:19:49) then sort of assume overtrust the (01:19:51) automation and assume, okay, well, it's (01:19:54) just a safe driver. If a human drove (01:19:55) that way, we'd say they're a safe (01:19:56) driver. And then the car would (01:19:58) catastrophically fail by driving into (01:20:00) concrete barriers or parked cars or (01:20:03) firet trucks or or other obstacles um (01:20:06) resulting in in fatal incidents. Um so I (01:20:09) think that's sort of my concern and I (01:20:11) think that there's lots of reasons to (01:20:12) think that that we would see that happen (01:20:14) in warfare because the environment is (01:20:17) uncontrolled. We don't know what we're (01:20:18) going to fight. The enem is going to do (01:20:19) creative things, right? If you were (01:20:21) deploying it to a conflict like what's (01:20:24) unfolding between Russia and Ukraine, (01:20:28) well, you could test the system and you (01:20:29) could get good data and maybe like you (01:20:32) have a good sense of what's going to be. (01:20:33) But in situations where in peace time (01:20:35) you're planning, I think we should (01:20:37) expect that the systems will be less (01:20:39) capable in wartime. Um, and that's (01:20:43) something that militaries need to (01:20:44) account for. (01:20:45) >> Yeah. Uh and so so an obvious (01:20:48) consequence is um mistakes that lead to (01:20:52) more casualties or mistakes that lead to (01:20:55) um kind of strategic blunders uh for (01:20:58) whoever's weapons those are. Um, does (01:21:02) this also make it more likely uh that (01:21:05) there are escalations of conflicts (01:21:08) because of kind of mistakes that cause (01:21:12) adversaries to think a particular thing (01:21:14) is happening that isn't happening? (01:21:16) >> That's something that I'm certainly (01:21:17) deeply concerned about. Um, and you (01:21:21) know, we have situations where countries (01:21:23) will often be in militarized disputes (01:21:26) short of war, but they have their ships (01:21:29) and aircraft operating in close (01:21:31) proximity. Um, and we we get incidents (01:21:35) sometimes, we had them during the cold (01:21:36) war between the US and uh, Soviet Navy (01:21:40) and and Air Force. We have had incidents (01:21:42) between um, the US and China, for (01:21:44) example. Um, and I think that these are (01:21:48) dicey when there's humans involved. I (01:21:50) think one concern is that machines do (01:21:53) something surprising and then you know (01:21:56) we don't even know like was that (01:21:57) intentional act on the part of the other (01:21:59) country or not. People aren't sure how (01:22:01) to interpret that. I think that's (01:22:03) certainly one concern but that concern (01:22:06) about escalation control still exists in (01:22:08) wartime that a lot of times countries (01:22:11) are still managing escalation in (01:22:12) wartime. we could see very tightly in (01:22:15) the Russian Ukraine war. The two (01:22:17) countries are going sort of all out (01:22:19) against each other. Um but Russia is (01:22:23) being very careful in calibrating its (01:22:25) escalation against NATO and the US has (01:22:28) been very cautious in its support for (01:22:30) Ukraine and not over escalating the (01:22:32) conflict. And so that's an area where (01:22:34) now we've had recently for example some (01:22:36) Russian provocations of drones flying (01:22:38) into NATO airspace. That seems like a (01:22:41) very intentional act on the part of (01:22:42) Putin trying to to sort of slowly erode (01:22:46) at some of NATO's deterrence. Um, but (01:22:49) that could look very different if you (01:22:50) had some crop of drones do something or (01:22:53) even strike targets and then you don't (01:22:54) know whether that was intentional or (01:22:56) not. Um, or even you could have drones (01:22:58) just do it by accident and then maybe (01:23:00) intent unintentionally cause escalation. (01:23:02) >> I guess I'm interested in really (01:23:04) concretely understanding the stakes. Um, (01:23:07) so what kind of what kinds of outcomes (01:23:10) um do you think are yeah most worrying (01:23:14) and people should be most paying (01:23:15) attention to when thinking about um (01:23:18) these weapons being deployed? (01:23:21) >> Yeah, I think there's there's several (01:23:22) when it comes to autonomous weapons (01:23:24) specifically. I think one concern of (01:23:26) course is that you could see much (01:23:28) greater civilian harm on the battlefield (01:23:30) either because of accidents because some (01:23:32) you know autonomous weapons or group of (01:23:34) them strike the wrong targets and and (01:23:36) cause civilian casualties. It could be (01:23:38) because they strike the right targets (01:23:40) but they're not correctly accounting for (01:23:42) civilians nearby. So okay it's a it's a (01:23:45) valid enemy tank but it's parked next to (01:23:47) a hospital and it caused all these (01:23:49) civilian deaths just to take out this (01:23:50) tank. it's not not proportional in terms (01:23:52) of the the military necessity of (01:23:55) striking that target. Um it could be (01:23:58) that that that there's a slow erosion of (01:24:01) human responsibility (01:24:03) that (01:24:04) >> by automating attacks, humans sort of (01:24:09) don't feel morally responsible anymore. (01:24:11) And so even if humans are sort of (01:24:13) pushing the button and authorizing it, (01:24:14) humans are just less engaged with what's (01:24:17) occurring. Um, that's I think one (01:24:19) concern. Certainly for countries that (01:24:21) just aren't concerned about civilian (01:24:23) casualties, this could allow greater (01:24:25) destruction. Um, and then I think in (01:24:28) terms of escalation, I guess my concern (01:24:30) would be that that autonomous weapons (01:24:32) sort of introduce another there's sort (01:24:35) of concept of the slippery slope (01:24:37) >> towards warfare, right? Um, and a lot of (01:24:40) a lot of conflict short of war involves (01:24:43) brinkmanship where um opposing leaders (01:24:47) might be sort of dragging each other (01:24:50) further down the slipper slip, but (01:24:52) nobody really knows when they're going (01:24:53) to tip over the edge. That's certainly (01:24:54) what Putin is doing with these (01:24:55) provocations against NATO. Um, is (01:24:58) engaging in this kind of brinkmanship (01:24:59) and that autonomous weapons might make (01:25:03) this slope slipperier in ways that are (01:25:05) hard to see and we don't understand. Um (01:25:08) and so you know take an example like the (01:25:09) Cuban missile crisis. Well maybe you (01:25:12) have um autonomous aircraft or drone (01:25:16) boats or automated missile defenses (01:25:18) shoot down um something and then maybe (01:25:21) make it harder for (01:25:23) political leaders to walk back, right? (01:25:25) Um maybe even there's an outcry (01:25:28) domestically and you know you know (01:25:31) remember the main we need to you know (01:25:34) remember these service members who were (01:25:35) killed and we need to strike back and we (01:25:37) can't show that we're weak. Um you know (01:25:41) I think that's a there aren't really (01:25:42) historical examples of accidental wars. (01:25:45) There are instances where you can see (01:25:46) miscalculation certainly in many cases. (01:25:49) Um but I think that we could see (01:25:51) circumstances where there's greater (01:25:53) miscalculation or there's greater (01:25:55) accidents that might escalate conflicts. (01:25:59) >> Yeah. Yeah. Um that that makes sense. Um (01:26:02) and is and is deeply unsettling. Um (01:26:05) another risk that came to mind uh for me (01:26:07) while um while reading your books uh is (01:26:11) kind of concentration of power. So, (01:26:14) could highly automated command (01:26:17) structures make it easier for a small (01:26:19) group um or even an AI system itself um (01:26:23) if that AI system uh were both very (01:26:26) capable and had had goals that weren't (01:26:29) perfectly aligned with um its kind of (01:26:32) creators goals. Um could could this (01:26:35) enable a small group or AI to seize hold (01:26:38) sorry uh to seize or hold power um (01:26:41) because there are fewer you need like (01:26:44) kind of the buy in and support of fewer (01:26:46) fewer humans at one time. (01:26:48) >> I mean I think the directionality of (01:26:50) that argument is right. I think the (01:26:52) question is kind of the magnitude of (01:26:53) like how strong would that effect be? Um (01:26:57) so dictators often um are able to (01:27:00) maintain control over larger populations (01:27:03) with the minority of the population on (01:27:06) their side based on you know an ethnic (01:27:08) minority or political minority. um you (01:27:11) know but they do need people certainly (01:27:13) one effect of this technology whether (01:27:15) it's autonomous weapons in the form of (01:27:17) robots or police robots or um you know (01:27:21) AI systems to process information might (01:27:23) be that you just need fewer people and (01:27:25) we've had instances where dictatorships (01:27:28) fall because the dictator tells the (01:27:29) troops fire on the protesters and they (01:27:32) just don't the troops lay down their (01:27:33) weapons and they said we're not going to (01:27:34) these are these are our families and (01:27:37) community members and we're not going to (01:27:38) shoot (01:27:39) Um and sort of you you you could take (01:27:42) away that ability for humans to just say (01:27:45) no, right? And so if you added like lots (01:27:48) of autonomous weapons, um you could (01:27:50) imagine like in 1989 when the Eastern (01:27:53) block had fallen like maybe there's (01:27:55) scenarios where it doesn't that allows (01:27:58) smaller groups to hold on to power (01:27:59) longer. I think in the extreme case when (01:28:02) you think about okay like would it allow (01:28:04) one individual or a small group to or an (01:28:06) AI system itself to like execute a coup (01:28:09) or take over like I think it requires a (01:28:10) little bit more like what's the specific (01:28:12) mechanic of that like how would that (01:28:13) work exactly um I think there's as we (01:28:18) see more infrastructure come online and (01:28:21) be more deeply integrated into cyerspace (01:28:24) that introduces lots of vulnerabilities (01:28:27) and areas where groups (01:28:29) using AI for cyber attacks or (01:28:31) conceivably down the road um AI itself (01:28:34) could do much more harm in the real (01:28:36) world by sort of taking over cyber (01:28:40) infrastructure right and that could look (01:28:41) like uh power grid that could look like (01:28:45) you know water treatment plants and and (01:28:48) uh you know nuclear power plants and and (01:28:50) other kinds of things certainly digital (01:28:52) infrastructure um one of the things that (01:28:54) you know in the 20th century that (01:28:55) countries would do and they had a coup (01:28:56) is they take over the radio station (01:28:58) right And now it's social media (01:29:00) platforms, other things. You could (01:29:01) imagine AI systems doing that and even (01:29:03) doing it in ways that might be subtly (01:29:05) manipulative, right? Like (01:29:06) >> you know platforms are not transparent (01:29:09) about the algorithms behind their social (01:29:10) media platforms and what how what kind (01:29:13) of content is promoting that's already (01:29:14) very contested politically. Um in AI (01:29:18) systems it's worse because the companies (01:29:21) themselves don't even really know what's (01:29:23) driving the models to generate certain (01:29:26) types of things, right? Like it's much (01:29:28) harder. So, so could you get like weird (01:29:30) sneaky biases? I think that's plausible. (01:29:33) I think to (01:29:36) um to get to more cat like really truly (01:29:38) catastrophic scenarios like some sort of (01:29:41) small group or an AI sort of taking over (01:29:43) a political system, you sort of need to (01:29:45) I think probably end up in a world (01:29:47) that's maybe more wired digitally than (01:29:49) today. um where the trend lines are and (01:29:52) where (01:29:54) um like military forces are much more (01:29:57) highly automated than they are now, (01:29:59) right? And there still probably would be (01:30:01) humans in the loop for lots of sensible (01:30:02) reasons for militaries. Like I would be (01:30:05) more worried about maybe things like um (01:30:09) small groups of people or AI like taking (01:30:11) control of corporations which have a lot (01:30:13) of power in modern society um and are (01:30:16) sort of well baked sort of like sort of (01:30:18) designed for this kind of thing. um and (01:30:22) that and then leading to political power (01:30:24) or influencing the information (01:30:26) environment or just AI systems (01:30:27) themselves become so central to the (01:30:29) information environment that they're (01:30:31) being used to manipulate information and (01:30:34) politics like sort of in some way (01:30:36) everything in society is downstream of (01:30:40) information flows right culture politics (01:30:44) like is all so like if you can (01:30:46) manipulate that that seems like like a (01:30:49) really alarming failure mode and then (01:30:51) who needs to (01:30:53) >> do some sort of 20th century style coup (01:30:56) with the tanks, you know, it doesn't (01:30:58) matter. You're you're in control. (01:31:00) >> Yeah. Yeah. Um Okay. So, those are some (01:31:03) some things that worry you. Um I'm (01:31:06) interested in kind of the strongest case (01:31:08) that these weapon systems will actually (01:31:10) make us safer and war less deadly. Um I (01:31:15) guess just sticking on kind of coups and (01:31:17) concentration of power. Um could (01:31:20) automation make these things more (01:31:22) difficult by improving transparency and (01:31:24) monitoring and just making it harder for (01:31:27) um kind of things to be done in secret (01:31:30) or in surprise. (01:31:32) I think there's a broader case to be (01:31:35) made that we are sort of on a trajectory (01:31:38) towards greater transparency towards a (01:31:40) world of of increasingly radical (01:31:42) transparency which just secrets are are (01:31:44) harder to keep (01:31:45) >> than they used to be. That's true in our (01:31:47) personal lives. That's true for (01:31:48) companies. is true for governments and (01:31:50) and intelligence communities that as (01:31:54) like to steal secrets back in the old (01:31:57) days, you had to get somebody into like (01:32:00) a vault and get a hold of paper (01:32:01) documents and then you know like the (01:32:04) people that that stole out documents for (01:32:06) the Pentagon papers shoved them in their (01:32:08) pants to to sneak them out and make (01:32:10) photo copies or you know you got to get (01:32:11) a camera in to take pictures and things (01:32:13) where now everything's digital and so if (01:32:15) you can hack the right system you get (01:32:16) access to these huge troves and we've (01:32:18) seen (01:32:19) um you know and things like like (01:32:21) Wikileaks like these massive dumps of of (01:32:24) documents hacking of things like the uh (01:32:27) Chinese hack of the office of personnel (01:32:29) management where millions of personnel (01:32:31) records by the US government were stolen (01:32:32) or the most recent Chinese telecom hack (01:32:35) where China's inside the telecom (01:32:37) infrastructure like the digital (01:32:38) infrastructure of modern society means (01:32:40) that you can hack these key companies (01:32:43) you just have access to everything like (01:32:45) you run the pipes and everything is (01:32:47) increasing the digital And that's (01:32:48) probably only going to increase, right? (01:32:50) So conversations that like this (01:32:54) conversation, right, we put online, (01:32:57) recorded, will be able to be digested by (01:33:00) AI, right? Like things are more (01:33:02) digitized, more communications in our (01:33:04) personal lives. The business has become (01:33:06) more digitized. And so that just maybe (01:33:08) makes it harder for anybody to do (01:33:09) anything in secret and including for (01:33:12) intelligence agencies uh and for (01:33:14) militaries. And that, you know, could be (01:33:17) good and bad in lots of ways. Um, but (01:33:19) one might argue might make war harder (01:33:22) because you just you can't get away with (01:33:25) stuff that people might have 50 years (01:33:27) ago. (01:33:28) >> Right. Right. Yeah. Okay. So, that's (01:33:30) that's one way it might make war harder. (01:33:33) Um, (01:33:35) which (01:33:36) isn't Yeah. which which you might think (01:33:39) um is a reason to think that it'll also (01:33:41) make wars uh less common and the world (01:33:44) more stable. Um what do you think is (01:33:47) kind of the strongest argument um that (01:33:49) integrating AI into military systems (01:33:51) could reduce suffering I guess rather (01:33:53) than increase it? (01:33:54) >> So I think there's actually a super (01:33:56) strong argument here. (01:33:57) >> Uhhuh. (01:33:58) >> Okay. So, and I think the the um (01:34:00) argument in a nutshell is that humans do (01:34:02) a terrible job at this and cause a lot (01:34:05) of civilian deaths through um war (01:34:08) crimes, through accidents, or just (01:34:11) through the imprecision of modern (01:34:13) weaponry. (01:34:15) And AI could do better. And just like (01:34:17) self-driving cars should be able to save (01:34:19) lots of lives on roads by just being (01:34:22) more precise, that AI could do the same (01:34:24) to warfare. that AI could enable (01:34:26) militaries to more precisely strike (01:34:27) military targets and not strike civilian (01:34:29) targets. Now, that does hinge on whether (01:34:31) militaries are trying to do that, which (01:34:34) is not always, like to be fair, it's not (01:34:35) always the case. It is a war crime to (01:34:38) intentionally kill civilians to strike (01:34:40) civilian targets. (01:34:42) Militaries also do that sometimes, (01:34:45) right? That's happened historically (01:34:46) throughout war that that nations often (01:34:48) will target civilians. Um and we (01:34:51) continue to see that in in in modern (01:34:53) conflicts where countries that are (01:34:54) deliberately killing civilians or or you (01:34:56) know not certainly not trying not to. Um (01:34:59) I think the strongest case would be if (01:35:02) you look at the pattern of precisiong (01:35:05) guided weapons over the course of the (01:35:08) 20th century it has led to um less (01:35:12) civilian suffering. So in World War II, (01:35:16) you had to basically drop like massive (01:35:19) amounts of ordinance to hit a bridge or (01:35:24) a factory inside a city. And that led to (01:35:26) wholesale devastation of cities in (01:35:30) Germany and Japan because it you just (01:35:33) couldn't actually target the (01:35:34) infrastructures precisely as militaries (01:35:36) might want to. Now there were some cases (01:35:39) you had um some countries sort of (01:35:41) attempting widespread aerial bombing (01:35:43) trying to to harm civilians but the US (01:35:46) army air force going after German cities (01:35:48) for example tried to do precision (01:35:49) bombing going after industrial targets (01:35:52) and just like it just wasn't precise (01:35:53) enough that changed with the advent of (01:35:55) precision guided weapons when you get to (01:35:58) um modern day where um the US military (01:36:02) at least with with you know GPS guided (01:36:04) bombs or laser guided bombs can strike (01:36:05) targets with a high degree of precision. (01:36:09) As a result, not only are there less (01:36:11) civilian casualties um when say an air (01:36:15) force using pursuit guided weapons as (01:36:16) bombing targets, our perception of (01:36:19) acceptable civilian casualties has (01:36:21) changed as a result. And so um for (01:36:24) example in debates here in the United (01:36:26) States about um US drone campaigns you (01:36:30) would have sort of this there became (01:36:32) this expectation that drone strikes (01:36:36) would have zero civilian casualties. Now (01:36:38) one could argue that's right or wrong (01:36:40) but it's a huge shift from how people (01:36:43) thought about that say 80 years prior. (01:36:45) Um, and so I think that's sort of the (01:36:47) strong case of technologies actually on (01:36:49) the side of a greater precision and that (01:36:53) could lead to less suffering and war. (01:36:56) >> And so overall, (01:37:00) how would you describe your general (01:37:02) feeling about um, yeah, this kind of (01:37:06) radical increase in automation that you (01:37:08) expect to happen over the next several (01:37:09) decades? (01:37:10) >> Well, I think I mean I feel great about (01:37:12) it all. I mean, I think that um (01:37:15) >> I think that it really depends on how (01:37:17) militaries use AI, and I think that (01:37:21) there are ways for militaries to use AI (01:37:24) um that might make warfare more precise (01:37:27) and more humane. We need to ensure that (01:37:29) we're adopting ways that don't lose our (01:37:32) humanity in the process. I think it's (01:37:34) important to keep humans in control of (01:37:37) warfare to manage escalation so that (01:37:39) humans have the ability to end wars. And (01:37:42) I think it's it's important for humans (01:37:44) to bear some moral responsibility for (01:37:48) killing and suffering in war. And that's (01:37:50) I think actually like a harder argument (01:37:52) because (01:37:54) it's it's real people that bear that (01:37:56) cost, right? That's someone who then is (01:37:59) suffering from PTSD afterwards because (01:38:03) of their effects in war. And you know, I (01:38:05) I fought in Iraq and Afghanistan. I've (01:38:07) had a a lot of friends who who've (01:38:10) continued to suffer after the war has (01:38:14) ended from not just physical but but (01:38:17) mental and emotional sort of injuries, (01:38:19) moral injuries that they might have (01:38:20) suffered during the war. Um, (01:38:23) and it's not really fair as a society (01:38:26) that as a democratic society we make a (01:38:28) decision as a whole to go to war and (01:38:31) it's a very small slice of people that (01:38:32) bear that cost for the war. Um, but I do (01:38:36) think it's worth asking what would it (01:38:39) mean for war? What would it mean for us (01:38:41) if no one slept uneasy at night if no (01:38:44) one was concerned about the suffering (01:38:47) and the casualties that occurred in war? (01:38:49) And would that make wars more likely or (01:38:53) more deadly? Um so I you know I I think (01:38:57) it I think it's possible to imagine a (01:39:00) future where we adopt this technology in (01:39:04) a way that does lead to positive (01:39:06) outcomes that makes warfare more precise (01:39:08) and humane and doesn't lose our humanity (01:39:09) in the process but I think it matters in (01:39:11) how we do it. Um and I think we want to (01:39:13) be thoughtful about how we use the (01:39:15) technology. (01:39:16) >> Yeah. Pushing on we often talk on this (01:39:19) podcast about transformative AI. Um so (01:39:23) AGI super intelligence um other terms (01:39:26) describing kind of highly advanced AI (01:39:28) systems uh and kind of achieving them is (01:39:31) the stated aim of many of the top AI (01:39:33) firms right now. So do you personally (01:39:36) buy the arguments that AGI could kind of (01:39:38) massively transform society in the next (01:39:41) few decades? (01:39:42) >> Yeah, I mean I think it's actually kind (01:39:44) of bonkers for people not to believe (01:39:47) that given everything we've seen with (01:39:48) the technology. We've been hearing for (01:39:50) years now, oh, deep learning is hitting (01:39:52) a wall. It's petering out. Maybe that'll (01:39:55) happen. (01:39:57) >> But that prediction has been (01:39:59) consistently wrong so far. And what (01:40:03) we've seen a pattern of is an AI system (01:40:06) will be released to great fanfare. (01:40:08) oftentimes people sort of poke at it and (01:40:10) realize well it's not quite as good as (01:40:12) maybe a company had hyped it out to be (01:40:15) and it has a bunch of criticisms or (01:40:17) people will say oh you know it can't do (01:40:19) this can't do it can't reason it's not (01:40:21) really reasoning and then six or eight (01:40:23) months later another AI system comes (01:40:25) along that like just totally obliterates (01:40:28) that criticism (01:40:30) and so given those trend lines I mean I (01:40:33) think it seems quite reasonable to think (01:40:34) that we're going to continue to see AI (01:40:36) improve and one of the things that I'm (01:40:38) certainly struck by is the these sort of (01:40:41) long timelines that sometimes people (01:40:43) have now towards AGI are like often (01:40:45) really short. So people will you critics (01:40:48) critics of AI is like well AGI's a (01:40:51) decade away like that's really close (01:40:53) like that's crazy like I remember things (01:40:56) 10 years ago it's not that long ago. So (01:40:58) like I you know I think I would say that (01:41:01) sort of my expectations for how fast (01:41:04) this is unfolding have certainly been (01:41:06) like many people been pulled forward (01:41:08) over the last couple years that things (01:41:09) that I sort of if I would have had to (01:41:11) make a best guess I thought well maybe (01:41:12) this will happen by 2040 are happening (01:41:14) like now right um and I'm like oh wow (01:41:17) and I and I I'm really struck by that (01:41:20) because I've been working on these (01:41:22) issues for um you know probably 15 years (01:41:27) in various forms in the Pentagon and in (01:41:29) in um here at the the think tank I work (01:41:32) at the center for new American security. (01:41:34) Um I was certainly working on AI issues (01:41:37) before people were calling them AI when (01:41:39) it was like automation or auty or (01:41:41) something before the deep learning uh (01:41:42) this version of the deep learning kind (01:41:44) of revolutionary kicked off and (01:41:47) it is sort of strikes me that I (01:41:49) continually am surprised by the pace of (01:41:51) progress. Um, and that sort of concerns (01:41:54) me a little bit. And I think what (01:41:56) worries me in particular is that a lot (01:41:59) of the technology seems to democratize (01:42:03) violence and and in particular at very (01:42:07) extreme scales. So, I think the (01:42:10) questions about autonomous weapons, for (01:42:13) example, that we've kind of been talking (01:42:15) about or or swarms and warfare are (01:42:18) interesting. I think they're concerning. (01:42:19) there's a lot that we ought to be doing (01:42:22) um that nations ought to come together (01:42:23) to sort of put some rules of the road in (01:42:25) place. That's not really the scary (01:42:27) stuff. The scary stuff is things like (01:42:28) biological weapons. It's the (01:42:30) intersection of AI and cyber security (01:42:34) and biological weapons. It's things like (01:42:37) um you know using AI to design much more (01:42:40) powerful synthetic biological weapons. (01:42:43) It's democratizing that technology as we (01:42:45) see AI tools and large language models (01:42:49) and more general purpose systems and (01:42:50) agents become open source and available (01:42:53) to anyone. (01:42:54) Um, you know, it's it's as cyber sec as (01:42:58) we continue to digitize our world. (01:43:02) >> More critical things becomeworked and (01:43:05) vulnerable to disruption through cyber (01:43:08) attacks. power grid, water treatment (01:43:11) plants. Um our our sort of information (01:43:15) systems are now vulnerable to disruption (01:43:17) digitally through um hacking of uh (01:43:22) telecom networks which has occurred (01:43:24) through manipulating social media. Like (01:43:27) those sort of long-term trends really (01:43:30) worry me. And I think that there are um (01:43:34) some like really concerning outlier (01:43:36) possibilities in in really horrible (01:43:39) types of catastrophic harm that could (01:43:40) come from that. They're probably not (01:43:43) likely, but like I kind of I kind of (01:43:45) don't want to find out. (01:43:47) >> Like how likely is it that someone makes (01:43:51) some horrible biological weapon that (01:43:53) kills millions of or or hundreds of (01:43:56) millions or billions of people? Like I (01:43:58) don't know. like let's not let's not (01:44:00) inch up to that line. (01:44:02) >> Um so those are the scenarios that (01:44:03) really concern me. (01:44:05) >> Yep. Yeah. Do you think that these ideas (01:44:09) are taken seriously in kind of national (01:44:11) security circles or do they still sound (01:44:13) kind of sci-fi? I I think generally (01:44:15) that's not kind of sci-fi like you know (01:44:18) in in Washington um people have (01:44:21) certainly got the AI bug and that's been (01:44:23) true for a while that Washington is all (01:44:26) in on (01:44:29) um integrating AI into the US military (01:44:32) on maintaining sort of US dominance and (01:44:34) artificial intelligence overall visav (01:44:37) China in particular (01:44:40) these you know things like super (01:44:42) intelligence (01:44:43) or AGI or like a bit of a dirty word. I (01:44:47) mean, sort of AGI is becoming a term (01:44:49) that's becoming a little bit more (01:44:50) normalized in in discussion about AI. (01:44:54) So, less so. But I think it's (01:44:57) interesting because the security the (01:45:00) sort of defense and security community (01:45:02) that I come from and that I work in (01:45:04) spends a lot of time thinking about (01:45:05) hypothetical scenarios. We do detailed (01:45:09) games and scenarios of, you know, (01:45:13) nuclear war with with China or Russia or (01:45:18) you know different you know there's a (01:45:19) war unfolding between different kinds of (01:45:21) countries and how does the US respond (01:45:23) and we do like really detailed analysis (01:45:25) that people take those things very (01:45:26) seriously so if I were to you know do a (01:45:30) project for example um and lots of (01:45:32) experts do this on you know how a major (01:45:35) war between the United States and China (01:45:36) might unfold over a period of uh not you (01:45:40) know maybe even years like a very (01:45:41) protracted conflict and maybe there's (01:45:43) limited nuclear use involved. People (01:45:45) take that like super seriously. But then (01:45:47) if you start talking about, okay, well, (01:45:48) I've got this um really sophisticated AI (01:45:51) agent and it escapes from a lab and it (01:45:54) um spreads on the internet and then it (01:45:56) hacks um you know like critical (01:45:58) infrastructure and takes down the power (01:46:00) grid. People are like, "What are you (01:46:02) talking about?" Like you've been (01:46:03) watching too many movies. And I've (01:46:06) observed this trend long enough in the (01:46:09) security space that it's interesting to (01:46:11) me that that's the same reaction that (01:46:13) people had to, you know, autonomous (01:46:15) weapons or drone swarms 15 years ago. (01:46:18) >> Well, those are just not like if you (01:46:19) were talking about that, people sort of (01:46:20) like, I think you watch too many (01:46:22) Terminator movies (01:46:23) >> and now those are taken quite seriously. (01:46:25) And so I do think that um people will (01:46:27) get there, but for some reason there is (01:46:29) a a real hangup on taking some of these (01:46:33) scenarios seriously. (01:46:35) >> Do you have a guess on what specifically (01:46:37) the hang-up is? (01:46:38) >> I I don't really know like I've I've (01:46:40) spent a lot of time actually puzzling (01:46:42) over this that I'll be in, you know, at (01:46:44) conferences or in conversations people (01:46:46) or in roundt discussions or private (01:46:48) workshops and I've thought this to (01:46:51) myself a lot like what is the hangup (01:46:53) here? And it seems like there's (01:46:55) something about this idea of AI reaching (01:47:00) or eclipsing human intelligence (01:47:02) that sort of is like really hard for (01:47:05) people cognitively that wrap their minds (01:47:06) around and you tend to get really weird (01:47:10) reactions from people when when you sort (01:47:12) of put that question on the table. I (01:47:14) think there's some people that just (01:47:15) reject it as like not possible, not (01:47:18) going to happen in any reasonable time (01:47:19) frame. It just it just strikes him as (01:47:20) fanciful. I think the flip side is I'll (01:47:23) often see a lot of writing in this space (01:47:26) where people get to this point of AGI (01:47:29) and then it's like magic happens. (01:47:31) >> Right. (01:47:32) >> Right. It's like oh well then the AI (01:47:33) becomes like because it's it it can (01:47:35) program itself and then it recursively (01:47:37) self-improves and then it's you know (01:47:39) super intelligent within a period of (01:47:41) hours or weeks or months or whatever you (01:47:43) know your sort of vision of what that (01:47:45) kind of takeoff looks like and then it (01:47:47) can just do anything right like it could (01:47:50) take over the world and it builds these (01:47:51) robot factories and you know it it and (01:47:53) you're like well like there are the (01:47:55) physical constraints (01:47:57) that exist um like how exactly would (01:48:01) those sort things unfold and it's like (01:48:04) well super intelligent you figure all (01:48:06) that out and so and I and I think (01:48:08) sometimes you get people then reacting (01:48:09) to that they're like well that's just (01:48:11) nonsense and it strikes them as fanciful (01:48:14) and so I think it's some there's (01:48:15) something about this idea I don't know (01:48:17) what it is of AI eclipsing human (01:48:20) intelligence that seems really hard for (01:48:22) people to just like like grapple within (01:48:26) a way that's grounded and that takes it (01:48:28) seriously and I don't I don't know why (01:48:30) that (01:48:32) Yeah, super interesting. Pushing on, (01:48:35) what impact will AI and autonomy have on (01:48:38) cyber warfare? Well, this is a place (01:48:41) where I do think the effects of AI will (01:48:43) be very dramatic and much faster than in (01:48:46) other areas like in physical domains, (01:48:49) but it's it's maybe worth starting where (01:48:51) we are today, which is the effect is (01:48:55) somewhat limited of AI and large (01:48:59) language models at least. Now, (01:49:01) automation is already widely used in (01:49:06) cyber security both for defensive and (01:49:08) offensive purposes. Um, the first (01:49:11) self-replicating worm, the Morris worm, (01:49:13) dates back to the 1980s. So, we've had (01:49:15) self-replicating malware for a while now (01:49:17) that can spread across computer (01:49:18) networks. Um, we've been some really (01:49:20) quite sophisticated cyber weapons. (01:49:23) Stuckset, the one that, um, is is widely (01:49:26) believed to be built by the United (01:49:27) States and Israel that took down Iranian (01:49:30) centrifuges to sabotage the nuclear (01:49:32) program, had really sophisticated (01:49:34) automation so that it could spread (01:49:37) across networks that were airgapped from (01:49:39) the internet and where the sort of (01:49:42) people controlling them, the the humans (01:49:44) couldn't direct uh, what the malware was (01:49:47) doing. and some forms of automated (01:49:50) vulnerability discovery and patching (01:49:52) have been around for several years now. (01:49:54) I think we should expect that AI will (01:49:56) continue um to to sort of advantage both (01:50:02) attackers and defenders here by finding (01:50:05) more vulnerabilities more effectively um (01:50:09) by being able to recon networks to (01:50:13) better understand what's happening sort (01:50:15) of without sort of throughout the um (01:50:18) killchain if you will of a cyber attack (01:50:20) that AI will start to sort of play roles (01:50:22) incrementally in each of these in (01:50:25) enhancing human productivity. (01:50:28) Um I think the sort of interesting (01:50:30) question is do we get to the a point (01:50:33) down the road where malware is much more (01:50:37) intelligent and adaptive than today? And (01:50:40) so today you have malware that um (01:50:42) spreads on its own that is (01:50:44) self-replicating that acquires resources (01:50:48) like botn nets that acquire computing (01:50:50) resources and then can use them for (01:50:52) things like distributed denial of (01:50:54) service attacks can sort of leverage (01:50:55) that. But when there are adaptations to (01:50:59) malware, those are done manually. And so (01:51:02) Configer uh sort of this huge worm that (01:51:04) spread across the internet several years (01:51:05) ago is a really interesting case where (01:51:07) there were a bunch of variants that (01:51:11) evolved over time. And so the the task (01:51:14) force that was sort of put together of (01:51:16) law enforcement and intelligence (01:51:17) communities and the private sector to (01:51:20) combat this worm was fighting different (01:51:23) variants over time but those were all (01:51:27) designed by humans (01:51:30) and you know so do we get to the point (01:51:32) where you have malware that's actually (01:51:34) able to evolve and adapt so much on it (01:51:37) own either it's more clever when it's on (01:51:40) a computer network and able to maybe (01:51:43) hide itself in response to threats or (01:51:45) adapt what it's doing to the network (01:51:47) itself. And you can imagine a more (01:51:49) capable (01:51:51) reasoning model that could, you know, (01:51:54) assess what's going on on computer (01:51:55) networks and then sort of make some (01:51:57) reasonable judgment about what to do. (01:51:59) Um, or able to actually change itself (01:52:03) over time and which would seem like a (01:52:06) much more dangerous kind of threat. And (01:52:08) we've seen you sort of like concerning (01:52:12) um attempts by language models to engage (01:52:16) in behavior like self-exfiltration (01:52:19) um copying itself, copying itself in (01:52:22) ways that would try to preserve its (01:52:23) goals or copying itself to overwrite the (01:52:27) goals that um a human would do of a new (01:52:30) system. Now, the models aren't very good (01:52:31) at that yet because they're just not (01:52:32) good enough yet at software engineering, (01:52:35) but you sort of have all of the pieces (01:52:37) in place. Self-replication (01:52:40) already exists. Uh the ability to (01:52:42) acquire computing resources already (01:52:44) exists. Um the sort of tendency of (01:52:48) models, not it's not common, but it (01:52:52) happens where models might attempt some (01:52:54) of these kind of concerning behaviors (01:52:56) like self exploration. It looks like (01:52:58) right now the missing pieces just (01:53:00) they're just not good enough. Like (01:53:02) that's going to get better. Like you (01:53:04) could really take that to the bank. It's (01:53:05) going to get better. On my what (01:53:06) timeline? I don't know. So I think that (01:53:08) that's a a very troubling possibility in (01:53:12) the long term that you could end up with (01:53:15) malware that is um maybe feels more like (01:53:20) biological threats. (01:53:22) >> Mhm. (01:53:23) >> Right. were during co we saw different (01:53:24) variants over time and then you're sort (01:53:27) of fighting against this threat that's (01:53:28) continuing to evolve and that's a seems (01:53:30) like a really difficult problem. (01:53:31) >> Yeah. Yeah. And does that have (01:53:38) differing implications for either (01:53:40) different types of groups or different (01:53:42) powers? Um, or is that kind of just like (01:53:45) uniformly (01:53:47) like cyber war, cyber warfare gets more (01:53:52) impactful across the board? (01:53:55) >> I think it's obvious that it gets more (01:53:57) impactful because a lot of that has to (01:53:58) do with how well defenders um find ways (01:54:02) to shore vulnerabilities inside their (01:54:05) networks. And so this question of how (01:54:07) does the offense defense balance how (01:54:09) does AI change the offense defense (01:54:11) balance in cyber security is I think a (01:54:13) really critical one and I've seen (01:54:14) compelling arguments from good analysts (01:54:17) on both sides of this equation (01:54:18) >> huh (01:54:19) >> so I think it's worth starting with okay (01:54:21) so today right now (01:54:24) um cyber security greatly benefits (01:54:27) attackers and the reason why is that (01:54:30) ultimately attackers get into computers (01:54:33) and networks by finding finding (01:54:35) vulnerabilities by finding mistakes, (01:54:36) bugs in code that they can then exploit. (01:54:40) And so the problem that defenders had is (01:54:42) that you have these massive massive code (01:54:46) bases for, you know, an operating system (01:54:48) on a computer or a um industrial control (01:54:53) system for some industrial plant or (01:54:55) something. And it's a little bit for (01:54:58) defenders like trying to defend this (01:55:00) castle that has these sprawling walls (01:55:05) that stretch and snake over hills for (01:55:08) miles. And you've got to cover every (01:55:11) single possible entry point. The (01:55:12) attackers only need to find one (01:55:13) vulnerability, one door that's unlocked, (01:55:16) one secret tunnel that you didn't know (01:55:17) about that they can get in through. And (01:55:19) then once they're in, like they can (01:55:21) cause all sorts of problems. They can (01:55:22) escalate their privileges. they can (01:55:24) create new vulnerabilities to find other (01:55:26) ways in. And so, um, that's kind of the (01:55:30) status quo today. One of the, I think, (01:55:33) strong cases for a couple of them for (01:55:36) how this technology might benefit (01:55:38) defenders is um, one is that if you have (01:55:42) AI that can be used to automatically (01:55:44) find vulnerabilities, sort of finding a (01:55:47) vulnerability and finding the patch for (01:55:48) the vulnerability are are sort of the (01:55:50) same, right? So if you can find the (01:55:52) vulnerability, you know how to patch it. (01:55:54) And if defenders use this technology to (01:55:57) run it against their networks before (01:55:59) it's deployed, or they're just really (01:56:01) much more assertive in doing that, they (01:56:03) can automate a process that right now is (01:56:05) manual. And um and so so that can allow (01:56:09) them to find and patch these (01:56:10) vulnerabilities before attackers can get (01:56:11) in. So that sort of starts to level the (01:56:13) playing field. Attackers can use these (01:56:15) too, but right now what's limiting (01:56:18) defenders is this like this the the (01:56:22) human cost of going through all this (01:56:24) code and automation really relatively (01:56:28) advantages them. Now that hinges a lot (01:56:30) on do attack defenders actually do that (01:56:32) and that's a big problem right now in (01:56:33) cyber security is a lot of times it's (01:56:36) it's actually a vulnerability that we (01:56:38) knew about that the patch is available (01:56:42) but people haven't updated their (01:56:44) networks or their computers and that's (01:56:45) not always true but that is a consistent (01:56:47) problem. Um and so you know do do (01:56:50) defenders actually do that? The other (01:56:52) really interesting compelling case is (01:56:55) that AI I'm not sure that we're there (01:56:57) yet, but as AI gets better at writing (01:56:59) code, we just have fewer bugs. H and (01:57:03) that actually is I think really (01:57:04) compelling because that doesn't really (01:57:06) hinge on defenders necessarily doing (01:57:08) anything intentionally. It's just that (01:57:10) as we evolve over time to a world where (01:57:13) maybe more and more software is just (01:57:15) written by AI, if the AI gets pretty (01:57:18) good at it, there just might be fewer (01:57:21) vulnerabilities in the first place and (01:57:22) that actually just sort of shores up (01:57:23) defenses. Um, but a lot of it depends (01:57:27) upon how the technology is employed by (01:57:29) both sides. (01:57:31) >> Yeah, I mean, yeah, that last point (01:57:34) sounds pretty potentially huge. Um, (01:57:39) I guess (01:57:42) it does seem like things point in (01:57:44) different directions and (01:57:48) it seems like it's not clear to you that (01:57:51) one is definitely going to dominate. Um (01:57:53) but if you were to describe at least a (01:57:57) plausible scenario um that you put some (01:58:00) stock in where cyber capabilities (01:58:02) do end up genuinely shifting the global (01:58:06) balance of power or triggering (01:58:08) escalation between major states. Um what (01:58:11) is a scenario that uh kind of (01:58:15) might explain uh why that would end up (01:58:18) happening? (01:58:19) I think that the scenario that worries (01:58:22) me the most is that we end up in a world (01:58:27) where malware is much more intelligent (01:58:29) and adaptive and you can end up with (01:58:34) malware on the internet like we have (01:58:36) worms and botnetss today that are (01:58:40) intelligent. they're able to to to have (01:58:42) goals and to plan to execute them, to (01:58:44) adapt to defenders. And it's just like a (01:58:47) much more difficult problem for (01:58:50) defenders to go against (01:58:52) >> that. Um it's not like, okay, they're (01:58:56) they're fighting a a botnet and then (01:58:58) they could defeat it and then it's done. (01:59:00) It's that this is like an intelligent (01:59:02) and adaptive adversary in and of itself. (01:59:05) and sort of whether it was written I (01:59:08) mean ultimately would have been designed (01:59:09) and written presumably by humans but (01:59:11) whether it was used for a certain goal (01:59:13) and sometimes becomes irrelevant once (01:59:15) the malware is launched it's already (01:59:16) true today where there are lots of (01:59:19) examples where an actor sort of is (01:59:23) trying to do something that still might (01:59:25) not be great like they're trying to (01:59:27) steal passwords for example and then (01:59:29) they release some botnet that spreads (01:59:31) across the internet because of (01:59:32) replication and it causes all sorts of (01:59:34) problems (01:59:36) um or somebody releases something open (01:59:38) source and now there's all sorts of (01:59:40) variants of this that are used by (01:59:42) others. Um and that that seems like a (01:59:47) very different kind of world to be in to (01:59:48) be countering that kind of threat. And (01:59:51) why wouldn't this dynamic also be kind (01:59:55) of resolved or at least helped by um (01:59:58) this thing you've already described of (02:00:00) defenders being able to use capable AI (02:00:02) systems to defend against uh (02:00:04) increasingly kind of sophisticated uh (02:00:06) cyber attacks? (02:00:08) >> Well, presumably they would that (02:00:10) defenders would be using AI to improve (02:00:12) their defenses. I think the question, (02:00:14) you know, is like how does that net out (02:00:16) and why might we still have really nasty (02:00:18) threats? I mean one trajectory could be (02:00:20) that we just have the sort of this a (02:00:22) continuation of the relative balance we (02:00:24) have today which is that even though (02:00:28) people know vulnerabilities exist and (02:00:30) it's really important to defend against (02:00:32) threats there are still ways in humans (02:00:35) you know click humans are dumb and they (02:00:37) click on the link in the email that they (02:00:39) shouldn't or or you know um (02:00:44) people don't update their software in (02:00:46) ways that they should. So that could be (02:00:48) just one scenario where you just get (02:00:50) this the same context on a context on a (02:00:53) higher level. One way that you could see (02:00:56) this kind of really sophisticated (02:00:58) AIdriven malware benefit attackers in (02:01:01) relative terms is if they're just more (02:01:03) comfortable taking risks. Hm. (02:01:06) >> Then you could see maybe someone cobbles (02:01:09) together some um offensive cyber system (02:01:13) that has like an LLM as a component of (02:01:16) it and it's doing some reasoning and (02:01:19) it's writing software and it's sort of (02:01:22) not totally clear what this thing might (02:01:25) do, but an attacker has some it's just a (02:01:27) much higher risk tolerance, right? and (02:01:30) they're like, "Yeah, like let's I'm (02:01:31) going to use it to, you know, hack some (02:01:33) server, engage in some spam email thing (02:01:36) or some denial of service attack or (02:01:39) whatever they're trying to do." And (02:01:41) defenders have the same access, same (02:01:43) technology, maybe even a little bit (02:01:44) better if they have sort of early access (02:01:46) to models through more sophisticated (02:01:49) labs, but they're just more hesitant (02:01:51) because they're like, "Yeah, I don't (02:01:52) know what this thing's going to do, and (02:01:53) I can't have this sort of intelligent AI (02:01:56) defensive system running around on my (02:01:58) network." And then it decides that, you (02:02:00) know, the weakly are humans and it locks (02:02:03) out all the humans, (02:02:04) >> right? It does something strange, you (02:02:05) know, and and um that because of the (02:02:09) unpredictability of the systems that (02:02:11) defenders are a little more cautious in (02:02:13) employment and that might benefit (02:02:14) attackers. My colleague Caleb Withers at (02:02:17) the Center for New American Security has (02:02:18) a great report on this issue of cyber (02:02:21) security and why it might benefit (02:02:23) attackers that uh recently come out that (02:02:26) that's really worth checking out. Cool. (02:02:28) Okay. Yeah, we'll we'll link to that for (02:02:30) sure. Um, yeah. Before we move on, is (02:02:33) there anything that you feel is under (02:02:37) understood or underrated in the kind of (02:02:40) area of cyber warfare um as it gets kind (02:02:44) of more automated and more AI based um (02:02:46) that you'd yeah want to make a pitch (02:02:48) for? I mean, I think um (02:02:52) here's what worries me is that I see two (02:02:55) trend lines (02:02:57) that I think intersect in some troubling (02:03:01) ways. One is this trend line towards (02:03:04) more of human civilization and our lives (02:03:08) becoming digitized andorked and (02:03:10) accessible through computers. And that (02:03:14) trend line seems like it's likely to (02:03:16) just continue that we see exponential (02:03:18) growth in internet of things devices in (02:03:22) um network uh bandwidth for both wired (02:03:24) and wireless networks. It's more things (02:03:26) are becoming digitally connected which (02:03:28) makes them inherently vulnerable to (02:03:30) cyber attack. How vulnerable depends a (02:03:32) lot on how much defenders do but things (02:03:34) that 40 years ago you couldn't attack a (02:03:37) power station through the computer (02:03:40) network. There was like no way to do (02:03:41) that. Now it's been done. Russia has (02:03:44) taken down um you know elements of (02:03:47) Ukraine's power grid through cyber (02:03:49) attack. So that's been demonstrated for (02:03:51) example. So you have this long trend (02:03:54) line and then you have this simultaneous (02:03:56) trend line of artificial intelligence (02:03:58) becoming more capable. And that sort of (02:04:01) worries me that you could end up in this (02:04:04) place where you have much more (02:04:06) intelligent forms of malware, much more (02:04:08) sophisticated ones. And there's sort of (02:04:10) just this greater inherent vulnerability (02:04:13) in society over time (02:04:15) >> that um lots of things that we care (02:04:18) about (02:04:19) >> are actually now vulnerable just as the (02:04:24) systems that might hack them are (02:04:26) becoming much more sophisticated and (02:04:28) that that feels uncomfortable. (02:04:31) >> Yep. Yep. I I'm unsettled. (02:04:33) Um pushing onto another topic. Uh I'm (02:04:37) interested in how the integration of AI (02:04:40) and autonomy into military systems um (02:04:42) shapes the balance of power between the (02:04:44) US and China in particular. Um so yeah (02:04:47) is there is there kind of a highlevel (02:04:49) answer to that question before we get (02:04:51) into some details? (02:04:52) >> Well there are some ideas. So, one (02:04:55) argument that I've heard is that China (02:04:58) will be more willing to automate systems (02:05:03) because it's an authoritarian regime and (02:05:05) they don't trust their people. (02:05:09) I'm not sure that that's valid. Like, (02:05:10) that's a thing I hear in Washington, (02:05:12) people saying about China. When I talk (02:05:14) to Chinese military officers who engage (02:05:19) or former Chinese military officers who (02:05:20) engage in these kinds of issues, (02:05:22) thinking about AI in the military, I (02:05:23) don't really hear that. I hear a healthy (02:05:26) skepticism about AI and a desire to (02:05:30) adopt AI into the military, but also (02:05:32) concerns that, you know, maybe humans (02:05:34) should be on the loop or what sometimes (02:05:37) the the Chinese translation is sort of (02:05:39) above the loop. It has sort of this idea (02:05:40) of supervisory human control. Maybe (02:05:42) humans can't be in the middle of (02:05:44) everything, but we do know that these AI (02:05:46) systems, they they screw things up. We (02:05:47) don't necessarily trust them. Um, we (02:05:49) want to maintain control. There is a (02:05:51) strong desire in the Chinese system for (02:05:54) control from the top down. Um I think (02:05:58) one concern that I have would be about (02:06:02) risktaking (02:06:03) and asurances in sort of test and (02:06:06) evaluation of AI systems that there are (02:06:09) pretty robust procedures in place within (02:06:11) the US military to test new weapon (02:06:14) systems to make sure that they're (02:06:15) reliable once we deploy them. And that (02:06:18) the Chinese military, the People's (02:06:20) Liberation Army might be more willing to (02:06:22) take risk because they feel behind. they (02:06:24) feel like they need to catch up to the (02:06:26) United States or they get guidance from (02:06:28) senior leaders do this they're just (02:06:31) going to do it and like you know does (02:06:33) this drone have a high failure rate like (02:06:35) doesn't matter (02:06:37) the political she says do it so we need (02:06:39) to deploy this thing and and (02:06:43) that sort of concern about um risktaking (02:06:46) in accidents is something that I do (02:06:48) worry about (02:06:50) >> yeah can can you imagine kind of AI (02:06:52) accidents becoming the next flash (02:06:54) points. Um, so I guess kind of the (02:06:56) equivalent to near miss incidents during (02:06:58) the cold war. (02:06:59) >> Absolutely. Um, and that's I think a big (02:07:01) concern that I have that if you (02:07:03) transition to a world where say the (02:07:05) United States and China have uh drones (02:07:09) deployed at sea, in the air, under sea, (02:07:13) interacting in contested areas like the (02:07:15) South China Sea or near the Taiwan (02:07:17) Strait. Um, and they have some degree of (02:07:20) autonomy. They don't have to be fully (02:07:21) autonomous weapons, per se, but they (02:07:23) have some degree of autonomy that maybe (02:07:25) does something strange that causes an (02:07:26) incident. So, you've got an autonomous (02:07:28) boat and it gets too close to another (02:07:30) boat or causes a collision or a drone (02:07:33) that gets too close causes a collision (02:07:36) and then there's some mishap. There's a (02:07:37) political incident. Now, I don't really (02:07:40) think that those incidents themselves (02:07:42) then like lead to fullscale war like (02:07:46) that. Not unless leaders are looking for (02:07:48) an excuse to go to war. Um, but I don't (02:07:51) know that it's helpful to sort of (02:07:53) introduce a lot of this maybe more (02:07:56) friction and unpredictable dynamic into (02:07:57) that situation. And there might be (02:07:59) situations where leaders feel compelled (02:08:02) to respond in some way to look strong (02:08:04) and I think that would be concerning. (02:08:06) Yeah, I guess in in the nuclear space um (02:08:12) something roughly like nuclear parody uh (02:08:15) has created a kind of uneasy stability. (02:08:18) Is there a version of that for kind of (02:08:21) AI and automation parody in the military (02:08:24) that that does the same? (02:08:27) >> Um it depends a little bit I think on (02:08:30) the scenario that we're envisioning. (02:08:32) Mhm. (02:08:32) >> Um, so if you think about, you know, one (02:08:35) of the the scenarios that US war (02:08:38) planners really worry about is a Chinese (02:08:42) invasion of Taiwan. And (02:08:46) um, (02:08:47) she has said that the Chinese leader has (02:08:50) said that, you know, they Taiwan is is (02:08:53) part of China. That's their view. And (02:08:55) they intend to retake control by force (02:08:58) if necessary. and has directed the (02:09:01) Chinese military to be prepared to do so (02:09:03) by 2027. Now I don't know realistically (02:09:06) whether they'll be able to do that. I (02:09:07) think probably not. (02:09:09) um you know if technology I think the (02:09:12) key sort of threshold in that case in (02:09:14) that particular scenario is if they (02:09:15) think they could get away with it and I (02:09:18) think if you got to the place where they (02:09:20) believed their technology gave them an (02:09:22) advantage enough that they thought that (02:09:24) they could get away with it and keep the (02:09:26) US at bay and coupled with you know the (02:09:29) US is weak this the Chinese view the US (02:09:31) is weak it's a declining power it's (02:09:34) riven with domestic strife um the US (02:09:37) doesn't really care that much about (02:09:38) Taiwan when it when push comes to shove. (02:09:41) Um the US doesn't have the stomach for a (02:09:44) long protracted war even if the US does (02:09:47) get involved in the fight like China's (02:09:50) willing to gut it out over the long term (02:09:52) if it comes down to a bloody um you know (02:09:54) fight on the island and the US doesn't (02:09:55) have the stomach for that and we could (02:09:58) see how weak the US is in Ukraine and (02:10:00) they're not willing to support (02:10:01) Ukrainians and even US isn't even (02:10:03) fighting there. So that might lead China (02:10:09) to then say, you know what, I'm going to (02:10:11) do it. Similarly to Putin thinking, (02:10:14) okay, I'm going to invade Ukraine. I'm (02:10:16) going to do it. And and in some cases (02:10:18) from these leaders, maybe it's a a (02:10:19) legacy issue for them, right? It's it's (02:10:22) one where they see a retaking territory (02:10:26) that they see as theirs is something (02:10:29) that they want to leave as a defining (02:10:30) legacy. And you can have situations (02:10:32) where then authoritarian leaders engage (02:10:34) in some pretty risk-taking behavior. And (02:10:36) even if it looks dumb to us, like it's (02:10:39) it's very clear that Russia's invasion (02:10:42) of Ukraine has net made Russia (02:10:45) economically, polit politically, and (02:10:46) militarily weaker and has strengthened (02:10:48) NATO, he still did it. And I think (02:10:52) that's that would be concerning. (02:10:54) >> Yeah. So it seems like there are just so (02:10:57) many parallels to cold war era. um kind (02:11:00) of building up of nuclear stock piles (02:11:03) and um nuclear uh technology (02:11:08) is and and and it seems like China has (02:11:13) these interests um that could be really (02:11:17) benefited by having this at least (02:11:20) temporary if not indefinite strategic (02:11:22) advantage uh militarily. Are the US and (02:11:26) China currently in a race to kind of (02:11:29) automate um and build AI into their (02:11:31) military systems? (02:11:33) >> Well, they're certainly in a competition (02:11:36) >> militarily to maintain an advantage over (02:11:38) the other and to adopt AI. Sometimes (02:11:41) it's characterized as an arms race. (02:11:43) Yeah, (02:11:43) >> it is clearly not an arms race if you (02:11:46) use that term in a precise academic way. (02:11:48) and the way that academics talk about (02:11:50) arms races and we have historical (02:11:51) examples of the nuclear arms race, the (02:11:53) arms race among battleship construction (02:11:55) in the early 20th century (02:11:58) where academics are finding sort of (02:12:00) above normal levels of defense spending (02:12:03) that's driven by two countries competing (02:12:05) against another. So it's kind of hard to (02:12:08) pin down numbers of AI spending inside (02:12:11) militaries. Um Bloomberg had done some (02:12:13) really interesting work a couple years (02:12:14) ago pouring through the defense (02:12:18) department budget to try to figure this (02:12:19) out like how much is defense department (02:12:21) spending on artificial intelligence and (02:12:23) they don't have a good answer (02:12:24) internally. Do doesn't interestingly and (02:12:26) bloom came up about 1%. (02:12:29) >> That's not an arms race. (02:12:30) >> That's not even a priority, (02:12:32) >> right? Like when you have see senior (02:12:35) defense leaders saying oh AI is the (02:12:36) number one priority. No it's not. your (02:12:38) joint strike fighter is your number one (02:12:39) priority when they look at what you're (02:12:41) actually doing. So I think it's clearly (02:12:45) not an arms race. I do think that there (02:12:47) is a an adoption competition (02:12:50) in AI of how do militaries find ways to (02:12:54) import the technology. both the US and (02:12:56) China are going to have access to (02:12:56) roughly the same level of AI technology (02:12:59) and whether opening eyes a couple months (02:13:01) ahead of Deep Seek like just doesn't (02:13:02) matter because let's say that there's (02:13:06) you know a gap of 6 to 12 months between (02:13:10) leading labs in the United States and (02:13:12) China well if the US military is (02:13:17) charitably five years behind the (02:13:19) frontier of AI in adoption maybe more (02:13:21) like 10 like that oneear advantage means (02:13:24) nothing it's really contest of of (02:13:26) adoption. Um, but it becomes the most (02:13:29) critical thing is figure out how do you (02:13:31) use this technology in a way that's (02:13:33) constructive that actually advantages (02:13:35) war fighters. And I think that's a a (02:13:38) tricky one. Has a lot to do with how (02:13:39) militaries organize themselves and (02:13:41) create the right incentives internally (02:13:42) for experimentation and reorganization. (02:13:46) Um, and I think it's just not actually (02:13:48) clear who is an advantage there. (02:13:50) >> Okay. (02:13:53) you've argued that military power (02:13:55) depends kind of less on really excellent (02:13:58) algorithms um and more on data, compute, (02:14:02) talent and institutions. Um this is a (02:14:05) bit this is a bit of a kind of (02:14:07) multi-layer question, but uh between the (02:14:10) US and China, where does each side (02:14:12) actually hold an edge? Um and I guess (02:14:15) I'm interested in kind of like currently (02:14:17) and kind of where things are going, (02:14:20) >> right? So the question that I um sort of (02:14:22) had at the outset of four battlegrounds (02:14:24) my most recent book was (02:14:26) >> the US and China are in this (02:14:27) competition. Well what does it mean to (02:14:30) compete in artificial intelligence? Um (02:14:34) what are the thing that you're competing (02:14:35) over exactly? What are the sources of (02:14:36) national advantage? So if you compare it (02:14:38) to the industrial revolution we saw that (02:14:40) during the industrial revolution nations (02:14:42) rose and fell on the global stage based (02:14:43) on how rapidly they industrialized but (02:14:46) also the key metrics of power changed. (02:14:48) So now you want to count um (02:14:50) manufacturing output and coal and steel (02:14:53) production and oil became a geostrategic (02:14:55) resource worth fighting over. So what is (02:14:57) that in an age of AI? Of course at the (02:15:00) technical level there are three core (02:15:02) technical inputs into AI systems (02:15:04) algorithms data and computing power or (02:15:07) compute. When you're looking for a (02:15:09) competitive advantage there it's not (02:15:11) that algorithms don't matter. I don't (02:15:13) think that they are likely to lead to a (02:15:14) competitive advantage because the (02:15:16) algorithms are really hard to keep (02:15:17) secret and they proliferate pretty (02:15:19) quickly. Now that could change. You (02:15:21) could end up in a world where the (02:15:22) leading labs really go dark in terms of (02:15:26) sharing their secret sauce and they stop (02:15:28) publishing papers and they stop sort of (02:15:30) sharing even slightly sanitized details (02:15:33) about what they're doing and they're (02:15:35) much more like um pharmaceutical (02:15:37) companies maybe that like really work (02:15:39) hard to kind of keep trade secrets. that (02:15:42) might happen naturally as AI becomes (02:15:45) more valuable. Um, but right now we (02:15:47) still see like an awful lot of (02:15:48) transparency from the leading AI labs. (02:15:50) And so I don't know that that's like a (02:15:51) source of competitive advantage today. (02:15:54) Um, so sort of data and computing power (02:15:58) are certainly areas of competitive (02:16:00) advantage. I think I've certainly seen (02:16:02) people make an argument that China has (02:16:03) an advantage in data uh because it's an (02:16:06) authoritarian system. the government is (02:16:08) putting in place all these surveillance (02:16:10) measures to collect data on their (02:16:11) citizens. It's true the Chinese (02:16:13) Communist Party is doing a lot of really (02:16:14) dystopian scary things to gather (02:16:17) biometric data and genetic data on their (02:16:19) citizens and there's um surveillance (02:16:21) cameras everywhere. They're increasingly (02:16:23) incorporating AI. I don't know that that (02:16:26) nets out to a data advantage from China (02:16:29) in particular because um what matters (02:16:32) for companies is like the US companies (02:16:36) are not confined to the US population. (02:16:38) Sometimes people oh China's got a bigger (02:16:39) population. Well, US companies have (02:16:41) global reach. Facebook and YouTube have (02:16:44) over two billion users each. (02:16:46) >> Um in fact, US companies have done a lot (02:16:48) better job of global penetration than (02:16:49) Chinese companies today with some (02:16:51) exceptions like like Tik Tok. Um and and (02:16:55) the the sort of collection that Chinese (02:16:58) government is doing doesn't always (02:17:00) translate into advantages for companies. (02:17:02) Um obviously some companies like if (02:17:04) you're a facial recognition company, (02:17:06) you're going to have advantages over (02:17:08) over a US company. Um because we don't (02:17:11) have the same degree of public (02:17:12) surveillance here in the United States. (02:17:14) Not to be clear, right? Um but (02:17:18) um but the Chinese government has put in (02:17:21) place protections on consumer data (02:17:22) privacy and because they don't really (02:17:24) want Chinese companies to assume spying (02:17:26) powers that the government has and um (02:17:29) they do are concerned about sort of (02:17:31) keeping some of these big tech companies (02:17:33) in check. So I don't think it's like as (02:17:34) I think it's more of a of a wash (02:17:36) probably on data. A lot of it has to do (02:17:38) with how to use data better. Um, but (02:17:40) that's a that's a really interesting (02:17:41) contested area and I've heard different (02:17:42) arguments on kind of different sides of (02:17:44) this. I think computing power is very (02:17:46) clear to us is a massive advantage here. (02:17:49) Um, partly because of Nvidia. A lot of (02:17:53) of course the the technical choke points (02:17:55) actually exist below the stack earlier (02:17:58) in the supply chain at the semiconductor (02:18:01) manufacturing equipment at software (02:18:02) that's used to make the most advanced (02:18:05) chips at fabs in TSMC and which is kind (02:18:08) of wild because the idea that these most (02:18:12) advanced chips in the world are built at (02:18:14) this island 100 miles off the coast of (02:18:16) China that the Chinese Communist Party (02:18:18) has pledged to absorb by force if (02:18:19) necessary like none of that's good but (02:18:22) because they're relying US technology. (02:18:23) US has put in place extr territorial (02:18:26) export controls on these chips. So right (02:18:29) now if a chip is designed in China by a (02:18:33) Chinese chip design company manufactured (02:18:35) at TSMC and then shipped back to China (02:18:38) for use inside China that's banned by US (02:18:42) export controls above a certain (02:18:45) threshold in terms of chip performance. (02:18:46) And there have been a couple incidents (02:18:48) where maybe that's a little bit leaky, (02:18:49) but I think if the US can crack down on (02:18:50) export control enforcement, that's a (02:18:52) clear competitive edge for the United (02:18:54) States. (02:18:55) >> Yeah. (02:18:55) >> Not if we just sell those chips to (02:18:58) China, um which we've seen some recent (02:19:01) moves of the Trump administration to do, (02:19:02) but I think it's it's a potential (02:19:03) advantage there for the US to harness. (02:19:05) Um people also matter a lot. I think (02:19:08) this is a yes if you look at the numbers (02:19:11) China is training more AI scientists and (02:19:15) engineers and in fact China produces (02:19:18) more of the top AI scientists engineers (02:19:20) than any other country if you look at um (02:19:23) like like research publications in in (02:19:26) conferences for example um but those top (02:19:30) Chinese scientists don't stay in China (02:19:31) many of them come to the United States (02:19:34) and they come to um US universities to (02:19:37) do graduates who study many of them stay (02:19:38) here afterwards. I think that's a huge (02:19:41) sort of latent strength of the United (02:19:42) States that the best scientists and (02:19:44) engineers around the world they want to (02:19:46) come to the US to study and to work in (02:19:48) companies and that's a place where the (02:19:50) US immigration policy is just shooting (02:19:52) ourselves in the foot because that is (02:19:53) like very much one where if we constrain (02:19:57) ourselves to US generated engineers we (02:20:00) will lose this competition with China (02:20:02) but if we're drawing on the best (02:20:03) scientists from the world (02:20:06) the US has has an unparalleled advantage (02:20:08) here because people don't want to (02:20:10) engineers they don't want to go to China (02:20:12) to to live there. Um and then I think (02:20:15) the the last maybe key piece of this is (02:20:17) what we're talking earlier about (02:20:18) institutions of this sort of question of (02:20:20) adoption. Um and I think this is more of (02:20:23) probably a level playing field between (02:20:24) the US and China of whether it's in the (02:20:27) military or or other areas. (02:20:30) um you know how how do we find ways to (02:20:36) adopt AI (02:20:38) that are beneficial across society? Um (02:20:42) and I I don't I think it's kind of an (02:20:44) even question. We have very different (02:20:45) political systems. Um some cases that (02:20:48) benefits China because they can move (02:20:51) faster in many ways. they don't have the (02:20:53) same kind of public give and take that (02:20:55) we have between civil society and the (02:20:57) media and the government and and the (02:21:00) private sector and I do think at least (02:21:02) the better outcomes here in the United (02:21:04) States. Um so I think that's that's very (02:21:06) much an open question but I do think the (02:21:08) US has like huge competitive advantages (02:21:10) in computing hardware and talent if we (02:21:13) harness them. Yeah. (02:21:14) >> And there's more we have to do there. (02:21:17) >> Do you have a prediction of which of (02:21:19) those factors will prove most decisive? (02:21:22) If the current trends that we're seeing (02:21:24) hold, (02:21:27) I think that the advantage that US labs (02:21:31) have over Chinese labs is effectively (02:21:35) negligible (02:21:36) in terms of societal adoption. The (02:21:39) technology proliferates way too quickly. (02:21:41) And what's going to matter more is how (02:21:43) societies adopt AI to increase (02:21:46) productivity, to increase welfare. Um (02:21:51) I I know sometimes (02:21:54) um (02:21:55) people can sort of look at authoritarian (02:21:57) systems and maybe be a little bit (02:22:00) envious because they can move faster. A (02:22:02) lot of times I think ear you saw for (02:22:04) example early in co that the Chinese (02:22:07) Communist Party was way more effective (02:22:08) at like closing everything nationwide (02:22:10) and containing the spread once they got (02:22:12) serious about it in a way that in the US (02:22:14) you had like protests and people didn't (02:22:16) want to listen. is like a much much more (02:22:18) thorny problem here. I don't think in (02:22:20) the long run that authoritarian systems (02:22:22) are better than democratic ones. (02:22:23) Democratic ones are messier, but (02:22:25) authoritarian systems end up being very (02:22:27) brittle because it ends up being (02:22:29) whatever answer the top leadership (02:22:32) wants, right? (02:22:33) >> And they don't always get it right. And (02:22:34) I think we could saw actually with with (02:22:36) CO is a great example here. Over time, (02:22:38) China's zero co policy was um (02:22:40) counterproductive and and destructive. (02:22:43) And I do think that this give and take (02:22:45) is a lot messier. (02:22:46) um in democratic societies, but it's (02:22:49) probably in the long run going to lead (02:22:50) to better outcomes. But it does depend a (02:22:52) lot on like how do we manage this (02:22:54) transition? So if AI has really (02:22:57) disruptive effects on jobs and the labor (02:23:00) market, like are we going to take care (02:23:02) of people? And I don't know that we've (02:23:04) done a great job in the United States of (02:23:06) of managing the effects over the last (02:23:09) several decades of job disruption from (02:23:12) globalization and automation. Um, and (02:23:15) it's led to a lot of discontent today. (02:23:17) And I remember in the '90s debates about (02:23:19) globalization and sort of, you know, at (02:23:21) the time what we heard from leadership (02:23:22) was, well, sure, maybe some jobs will be (02:23:24) disrupted, move overseas, but we'll (02:23:26) we'll help people reskill and get new (02:23:28) jobs. It we don't really put in place (02:23:31) those social safety nets to the extent (02:23:33) that I think are necessary. Um, and I (02:23:35) think that's a going to be essential to (02:23:37) managing this transition effectively. (02:23:39) >> Yeah. How how would you characterize (02:23:41) China on that front? Well, the Chinese (02:23:43) Communist Party um and this the current (02:23:46) iteration under Shei um cares immensely (02:23:50) about political stability and so we've (02:23:52) seen just incredibly economic growth (02:23:54) over the last really unprecedented in (02:23:56) human history. Economic growth in China (02:23:58) over the last several decades um pulling (02:24:02) you know hundreds of millions of people (02:24:03) out of poverty um increasing welfare (02:24:06) across their society. In previous (02:24:09) leaders, the party had been willing to (02:24:13) have some degree of political openness (02:24:16) in small degrees um and prioritize (02:24:19) economic growth. I've seen think we've (02:24:21) seen under she that flips and she has (02:24:23) really prioritized (02:24:25) um control, political control, and been (02:24:28) willing to sacrifice some economic (02:24:30) growth. I think whether China is able to (02:24:32) navigate that effectively is a really (02:24:35) open question. So there's this idea of (02:24:37) the authoritarian dilemma that basically (02:24:40) authoritarian governments have this (02:24:42) choice of either they can allow (02:24:46) um openness and free economic trade and (02:24:48) open information and allow economic (02:24:50) growth but that will lead over time to (02:24:52) greater political openness as well. Um (02:24:55) or they can crack down (02:24:58) and they go the way of North Korea (02:25:00) >> and you get political control but you (02:25:02) you stifle economic growth. Um, and (02:25:04) actually like North and South Korea are (02:25:06) sort of like a really interesting kind (02:25:07) of examples of this. Um, and so I think (02:25:10) how today China's been able to navigate (02:25:12) that extremely effectively. They've been (02:25:13) able to basically have their cake and (02:25:14) eat it too and maintain political (02:25:16) control and economic growth. Um, and I (02:25:19) think how China navigates that is going (02:25:20) to be really critical uh to sort of (02:25:23) answering this question of whether China (02:25:25) is able to continue to grow as an (02:25:26) economic power. um whether they, you (02:25:29) know, eclipse the United States or come (02:25:31) closer to it or they kind of end up some (02:25:32) people argue we sort of hit peak China (02:25:34) in terms of their relative power to the (02:25:36) United States. Um and at least the (02:25:39) current leadership seems to be really (02:25:40) prioritizing political control over (02:25:42) growth. (02:25:43) >> Yeah, I I I want to ask about this (02:25:46) comparison to North Korea. Um, North (02:25:49) Korea has done a depressingly good job (02:25:54) at um its kind of authoritarian um (02:25:59) ruling of people. Uh, (02:26:02) China (02:26:03) could in theory potentially do even more (02:26:08) if it wanted to because of (02:26:11) this trajectory toward um really really (02:26:15) really capable AI systems in kind of (02:26:19) surveillance and censorship and (02:26:20) repression. Um those could get much (02:26:23) cheaper and more effective. Um, does (02:26:25) that seem like a plausible route China (02:26:28) could go where I guess I'm interested in (02:26:31) particular, um, one of our past guest, (02:26:33) Tom Davidson, argued that AI could kind (02:26:36) of enable new forms of authoritarianism (02:26:38) that can actually be like really stably (02:26:41) locked in. Um, yeah. Does that does that (02:26:43) seem plausible? (02:26:46) Well, I think it's very clear that China (02:26:48) is already building this very dystopian (02:26:52) technosurveillance system within the (02:26:54) country to monitor and surveil and (02:26:56) control its population and the trends (02:27:00) sort of are in the direction of (02:27:02) technology enabling even greater (02:27:04) authoritarian control. The most extreme (02:27:06) version of this is in Shing Jong where (02:27:09) China has had this really intense (02:27:10) crackdown on the weaguer population (02:27:13) there and you had sort of these (02:27:15) concentric circles of control from (02:27:18) actual um prisons that many weaguers are (02:27:21) imprisoned to for those that are (02:27:23) released they're on like a sort of (02:27:25) degree of you know house arrest or even (02:27:27) within kind of the cities there are (02:27:29) physical checkpoints police checkpoints (02:27:31) that are doing biometric scanning that (02:27:33) are monitoring license plate readers to (02:27:35) track where people go. Knives have QR (02:27:38) codes on them to track knives. Like the (02:27:40) technology is allowing a lot of (02:27:42) monitoring of surveillance. It wouldn't (02:27:43) be possible. And then we've seen sort of (02:27:46) this Shing Jongriification of the rest (02:27:48) of China where there are um surveillance (02:27:51) cameras that are widely deployed (02:27:53) throughout the country. It's when I was (02:27:55) in um Beijing last I was sort of really (02:27:58) struck by this that I've heard about all (02:27:59) the surveillance cameras and I didn't (02:28:02) really appreciate u just how omnipresent (02:28:05) they are until was physically there that (02:28:07) you'd be on like a street corner they're (02:28:09) in every street corner there's multiple (02:28:11) of them pointing in different directions (02:28:12) halfway down the street they're very (02:28:14) obvious because of course (02:28:16) >> the party wants you to know that they're (02:28:18) watching (02:28:19) >> right (02:28:19) >> and so in Tianaan Square I you know I (02:28:21) sort of roughly estimated maybe like 200 (02:28:23) cameras (02:28:24) across TM Square. Now Tman Square is (02:28:25) huge to be fair. It's massive. But I (02:28:28) realized that the goal of the party is (02:28:31) not even necessarily to like okay if (02:28:35) there's another, you know, massive (02:28:37) protest in Tianaan Square to capture (02:28:40) everyone there and their faces and who's (02:28:42) the leaders. It's to prevent the protest (02:28:43) from ever even happening in the first (02:28:44) place. Protests would never even make it (02:28:46) there because there's so much monitoring (02:28:48) of there. And so it's down to like (02:28:50) things that almost seem absurd in the (02:28:53) degree of control. Um, China has this (02:28:55) social credit system that is a little (02:28:57) bit sort of a character caricature in (02:28:59) how people talk about it. It's not one (02:29:01) system. It's sort of a whole bunch of (02:29:03) different credit scores and blacklist. (02:29:05) Some of which are for financial credit, (02:29:06) but some of them are sort of more social (02:29:08) in nature. So someone might get score (02:29:11) based on like whether they separate (02:29:13) their trash and recycling like down to (02:29:16) that level of control over what people (02:29:17) do. And of course, technology enables (02:29:20) really high degrees of control. So they (02:29:22) can do things like, okay, if you buy use (02:29:24) a credit card to buy a hotel room, we (02:29:27) don't need a credit card. Maybe we just (02:29:28) monitor what hotels are doing. Buy a (02:29:29) hotel room in the city in which you (02:29:30) live. (02:29:33) What are you up to? (02:29:34) >> God. (02:29:35) >> Right. And so they they're they can get (02:29:37) that data. Maybe they could look into (02:29:38) this and you know, what's this person (02:29:40) doing? um things like if if two people (02:29:43) are in an internet cafe at the same time (02:29:46) every day like that's that's an (02:29:49) interesting coincidence like what's (02:29:50) going on there and so I think that over (02:29:52) time that's going to become even more (02:29:54) China has put in place this intense (02:29:56) degree of control over the internet (02:29:59) and in the '9s people sort of thought (02:30:01) that was not going to be possible Bill (02:30:03) Clinton had famously compared (02:30:05) controlling the internet to nailing (02:30:06) jello to the wall well they did it they (02:30:09) did it and and it's Not that you can't (02:30:11) get outside of these the great firewall (02:30:13) within China through VPNs, but it's it's (02:30:15) hard and VPNs are kind of spotty and (02:30:18) people just don't do it. And so there's (02:30:19) so much control and so much censorship (02:30:21) inside China and propaganda by the party (02:30:23) that the party's been extremely (02:30:25) effective at controlling the information (02:30:26) environment inside China. And now (02:30:28) they're using technology to do the same (02:30:30) to physical space to control people's (02:30:32) movements and where they go and what (02:30:34) they do. And I think technology will (02:30:37) enable an unprecedented degree of (02:30:40) authoritarian control. (02:30:42) To what extent that allows this kind of (02:30:45) lock in that you're concerned about, I I (02:30:48) don't know, but I certainly think the (02:30:50) trend lines are very worrying in terms (02:30:52) of enabling greater technological (02:30:54) control. And by the way, a lot of (02:30:55) technology is spreading outside of China (02:30:56) >> to other countries as well, (02:30:58) >> right? (02:30:58) >> And we're starting to see the spread of (02:30:59) this sort of global spread of more (02:31:01) techno authoritarianism. (02:31:03) >> O, (02:31:04) >> it's all bad. This is the challenge is a (02:31:06) lot of these topics are just like it's (02:31:07) like scary things and like what's and I (02:31:10) um (02:31:10) >> I think we're doing a great job of of (02:31:12) sort of hiring to highlight like what (02:31:14) what's the other side? Is there a (02:31:15) positive side? And there are, but it's (02:31:17) just like none of it (02:31:19) >> makes you sleep easy at night. (02:31:21) >> No, no, (02:31:22) >> no. (02:31:23) >> Let's talk a bit about the policy tools (02:31:24) and governance ideas that could make uh (02:31:27) kind of AI and autonomy safer. Um, some (02:31:32) people want a global ban on autonomous (02:31:34) weapon systems entirely. Uh, what do (02:31:36) what do you think of that as a proposal? (02:31:39) >> I mean, I I understand the sentiment. I (02:31:41) think it's just not realistic. And um I (02:31:45) I right now we're in a path where there (02:31:47) is no global regulation. There's no (02:31:50) rules of the road for how militaries in (02:31:52) corporate autonomy. I don't think that's (02:31:53) a good position to be in. I think it's (02:31:55) dangerous. I think a sort of anything (02:31:57) goes approach is probably could take us (02:31:59) in a in a dangerous place. So I would (02:32:01) like to see some guard rails put in (02:32:03) place, but I think that they have to be (02:32:05) things that we think militaries are (02:32:06) actually going to get on board with and (02:32:08) none of the major military powers have (02:32:10) said that they support ban. I think you (02:32:13) might be able to envision more narrow (02:32:15) tailored kinds of restrictions that (02:32:17) might be achievable. One is surrounding (02:32:19) maintaining human control over nuclear (02:32:20) weapons as we've talked about. Another (02:32:22) one that I think is actually kind of (02:32:24) interesting and worth exploring would be (02:32:26) a ban on anti-personnel autonomous (02:32:28) weapons. So, it's sort of driving a (02:32:30) distinction between those that what the (02:32:32) military call anti-material autonomous (02:32:34) weapons, those that target physical (02:32:36) objects, tanks, airplanes, radar, (02:32:39) submarine, ships versus targeting people (02:32:43) directly. And I think that that's there (02:32:46) might be traction there. You might have (02:32:47) more possibility for a whole bunch of (02:32:49) reasons. One is it's just less (02:32:51) militarily valuable to use autonomous (02:32:53) weapons to go after people. And there's (02:32:55) probably a case to be made that you (02:32:57) might need autonomous weapons in some (02:32:59) kinds of really highintensity conflicts (02:33:02) if you have say a swarm of um drones (02:33:04) attacking enemy radars and their (02:33:06) communications are jammed. there's just (02:33:08) like there's not a good way to execute (02:33:11) that problem to take down this um (02:33:13) integrated air and missile defense (02:33:15) system without using autonomous weapons (02:33:18) and keeping a human in the loop. Well, (02:33:20) people don't move that fast, right? Like (02:33:23) outrunning, you know, bullets has not (02:33:25) been effective since the age of the (02:33:27) machine gun. And so, you know, I think (02:33:29) you could make a case that going after (02:33:30) people, you can keep humans in the loop. (02:33:32) Now, I think we could see on the (02:33:33) battlefield in Ukraine (02:33:36) the case for using at least in a (02:33:39) military sense anti-personnel autonomous (02:33:40) weapons because you do have a lot of (02:33:41) drone uh uh jamming on the front lines (02:33:44) of drones and people using drones to (02:33:46) target individuals. Um but it's not as (02:33:51) critical necessarily for militaries. And (02:33:53) then I think that the ick factor is a (02:33:56) lot higher when going after people. Um, (02:33:58) and it's not just that like, oh, it (02:34:01) feels uncomfortable, but like this this (02:34:03) really does matter in terms of getting (02:34:05) public support, but it's not just that (02:34:06) like I don't like that. I think you can (02:34:08) make a very reasonable argument that if (02:34:11) you are an enemy combatant or civilian (02:34:15) that is in a situation where autonomous (02:34:16) weapons are targeting you and they're (02:34:18) going after a physical object, you can (02:34:20) escape being targeted. You can climb out (02:34:22) of that tank and run away. You can't (02:34:25) stop being a person. (02:34:27) And so if the autonomous weapon goes ary (02:34:30) or you want to surrender or it's having (02:34:33) a mistake, I think like that having (02:34:36) weapons that sort of target people (02:34:37) doesn't it it could be really risky and (02:34:40) cause quite a bit of harm and it could (02:34:42) take you on a path towards greater (02:34:43) civilian harm. Maybe you have um (02:34:46) countries deploying these swarms of (02:34:48) anti-personnel drones and they're like, (02:34:50) "Well, well, these people are all enemy (02:34:52) cabbands." And they're basing it on, you (02:34:55) know, some algorithm that looks at their (02:34:58) cell phone connections or their (02:35:00) geoloccation data or their social media (02:35:02) and says, "Well, we've labeled them that (02:35:03) they're all affiliated at least greater (02:35:04) civilian harm." I think there's a lot of (02:35:06) ways in which you could sort of say that (02:35:08) that's much more troubling and that (02:35:09) might be a place for countries to (02:35:11) explore. Um, so I think like to me I (02:35:15) would focus more on like what's what's (02:35:17) achievable here and and I think that (02:35:20) part of the challenges here some of the (02:35:22) people who have pushed for a ban (02:35:24) autonomous weapons are coming out of the (02:35:25) humanitarian disarmament community where (02:35:27) there were a lot of successes in the '9s (02:35:29) after the end of the cold war with bans (02:35:30) on landmines cluster munitions and um (02:35:34) we're in a very different geopolitical (02:35:36) environment now particularly since (02:35:38) Russia's invasion of Ukraine and if (02:35:40) you're you know countries are now (02:35:41) looking at landmines quite differently. (02:35:44) Um certainly um and sort of if you're if (02:35:47) you're looking at okay we're going to (02:35:49) we're going to disarm say Ukrainians and (02:35:52) say they can't use these weapons that (02:35:53) might help them in the war but you (02:35:55) better have a really strong case to do (02:35:57) that. So I think it's a much we got to (02:35:58) just factor in the political realities (02:36:00) here. (02:36:01) >> Yeah. Yeah. That that makes tons of (02:36:03) sense. Um so those that's kind of a a (02:36:06) broadband and then also narrow bands as (02:36:08) as policy ideas. Um, are there other (02:36:11) types of kind of proposals for major (02:36:14) treaties aiming to govern uh govern (02:36:16) autonomous weapon systems that you think (02:36:18) have promise? (02:36:20) >> Um, I think that something that involves (02:36:23) a sort of um rules of the road between (02:36:27) nations involved in air and situations (02:36:30) where their naval and air forces might (02:36:33) be interacting in close proximity in (02:36:34) crisis could be very helpful here. Um (02:36:37) during the um cold war, the US and (02:36:41) Soviet Union agreed to um the incident (02:36:45) agreement that helped sort of deconlict (02:36:47) their forces so that they were less (02:36:49) likely to get into some of these kinds (02:36:50) of altercations that are that could be (02:36:53) dangerous and destabilizing. It's (02:36:54) something like an autonomous incident (02:36:56) agreement where countries might agree to (02:36:59) to sort of put some rules in the road in (02:37:00) place of like how are autonomous systems (02:37:03) going to behave? How can we communicate (02:37:04) effectively to people? What they're (02:37:05) going to do? And and one might criticize (02:37:08) it. Well, these aren't going to hold in (02:37:10) wartime. All the rules are going to come (02:37:12) off. It's not designed to solve that (02:37:13) problem. It's not solve a wartime (02:37:15) problem. It's designed to solve a peace (02:37:17) time problem in these militarized (02:37:19) disputes where countries don't really (02:37:21) want to necessarily go to war, but (02:37:23) they're engaging in some level of (02:37:25) brickmanship. So, that might be (02:37:26) something worth exploring. I think (02:37:27) there's value there. And I think the (02:37:29) last thing would be maybe um (02:37:32) promulgating global norms about better (02:37:36) safety and testing evaluation for (02:37:39) autonomous systems or particularly in (02:37:41) weapon systems. Um a good and because (02:37:44) like I don't think that accidents really (02:37:46) benefit anyone here. And so if some (02:37:48) country develops some autonomous weapon (02:37:52) and then it has an accident and kills a (02:37:54) bunch of people, you know, I don't think (02:37:56) that's sort of like in anyone's (02:37:57) interest. We're finding ways to improve (02:38:00) the reliability of systems and reduce (02:38:02) access to make countries more conscious (02:38:04) of safety. I think beneficial and an (02:38:06) analogy here would be some of the work (02:38:07) that the US does in promagating norms on (02:38:11) legal weapons reviews. So the US and (02:38:13) other many other western militaries do (02:38:16) um legal reviews of new weapons to (02:38:18) ensure they comply with the law of war (02:38:20) with existing treaties and the Geneva (02:38:22) conventions. And not all countries do (02:38:25) this, but the US and other countries (02:38:26) have been very active in sort of trying (02:38:29) to sort of spread this knowledge to (02:38:32) others and encourage other countries to (02:38:34) do this. And I think there's merit in (02:38:36) doing the same on test and evaluation (02:38:38) for some of these AI enabled systems and (02:38:40) militaries. Nice. Yeah, I like that. (02:38:43) That is a thing that a country that (02:38:46) wanted to be safety oriented could do (02:38:48) kind of unilaterally to reduce risk that (02:38:50) doesn't require rivals to cooperate. Are (02:38:53) there other things like that or is that (02:38:55) kind of a a particularly good example? (02:38:59) >> I think you could argue that there are (02:39:01) similar analogies in the risk (02:39:05) surrounding loss of control of much like (02:39:07) highly advanced AI systems that um for (02:39:11) example just having more awareness of (02:39:14) those risks demonstrating you know sort (02:39:17) of leading labs demonstrating the sort (02:39:18) of protections that they're putting in (02:39:19) place to be more safe. um to guard (02:39:22) against agentic systems doing something (02:39:25) squirrely and engaging in deceptive (02:39:27) behavior, self-exfiltrating. Those are (02:39:29) like good norms to promulgate to others (02:39:31) particularly as we see not only the (02:39:34) frontier of AI advance but of course the (02:39:36) technology proliferates super super (02:39:38) rapidly right and so today's frontier (02:39:41) systems is tomorrow's commodity system (02:39:44) and um spreading those norms similarly (02:39:46) spreading norms around protections (02:39:49) against biological hazards as AI becomes (02:39:52) more capable I think would be areas (02:39:53) where there's just a lot of value in (02:39:58) helping to ensure that other people are (02:40:00) engaging in responsible behavior. (02:40:01) >> Mhm. Nice. Are there purely kind of (02:40:05) technical ways to make AI use in warfare (02:40:08) safer? So things like explanability (02:40:12) requirements for any system used in (02:40:14) targeting or early warning or failsafe (02:40:16) mechanisms that kind of default to human (02:40:19) control when sensors disagree. (02:40:22) >> You can imagine lots of things that you (02:40:23) can put in place. (02:40:24) >> Um (02:40:25) >> are there any you're particularly (02:40:26) excited about? So I a couple um I like (02:40:30) the idea so taking the analogy from (02:40:33) stock trading and look at looking at (02:40:34) things like flash crashes y (02:40:36) >> and trying to avoid a similar sort of (02:40:38) scenario. I like the idea of having (02:40:39) human circuit breakers in AI systems. So (02:40:42) even if we end up in a world where maybe (02:40:44) there are autonomous weapons in in (02:40:46) physical autonomous weapons or (02:40:48) autonomous cyber weapons that there's (02:40:50) some bounds on their behavior such that (02:40:52) if they go ary there are limits in how (02:40:55) bad it gets before a human has to take (02:40:58) some positive action for things to (02:41:00) continue. And so I think there's like (02:41:02) there's like things that you would take (02:41:03) to improve just reliability and reduce (02:41:05) the risk of accidents. But I think we (02:41:06) should assume that there will be (02:41:08) accidents. Bad things will happen. And (02:41:11) then how do we put in place these (02:41:13) boundaries in their behavior or human (02:41:15) circuit breakers so things don't get too (02:41:18) badly out of control? Um that's (02:41:20) something that seems like something that (02:41:22) I would like to see militaries do. (02:41:24) >> Nice. Yeah. Um I guess just before we (02:41:27) move on, are there any any things in (02:41:29) this kind of solutions or policy (02:41:32) areas um that you really want people to (02:41:36) know about and consider that are maybe (02:41:38) underrated? (02:41:41) um in the military space specifically or (02:41:43) just broadly? (02:41:44) >> I think military. Yeah. (02:41:46) >> So there's this interesting question. So (02:41:48) the US has a policy now um maintaining (02:41:51) human control over decisions relating to (02:41:54) nuclear use. (02:41:56) It's really uncertain what that means. I (02:41:59) would love to see work inside the US (02:42:03) military to better characterize and some (02:42:05) of that work might be classified. It's (02:42:06) not public. That's fine because it's (02:42:08) very sensitive information. But what (02:42:10) does that mean? What's inbounds? What's (02:42:12) out? What is sort of an acceptable use (02:42:13) case look like? What is it not? And over (02:42:16) time, we might get some rules and (02:42:18) policies and practices in place to more (02:42:21) accurately characterize what good uses (02:42:23) of AI and automation are in the nuclear (02:42:25) enterprise and what aren't. So maybe (02:42:26) give an um analogy here. There's this (02:42:29) concept today nuclear operations of dual (02:42:32) phenomenology for early warning systems (02:42:34) that if we are looking at a a potential (02:42:37) missile launch against the United States (02:42:39) that we want two completely independent (02:42:41) ways of sensing that missile launch. One (02:42:45) could be for example satellites and (02:42:46) another one could be radar systems that (02:42:49) we have two totally independent ways of (02:42:50) verifying it. Now one could imagine (02:42:53) something in the AI space of okay let's (02:42:55) say I have some AI system churning (02:42:57) through data like we were talking about (02:42:59) earlier and it's it's you know coming up (02:43:01) with some conclusion not only that we (02:43:03) want some degree of auditability (02:43:05) explanability with the system of what is (02:43:07) it doing but also maybe I have a (02:43:08) completely independent way a different (02:43:10) algorithm trained on a different data (02:43:12) set that's that's doing something (02:43:14) similar and that I can compare those two (02:43:17) and there might be some merit in really (02:43:21) high-risisk applications in having (02:43:23) something like that. (02:43:24) >> Cool. Cool. Okay, we have time for one (02:43:26) last question. Um, we haven't talked (02:43:29) very much about your experience um, in (02:43:31) the army. I'm wondering if there are any (02:43:36) kind of experiences you've had in that (02:43:38) context that have informed how you think (02:43:41) about kind of autonomous weapons and (02:43:44) what it means for us to be integrating (02:43:46) AI more and more heavily into into war. (02:43:51) So, one maybe uh incident, a personal (02:43:54) example that comes to mind for me (02:43:56) sometimes is an incident that I was in (02:43:58) when I was an Army Ranger deployed in (02:44:01) Afghanistan and I was part of a small (02:44:04) recon team and we were out operating (02:44:06) among the mountains and there were just (02:44:09) three of us that had gone out on to (02:44:12) patrol this sort of ridge along this (02:44:14) line and we saw an Afghan man (02:44:17) approaching us along this ridge line (02:44:20) coming in our general direction. And (02:44:21) from a distance, we couldn't tell if, (02:44:23) you know, he was armed or not. Maybe he (02:44:25) had a weapon under his cloak, if he had (02:44:27) a radio on him. We couldn't certainly (02:44:28) see. Maybe there were others like (02:44:31) nearby. Maybe he was scouting for (02:44:32) somebody. Maybe he's just a goater. We (02:44:34) we couldn't tell. So, he we lost sight (02:44:38) of him and I was a little bit concerned (02:44:40) that he might be sort of coming up (02:44:42) behind us. So, I went and maneuvered in (02:44:44) a position where I could get eyes on (02:44:46) him. And I ended up in a sort of this um (02:44:49) rock position where I was up above him (02:44:51) looking down through this crack in the (02:44:53) rocks and he had his back to me. (02:44:55) >> And I was pretty close to be honest. I (02:44:57) could hear him pretty clearly and he was (02:45:00) talking and I couldn't tell, you know, I (02:45:03) don't I don't speak posture. I could say (02:45:05) stop and that was about it. And so I (02:45:06) couldn't tell what he was saying and I (02:45:08) didn't know if he was talking to some (02:45:10) other people that might be nearby that (02:45:12) were out of sight that I couldn't see or (02:45:14) he was talking on a radio for example (02:45:16) maybe relaying information and other (02:45:17) fighters were going to come attack us (02:45:19) and we' actually seen that exact (02:45:21) scenario happen previously where (02:45:23) somebody had come looking like they were (02:45:26) hurting goats as cover but they had a (02:45:28) radio on them and reporting information. (02:45:31) And so I settled into a position um with (02:45:34) my sniper rifle and I was ready to shoot (02:45:35) him if I saw that he had a weapon or (02:45:38) there were other fighters and and I sort (02:45:41) of gauge that he was an enemy combatant. (02:45:43) Um and I watched him for a while and I (02:45:47) was looking for some sign and and sort (02:45:49) of in my head I was sort of you know (02:45:51) weighing this decision. Do I shoot this (02:45:54) man? And then I heard him start singing (02:45:59) and it struck me. I just instantly (02:46:01) relaxed because it struck me as like a (02:46:03) bizarre thing to be doing if he was an (02:46:06) enemy fighter reporting information (02:46:08) about it. He probably wasn't singing out (02:46:09) information over the radio and I just (02:46:11) instantly relaxed. I thought, you know, (02:46:12) he's just he's just a goat her out here. (02:46:14) He's talking to himself or his goats and (02:46:16) he's singing. He's enjoying the view. (02:46:18) And I watched him for a little bit (02:46:19) longer and then ended up leaving. (02:46:22) And I think about that sometimes when I (02:46:24) think about the decisions that machines (02:46:25) might make in warfare because I was (02:46:29) relying on kind of this broader (02:46:31) contextual judgment of like would that (02:46:33) be a weird thing for a human to be doing (02:46:36) in that context and would a machine be (02:46:38) able to pick up on that and that (02:46:41) sometimes that sort of broader (02:46:42) understanding the broader context and (02:46:43) relying on judgment is things that a (02:46:45) doesn't necessarily do very good at and (02:46:48) in the big scheme of the war that (02:46:51) decision did not matter. It would not (02:46:53) have changed the outcome either way in (02:46:55) terms of the broader US campaign in (02:46:57) Afghanistan, but it mattered a lot to (02:46:59) him (02:47:01) and it mattered to me. And so to me, I (02:47:03) think about that when I think about the (02:47:04) stakes of autonomous weapons that (02:47:07) people's lives are on the line here and (02:47:10) we got to get these decisions right and (02:47:13) how do we find ways to use this (02:47:14) technology that um that doesn't lose our (02:47:18) humanity that doesn't doesn't cause more (02:47:21) suffering as a result. (02:47:23) >> Yeah. Yeah. Thank you for sharing that. (02:47:25) My guest today has been Paul Sharie. (02:47:27) Thank you so much for coming on. (02:47:29) >> Thanks for having me. It's been a great (02:47:30) discussion.