Note to readers: please click the share buttons above
Paul Scharre’s new book on autonomous weapons begins with an account of an incident he experienced while on patrol as a US Army Ranger in Afghanistan in 2004. A young girl of five or six years old herding a couple of goats approached Scharre’s team while they were taking cover in the mountains. As she looped around them, frequently glancing towards them, they realised she had a radio and was reporting their position, acting as a spotter for Taliban fighters.
What should the soldiers do? According to the laws of war, the girl was an enemy combatant whom they were allowed to shoot. If a person is participating in hostilities, regardless of their age, they are a lawful target for engagement.
Scharre and his squad had no doubt that it would have been quite wrong to kill the little girl, and so they moved away and regrouped in a safer area. But what would a machine have done in their place? If it had been programmed to kill enemy combatants lawfully, it would have attacked the child. The incident highlights one of the gravest concerns in the debate over whether to develop autonomous weapon systems – could a robot ever know when it is lawful to kill, but wrong to do so?
‘Army of None’ takes us into the future world of emerging weapons technology – lethal autonomous weapon systems, or ‘killer robots’, driven by artificial intelligence and able to wage war without the need for human command. Although they sound like the plot of a science fiction movie, the technology needed to build autonomous weapons already exists, and such systems are on the point of development.
Advanced weapons currently in service, such as the Brimstone missile, Israel’s Harpy drone, and the US Navy’s Aegis combat system, already show many autonomous features and give a hint of possibilities for the future. Rapid advances in computing technology, robotics, and the development of high definition sensors will soon enable the creation of weapons that are able to ‘think’ for themselves and undertake combat tasks without the need for a commander ‘in the loop’.
Perhaps optimistically, Scharre considers it unlikely that state-based military forces will use truly autonomous weapons without human supervision while waging war. His view is that humans and machines working as teams – ‘centaur warfighters’, named after the mythical half-human, half-horse creature – will provide the most advantageous combination between the precision of automation and the flexibility of human intelligence. However, he rightly highlights concerns that terrorist groups, criminals, and non-state actors could easily develop their own home-built autonomous weapons and use them without safeguards, employing off- the-shelf consumer drone and electronic technology and open source software.
The risk of accidents involving autonomous weapons is another area of concern. A fascinating section of the book documents a series of accidents caused by automated weapon systems such as the Aegis and Patriot air defence systems – and the biases and misunderstandings on the part of their human operators which led to the accidents. Scharre describes how ‘flash crashes’ caused by artificial intelligence systems used by banks and traders have resulted in lightning-fast and unpredictable slumps in prices on international stock markets, and draws frightening parallels with the rapid escalation that could develop during combat involving autonomous weapon systems to highlight the risks to strategic stability which these weapons pose.
‘Army of None’ doesn’t shy away from probing the intensely difficult moral and ethical issues which are the most problematic issue raised by the development of autonomous weapons. The book explains legal and ethical issues relating to the laws of armed combat in clear and simple terms, and presents a range of expert opinions from leading thinkers in the field whom Scharre has interviewed. Disappointingly, however, he argues that the development of precision weapons has reduced casualties in modern warfare and advances the suggestion that autonomous weapon systems may do likewise without exploring the counterview that the benefit of reduced casualties as perceived by governments may increase the risk that war becomes a more ‘thinkable’ and accepted method for resolving conflict.
The book’s finale is a thoughtful analysis of international arms control treaties over the course of history, as part of an examination of steps that could be put in place to curb the spread of autonomous weapon systems. Scharre argues that arms control treaties work well when they apply to weapons with horrific effects, which have limited military utility, and are possessed by a relatively small number of states. Unfortunately, he concludes, an all-out global ban on autonomous weapons is unlikely to work because of their perceived military value and their development by a wide range of militarised nations. He highlights difficulties in defining autonomous weapons in a way which would discriminate them from existing highly automated systems such as Brimstone and Aegis, which possessing states would be unlikely to want to surrender, and in ensuring that any international ban is not flouted during wartime.
However, ‘Army of None’ proposes a number of realistic alternatives to a ban treaty, including a ban on antipersonnel autonomous weapon systems, which might work because such systems would have a low military utility but a high potential for causing harm; a non-legally binding code of conduct to help establish norms for the control of autonomous weapons; and the establishment of a general principle that human judgment must always be involved in war and that there must always be a positive human involvement in lethal force decisions. The book’s conclusion is a powerful call for restraint, and an appeal for states and society to urgently develop an understanding of which uses for autonomous systems are acceptable, and which go too far.
Paul Scharre is well placed to write about autonomous weapon systems. After his tour in the US Army he worked as an analyst at the Pentagon, playing a leading role in developing policies on unmanned and autonomous systems and emerging weapons technologies for the US Department of Defense. He now directs the technology and national security program at the Centre for a New American Security – a centrist think tank specialising in US national security issues.
Scharre’s book tells you much of what you need to know about autonomous weapons, drawing from his own insider knowledge and adding insight from a series of expert interviews. The book does not go into as much detail as some of the specialist technical papers that Scharre has published, but instead provides a readable and accessible basic text on the topic. ‘Army of None’ is an important contribution to the debate on autonomous weapon systems and should definitely be read by anyone who wants a sound introduction to this worrying subject.
