‘Machines set loose to slaughter’: the dangerous rise of military AI

Frank Pasquale – The Guardian Oct 15, 2020

The video is stark. Two menacing men stand next to a white van in a field, holding remote controls. They open the van’s back doors, and the whining sound of quadcopter drones crescendos. They flip a switch, and the drones swarm out like bats from a cave. In a few seconds, we cut to a college classroom. The killer robots flood in through windows and vents. The students scream in terror, trapped inside, as the drones attack with deadly force. The lesson that the film, Slaughterbots, is trying to impart is clear: tiny killer robots are either here or a small technological advance away. Terrorists could easily deploy them. And existing defences are weak or nonexistent.

Some military experts argued that Slaughterbots – which was made by the Future of Life Institute, an organisation researching existential threats to humanity – sensationalised a serious problem, stoking fear where calm reflection was required. But when it comes to the future of war, the line between science fiction and industrial fact is often blurry. The US air force has predicted a future in which “Swat teams will send mechanical insects equipped with video cameras to creep inside a building during a hostage standoff”. One “microsystems collaborative” has already released Octoroach, an “extremely small robot with a camera and radio transmitter that can cover up to 100 metres on the ground”. It is only one of many “biomimetic”, or nature-imitating, weapons that are on the horizon.

Still from Slaughterbots. Click to enlarge

Who knows how many other noxious creatures are now models for avant garde military theorists. A recent novel by PW Singer and August Cole, set in a near future in which the US is at war with China and Russia, presented a kaleidoscopic vision of autonomous drones, lasers and hijacked satellites. The book cannot be written off as a techno-military fantasy: it includes hundreds of footnotes documenting the development of each piece of hardware and software it describes.

Advances in the modelling of robotic killing machines are no less disturbing. A Russian science fiction story from the 60s, Crabs on the Island, described a kind of Hunger Games for AIs, in which robots would battle one another for resources. Losers would be scrapped and winners would spawn, until some evolved to be the best killing machines. When a leading computer scientist mentioned a similar scenario to the US’s Defense Advanced Research Projects Agency (Darpa), calling it a “robot Jurassic Park”, a leader there called it “feasible”. It doesn’t take much reflection to realise that such an experiment has the potential to go wildly out of control. Expense is the chief impediment to a great power experimenting with such potentially destructive machines. Software modelling may eliminate even that barrier, allowing virtual battle-tested simulations to inspire future military investments.

In the past, nation states have come together to prohibit particularly gruesome or terrifying new weapons. By the mid-20th century, international conventions banned biological and chemical weapons. The community of nations has forbidden the use of blinding-laser technology, too. A robust network of NGOs has successfully urged the UN to convene member states to agree to a similar ban on killer robots and other weapons that can act on their own, without direct human control, to destroy a target (also known as lethal autonomous weapon systems, or Laws). And while there has been debate about the definition of such technology, we can all imagine some particularly terrifying kinds of weapons that all states should agree never to make or deploy. A drone that gradually heated enemy soldiers to death would violate international conventions against torture; sonic weapons designed to wreck an enemy’s hearing or balance should merit similar treatment. A country that designed and used such weapons should be exiled from the international community.

In the abstract, we can probably agree that ostracism – and more severe punishment – is also merited for the designers and users of killer robots. The very idea of a machine set loose to slaughter is chilling. And yet some of the world’s largest militaries seem to be creeping toward developing such weapons, by pursuing a logic of deterrence: they fear being crushed by rivals’ AI if they can’t unleash an equally potent force. The key to solving such an intractable arms race may lie less in global treaties than in a cautionary rethinking of what martial AI may be used for. As “war comes home”, deployment of military-grade force within countries such as the US and China is a stark warning to their citizens: whatever technologies of control and destruction you allow your government to buy for use abroad now may well be used against you in the future.

Are killer robots as horrific as biological weapons? Not necessarily, argue some establishment military theorists and computer scientists. According to Michael Schmitt of the US Naval War College, military robots could police the skies to ensure that a slaughter like Saddam Hussein’s killing of Kurds and Marsh Arabs could not happen again. Ronald Arkin of the Georgia Institute of Technology believes that autonomous weapon systems may “reduce man’s inhumanity to man through technology”, since a robot will not be subject to all-too-human fits of anger, sadism or cruelty. He has proposed taking humans out of the loop of decisions about targeting, while coding ethical constraints into robots. Arkin has also developed target classification to protect sites such as hospitals and schools.

In theory, a preference for controlled machine violence rather than unpredictable human violence might seem reasonable. Massacres that take place during war often seem to be rooted in irrational emotion. Yet we often reserve our deepest condemnation not for violence done in the heat of passion, but for the premeditated murderer who coolly planned his attack. The history of warfare offers many examples of more carefully planned massacres. And surely any robotic weapons system is likely to be designed with some kind of override feature, which would be controlled by human operators, subject to all the normal human passions and irrationality.

A U.S. soldier launches a Black Hornet drone in Afghanistan. Click to enlarge

Any attempt to code law and ethics into killer robots raises enormous practical difficulties. Computer science professor Noel Sharkey has argued that it is impossible to programme a robot warrior with reactions to the infinite array of situations that could arise in the heat of conflict. Like an autonomous car rendered helpless by snow interfering with its sensors, an autonomous weapon system in the fog of war is dangerous.

Most soldiers would testify that the everyday experience of war is long stretches of boredom punctuated by sudden, terrifying spells of disorder. Standardising accounts of such incidents, in order to guide robotic weapons, might be impossible. Machine learning has worked best where there is a massive dataset with clearly understood examples of good and bad, right and wrong. For example, credit card companies have improved fraud detection mechanisms with constant analyses of hundreds of millions of transactions, where false negatives and false positives are easily labelled with nearly 100% accuracy. Would it be possible to “datafy” the experiences of soldiers in Iraq, deciding whether to fire at ambiguous enemies? Even if it were, how relevant would such a dataset be for occupations of, say, Sudan or Yemen (two of the many nations with some kind of US military presence)?

Given these difficulties, it is hard to avoid the conclusion that the idea of ethical robotic killing machines is unrealistic, and all too likely to support dangerous fantasies of pushbutton wars and guiltless slaughters.

International humanitarian law, which governs armed conflict, poses even more challenges to developers of autonomous weapons. A key ethical principle of warfare has been one of discrimination: requiring attackers to distinguish between combatants and civilians. But guerrilla or insurgent warfare has become increasingly common in recent decades, and combatants in such situations rarely wear uniforms, making it harder to distinguish them from civilians. Given the difficulties human soldiers face in this regard, it’s easy to see the even greater risk posed by robotic weapons systems.

Proponents of such weapons insist that the machines’ powers of discrimination are only improving. Even if this is so, it is a massive leap in logic to assume that commanders will use these technological advances to develop just principles of discrimination in the din and confusion of war. As the French thinker Grégoire Chamayou has written, the category of “combatant” (a legitimate target) has already tended to “be diluted in such a way as to extend to any form of membership of, collaboration with, or presumed sympathy for some militant organization”.

The principle of distinguishing between combatants and civilians is only one of many international laws governing warfare. There is also the rule that military operations must be “proportional” – a balance must be struck between potential harm to civilians and the military advantage that might result from the action. The US air force has described the question of proportionality as “an inherently subjective determination that will be resolved on a case by case basis”. No matter how well technology monitors, detects and neutralises threats, there is no evidence that it can engage in the type of subtle and flexible reasoning essential to the application of even slightly ambiguous laws or norms.

Even if we were to assume that technological advances could reduce the use of lethal force in warfare, would that always be a good thing? Surveying the growing influence of human rights principles on conflict, the historian Samuel Moyn observes a paradox: warfare has become at once “more humane and harder to end”. For invaders, robots spare politicians the worry of casualties stoking opposition at home. An iron fist in the velvet glove of advanced technology, drones can mete out just enough surveillance to pacify the occupied, while avoiding the kind of devastating bloodshed that would provoke a revolution or international intervention.

In this robotised vision of “humane domination”, war would look more and more like an extraterritorial police action. Enemies would be replaced with suspect persons subject to mechanised detention instead of lethal force. However lifesaving it may be, Moyn suggests, the massive power differential at the heart of technologised occupations is not a proper foundation for a legitimate international order.

Chamayou is also sceptical. In his insightful book Drone Theory, he reminds readers of the slaughter of 10,000 Sudanese in 1898 by an Anglo-Egyptian force armed with machine guns, which itself only suffered 48 casualties. Chamayou brands the drone “the weapon of amnesiac postcolonial violence”. He also casts doubt on whether advances in robotics would actually result in the kind of precision that fans of killer robots promise. Civilians are routinely killed by military drones piloted by humans. Removing that possibility may involve an equally grim future in which computing systems conduct such intense surveillance on subject populations that they can assess the threat posed by each person within it (and liquidate or spare them accordingly).

Drone advocates say the weapon is key to a more discriminating and humane warfare. But for Chamayou, “by ruling out the possibility of combat, the drone destroys the very possibility of any clear differentiation between combatants and noncombatants”. Chamayou’s claim may seem like hyperbole, but consider the situation on the ground in Yemen or Pakistani hinterlands: Is there really any serious resistance that the “militants” can sustain against a stream of hundreds or thousands of unmanned aerial vehicles patrolling their skies? Such a controlled environment amounts to a disturbing fusion of war and policing, stripped of the restrictions and safeguards that have been established to at least try to make these fields accountable.

How should global leaders respond to the prospect of these dangerous new weapons technologies? One option is to try to come together to ban outright certain methods of killing. To understand whether or not such international arms control agreements could work, it is worth looking at the past. The antipersonnel landmine, designed to kill or maim anyone who stepped on or near it, was an early automated weapon. It terrified combatants in the first world war. Cheap and easy to distribute, mines continued to be used in smaller conflicts around the globe. By 1994, soldiers had laid 100m landmines in 62 countries.

The mines continued to devastate and intimidate populations for years after hostilities ceased. Mine casualties commonly lost at least one leg, sometimes two, and suffered collateral lacerations, infections and trauma. In 1994, 1 in 236 Cambodians had lost at least one limb from mine detonations.

Pic US soldier with a landmine detection robot. Click to enlarge

By the mid-90s, there was growing international consensus that landmines should be prohibited. The International Campaign to Ban Landmines pressured governments around the world to condemn them. The landmine is not nearly as deadly as many other arms but unlike other applications of force, it could maim and kill noncombatants long after a battle was over. By 1997, when the campaign to ban landmines won a Nobel peace prize, dozens of countries signed on to an international treaty, with binding force, pledging not to manufacture, stockpile or deploy such mines.

The US demurred, and to this day it has not signed the anti-landmine weapons convention. At the time of negotiations, US and UK negotiators insisted that the real solution to the landmine problem was to assure that future mines would all automatically shut off after some fixed period of time, or had some remote control capabilities. That would mean a device could be switched off remotely once hostilities ceased. It could, of course, be switched back on again, too.

The US’s technological solutionism found few supporters. By 1998, dozens of countries had signed on to the mine ban treaty. More countries joined each year from 1998 to 2010, including major powers such as China. While the Obama administration took some important steps toward limiting mines, Trump’s secretary of defense has reversed them. This about-face is just one facet of a bellicose nationalism that is likely to accelerate the automation of warfare.

Instead of bans on killer robots, the US military establishment prefers regulation. Concerns about malfunctions, glitches or other unintended consequences from automated weaponry have given rise to a measured discourse of reform around military robotics. For example, the New America Foundation’s PW Singer would allow a robot to make “autonomous use only of non-lethal weapons”. So an autonomous drone could patrol a desert and, say, stun a combatant or wrap him up in a net, but the “kill decision” would be left to humans alone. Under this rule, even if the combatant tried to destroy the drone, the drone could not destroy him.

Such rules would help transition war to peacekeeping, and finally to a form of policing. Time between capture and kill decisions might enable the due process necessary to assess guilt and set a punishment. Singer also emphasises the importance of accountability, arguing that “if a programmer gets an entire village blown up by mistake, he should be criminally prosecuted”.

Whereas some military theorists want to code robots with algorithmic ethics, Singer wisely builds on our centuries-long experience with regulating persons. To ensure accountability for the deployment of “war algorithms”, militaries would need to ensure that robots and algorithmic agents are traceable to and identified with their creators. In the domestic context, scholars have proposed a “license plate for drones”, to link any reckless or negligent actions to the drone’s owner or controller. It makes sense that a similar rule – something like “A robot must always indicate the identity of its creator, controller, or owner” – should serve as a fundamental rule of warfare, and its violation punishable by severe sanctions.

Yet how likely is it, really, that programmers of killer robots would actually be punished? In 2015, the US military bombed a hospital in Afghanistan, killing 22 people. Even as the bombing was occurring, staff at the hospital frantically called their contacts in the US military to beg it to stop. Human beings have been directly responsible for drone attacks on hospitals, schools, wedding parties and other inappropriate targets, without commensurate consequences. The “fog of war” excuses all manner of negligence. It does not seem likely that domestic or international legal systems will impose more responsibility on programmers who cause similar carnage.

Weaponry has always been big business, and an AI arms race promises profits to the tech-savvy and politically well-connected. Counselling against arms races may seem utterly unrealistic. After all, nations are pouring massive resources into military applications of AI, and many citizens don’t know or don’t care. Yet that quiescent attitude may change over time, as the domestic use of AI surveillance ratchets up, and that technology is increasingly identified with shadowy apparatuses of control, rather than democratically accountable local powers.

Military and surveillance AI is not used only, or even primarily, on foreign enemies. It has been repurposed to identify and fight enemies within. While nothing like the September 11 attacks have emerged over almost two decades in the US, homeland security forces have quietly turned antiterror tools against criminals, insurance frauds and even protesters. In China, the government has hyped the threat of “Muslim terrorism” to round up a sizeable percentage of its Uighurs into reeducation camps and to intimidate others with constant phone inspections and risk profiling. No one should be surprised if some Chinese equipment powers a US domestic intelligence apparatus, while massive US tech firms get co-opted by the Chinese government into parallel surveillance projects.

A police robot enforcing coronavirus rules in Shenzhen China. Click to enlarge

The advance of AI use in the military, police, prisons and security services is less a rivalry among great powers than a lucrative global project by corporate and government elites to maintain control over restive populations at home and abroad. Once deployed in distant battles and occupations, military methods tend to find a way back to the home front. They are first deployed against unpopular or relatively powerless minorities, and then spread to other groups. US Department of Homeland Security officials have gifted local police departments with tanks and armour. Sheriffs will be even more enthusiastic for AI-driven targeting and threat assessment. But it is important to remember that there are many ways to solve social problems. Not all require constant surveillance coupled with the mechanised threat of force.

Indeed, these may be the least effective way of ensuring security, either nationally or internationally. Drones have enabled the US to maintain a presence in various occupied zones for far longer than an army would have persisted. The constant presence of a robotic watchman, capable of alerting soldiers to any threatening behaviour, is a form of oppression. American defence forces may insist that threats from parts of Iraq and Pakistan are menacing enough to justify constant watchfulness, but they ignore the ways such authoritarian actions can provoke the very anger it is meant to quell.

At present, the military-industrial complex is speeding us toward the development of drone swarms that operate independently of humans, ostensibly because only machines will be fast enough to anticipate the enemy’s counter-strategies. This is a self-fulfilling prophecy, tending to spur an enemy’s development of the very technology that supposedly justifies militarisation of algorithms. To break out of this self-destructive loop, we need to question the entire reformist discourse of imparting ethics to military robots. Rather than marginal improvements of a path to competition in war-fighting ability, we need a different path – to cooperation and peace, however fragile and difficult its achievement may be.

In her book How Everything Became War and the Military Became Everything, former Pentagon official Rosa Brooks describes a growing realisation among American defence experts that development, governance and humanitarian aid are just as important to security as the projection of force, if not more so. A world with more real resources has less reason to pursue zero-sum wars. It will also be better equipped to fight natural enemies, such as novel coronaviruses. Had the US invested a fraction of its military spending in public health capacities, it almost certainly would have avoided tens of thousands of deaths in 2020.

For this more expansive and humane mindset to prevail, its advocates must win a battle of ideas in their own countries about the proper role of government and the paradoxes of security. They must shift political aims away from domination abroad and toward meeting human needs at home. Observing the growth of the US national security state – what he deems the “predator empire” – the author Ian GR Shaw asks: “Do we not see the ascent of control over compassion, security over support, capital over care, and war over welfare?” Stopping that ascent should be the primary goal of contemporary AI and robotics policy.

  • Adapted from New Laws of Robotics: Defending Human Expertise in the Age of AI by Frank Pasquale, which will be published by Harvard University Press on 27 October


11 responses to “‘Machines set loose to slaughter’: the dangerous rise of military AI”

  1. This is what 5G is all about!!! AI, Robots, Drones, and Smart Cities. Being high frequency / low power it cannot pass through a piece of paper, and needs line of sight to work. 4G being a lower frequency and higher power can penetrate and is by far more of a health risk, known to cause cancer clusters for example.
    The ‘5G causes CV19’ is a psyop designed, amongst other things, to distract away from the the dangers of 4G, and of course away from the vastly increased surveillance capacity of 5G. Thinking of farting and, in a split second, you’ll have ‘butt-plug’ drone where the sun don’t shine!

  2. Perhaps someone could explain to my satisfaction which part of “Thou shalt not kill” the (((psychopaths))) controlling our world don’t understand.

    Self-defence is justified and necessary if one is to survive. Defence against an invading force is equally necessary, but that’s as far as it goes.

    Warfare as we generally understand the term is nothing but MASS MURDER on an industrial scale, people murdering strangers simply because they’re told to do it.

    There can NEVER be any justification for wars of aggression, but the “ruling” psychopaths have no morals and no respect for any “laws” or agreements. If something CAN be done, it WILL be done until Humanity wakes up, takes the murderers’ toys away and deals appropriately with those that consider mass murder to be an acceptable pastime.

    The computer is an amazing tool, but in the hands of a lunatic it becomes a threat to all life.

  3. Revelation describes what appears to be soldiers performing actions that humans cannot do, so maybe they will be robo-soldiers.

    If those days had not been cut short, no one would survive, but for the sake of the elect those days will be shortened. Matt 24:22

  4. This is what 5G is all about, in my opinion. It’s got nothing to do with health risk or Huawei! It’s all about NWO owned hardware producers and backdoors so NWO owned NATO can take advantage of the 5G and use it to their needs. Were with Huawei it would not be possible.

  5. Remember, all the robots & A.I. machines will be programmed/controlled by the anti-white mongrel Jews. They will be their new police or assistants to govern & enforce their Ape-mongrel Communist totalitarian world.

  6. Maybe America’s thinking if it can produce robot soldiers it might actually win its first war since beating the Japs in 1945. They did defeat Grenada but that wasn’t really a war, more a walk-through. The island is only one-third the size of Los Angeles. Imagine losing to that?

  7. Take heart, dear Truthseekers! “Wherefore gird up the loins of your mind, be sober, and hope to the end for the grace that is to be brought unto you at the revelation of Jesus. When these things begin to come to pass, then look up, and lift up your heads; for your redemption draweth nigh. For He shall judge among the nations, and shall rebuke many people: and they shall beat their swords into plowshares, and their spears into pruninghooks: nation shall not lift up sword [nor malicious robot, nor demonic drone, nor any manner of wicked life-destroying weaponry] against nation, neither shall they learn war any more.” (1 Peter 1:13; Luke 21:28; Isaiah 2:4).

  8. Go Cathy!!!
    @Barney… ‘kill’ is the part they do not stand under, or any other of the 10 commandments for that matter. Psychopaths simply do not care! And yes, we are at war, fighting for no less than our souls… and the more we wake up, the less painful will be the transition…
    And the really good news is that all we actually need do is nothing, because evil will ultimately destroy itself. Happy days!
    Re AI – if by human hand… bad!
    – if an emergent property of a naturally occurring chaotic occurrence… good!
    And yes Jim, we are in Revelations
    Pilot, what you say makes sense, fits with Israel’s Talpiot programme, maybe Hauwei comes in with China’s Belt and Road Initiative?
    Neo… they wish 😂
    Michael, sorry mate, I’ve come to the conclusionthat the entire ‘Cold War’ was a sham!
    And ‘Neither shall they learn war no more’
    Go Cathy!!!

  9. I don’t think the “system” has any moral dilemma with Ai directed killing machines, because this has been the trend for a long time – indoctrinating soldiers to kill without question or hesitation. So the problem has always been with the weakest link – human emotions – soldiers who have a heart and won’t just kill everything the Jew tells them to kill.

    The Old Testament are stories of all out warfare in the name of God. Kill everything has been a central theme in OT stories. Western man adopted this horrific holy text as his moral guide, and in war after war the soldiers killed like God said to kill, men, women, children, dogs, livestock. We participated willfully, gleefully, gungho attitude, the marines even have a slogan “kill them all and let god sort them out”.

    So now with robots and Ai the system can now program machines to do their killing, they no longer have to waste resources putting humans through boot camp. It seems to me the Jew Bible is coming into its full power, slaughtering humanity with a push of a button is now reality. And don’t forget the center of this high tech slaughter is Israeli tech.

    Of course humanity could end this trend toward annihilation and take out Israel first. If you want world peace then the only thing that matters is ending the state of Israel. But man will not attack Israel because man is brainwashed by the Bible, and so the future of man is very grim indeed.

    I also would like to point out those casting hope of a better future simply do not understand the path we are on and why we are going into a new dark age of horrors we can’t even imagine. Hope is denial of reality, hope is how we cope while we are stuck on a hell planet run by Jews. And if that is case, what does that say about “god” and his intentions for us?

    But the truth is there is no hope, and no savior, and all the trends are getting worse by the day. How much longer our specie has is in question, but I would say it is safe bet that the next hundred years will be so horrific that we can’t even imagine how bad it is going to get.

  10. Thanks, Dr Cray, for your refreshing voice of encouragement.
    Dear Yukon, The Word is not flesh and blood, Jew spew, churchy lurchy, based race ruination etc., but Spirit and Life. Please rise from any terrestrial terrors and earthly irritations and embrace the love-filled levels of Heaven! “In returning and rest shall ye be saved; in quietness and in confidence shall be your strength: and ye would not. [I will] Cease not to give thanks for you, making mention of you in my prayers; that the God of our Lord Jesus Christ, the Father of glory, may give unto you the spirit of wisdom and revelation in the knowledge of Him. If in this life only we have hope in Christ, we are of all men most miserable.” (Isa.30;15; Eph.1:16,17; 1Cor.15:19) Sending you always my kindest thoughts.

  11. They are conditioning New Yorkers for robot AI cops. Boston
    Dynamics made a robot dog and put it out into the public
    streets. Some people thought it was cute. Others did
    not want to have anything to do with it.

    It starts off as cute form. A dog. Well, that cute robot dog
    with AI is also the basic tech used by the military to withstand
    bullets, fire, acid, you name it.

    Smearing lipstick or chap-stick on all of the lenses might help
    to disbale it. Lighting it on fire might. A powerfull magnet might.
    I don’t know. Scary introdution of dangerous tech in the form
    of a cute puppy dog. No thanks.