The perfect human and the flawed AI

There is a growing debate about whether machines should ever be allowed to select and attack targets without human involvement.

The United Nations Secretary General has called such weapons morally repugnant. The International Committee of the Red Cross wants strict rules prohibiting autonomous weapons whose effects cannot be understood or controlled, and those designed or used to target people directly.

The concern is understandable. The idea that a machine could decide who lives and who dies is deeply uncomfortable.

But beneath the debate sits an assumption that deserves much more scrutiny.

It is the assumption that keeping a human involved makes warfare more moral, more controlled and somehow safer.

Does it?

The changing battlefield

Weapons have been moving away from direct human control for decades.

Landmines do not know who steps on them. Defensive systems can already detect and intercept incoming missiles at speeds beyond human reaction. Loitering munitions can search an area for a particular type of target. AI enabled drones are being developed to navigate when communications have been jammed, recognise objects and continue towards a target after contact with the operator has been lost.

Most of the systems currently being used in Ukraine still involve some level of human control. But the direction is clear. Machines are becoming more capable of finding, selecting and attacking targets for themselves.

The arguments in favour are also clear. Machines can react faster than people. They do not become tired, frightened or confused. They can process more information than a human operator. They may be more accurate. They do not seek revenge because a colleague has been killed.

The arguments against are formidable. Machines can make mistakes at enormous speed and scale. They can be deceived. They may operate in situations their programmers did not anticipate. They cannot understand surrender, fear, ambiguity or human intention in the way that another person might. Responsibility becomes dangerously blurred when a machine kills the wrong person.

But before we conclude that human control is the answer, we need to be honest about humans.

The myth of the reliable human

Human targeting has never been reliably accurate.

People misread intelligence. They identify the wrong building. They mistake civilians for combatants. They act on information that is incomplete, outdated or deliberately misleading. Under pressure, they see what they expect to see.

Technology may improve the accuracy with which a weapon reaches its selected target. It cannot ensure that the right target was selected in the first place.

This is not merely a technical problem. It is a psychological one.  Human beings divide the world into groups. In peaceful conditions this can be relatively harmless. In war it becomes lethal.

The enemy stops being a collection of individual human beings. They become a category. Germans. Japanese. Communists. Terrorists. Insurgents. The category is then given characteristics. It is treacherous, savage, dangerous or less civilised. Eventually, its members cease to be people at all.

During the genocide in Rwanda, Tutsis were repeatedly described as cockroaches. The language did not simply express hatred. It helped make killing psychologically possible. The United Nations has documented how dehumanising language contributed to the progression from words to atrocities.

The same process appears throughout history. Once people believe that the enemy is not fully human, cruelty can feel justified. Suffering becomes deserved. Restraint begins to look like weakness.

Machines do not hate.

An AI system will not call its target a cockroach. It will not want revenge. It will not enjoy humiliation. It will not become brutalised by months of combat.  It will simply classify a person as a target.  Is that better or worse?

Humans are capable of mercy, and atrocity

The massacre at My Lai in Vietnam was not caused by autonomous weapons. American soldiers killed hundreds of unarmed civilians. They were human beings operating under human command.

But the killing was also stopped by humans. Helicopter pilot Hugh Thompson Jr and his crew placed themselves between American troops and Vietnamese civilians. They threatened to fire on their own side if the killings continued.

That is the contradiction at the heart of this debate. Humans commit atrocities. Humans also refuse orders, recognise innocence, show mercy and accept personal risk to protect strangers.

An AI system has no hatred, but it has no compassion either. It cannot be disgusted by an order. It cannot experience the moment when obedience becomes morally intolerable. It cannot look at a frightened child and decide that whatever the data says, something is wrong.

Unless, of course, we successfully programme it to recognise those circumstances.  But who decides what it should recognise? Whose morality is being programmed? What level of civilian death is considered acceptable? How much uncertainty is enough to prevent an attack? What happens when military advantage depends upon making the system less cautious?

AI does not remove human judgement. It moves it.  The judgement shifts from the person pulling the trigger to the people who design the system, define its objectives, choose its data, set its tolerance for error and decide where it may operate.

The human has not disappeared. The human has moved further away from the consequences.

Would AI have fought the First World War?

Would AI have sent millions of men into the battles of the First World War?  Possibly not, if its objective had been to achieve a clearly defined military result at the lowest possible human cost. It might have recognised more quickly that repeated attacks against entrenched positions were producing enormous casualties for limited territorial gain.

But that depends entirely on the objective it was given.

If the instruction had been to exhaust the enemy, regardless of human cost, AI might have pursued attrition with even greater efficiency. It would not have been troubled by casualty lists. It would not have felt the political unease of a government or the despair of a general watching another generation disappear.

AI does not decide what constitutes a worthwhile purpose.  Humans do.

If humans define success badly, AI may execute that failure faster, more consistently and on a much larger scale.

The danger is not that AI will become bloodthirsty. The danger is that it will be obedient.

The wrong comparison

We are comparing the flawed AI with an imaginary human.

This imaginary human has perfect information, remains calm under pressure, interprets every situation correctly, resists prejudice, refuses illegal orders and always values civilian life.  That human has never existed.

The real comparison is between a fallible machine designed by humans and a frightened, angry, tribal and fallible person operating inside a military system.  Neither can be assumed to be moral.

Human involvement matters, but only if the human has enough information, time and authority to exercise genuine judgement. A person who has seconds to approve a recommendation from a system they do not understand is not exercising meaningful control. They are providing moral cover for an automated decision.

Equally, removing humans from the final decision does not remove human responsibility. Someone chose the objective. Someone approved the system. Someone authorised its deployment. Someone decided what error rate was acceptable.

So what should we conclude?

AI warfare will be different from human warfare.

It may be more precise. It may reduce some forms of emotional violence, panic and revenge. It may protect soldiers by keeping them away from the battlefield.

It may also make killing cheaper, faster and psychologically easier for the countries using it. It may allow thousands of targets to be attacked before anyone has time to question the assumptions behind the operation. It may create atrocities without hatred, simply through classification, scale and obedience.

The question is not whether humans or AI are morally superior.

The question is where human judgement must remain, what that judgement must involve and who will be held responsible when it fails.  We should not keep a human involved because humans are perfect.

We should keep humans accountable because only humans can decide what a war is for, what price is acceptable and when the killing must stop.

The most dangerous weapon may not be the AI that disobeys us.  It may be the AI that does exactly what we ask.