Autonomous Fighting Machines & Heidegger

George Johnson at the New York Times writes an interesting reflection on the questions raised by autonomous fighting machines. He comes to a key question:

As the thinking machinery continues to evolve, the strategists will keep asking themselves the same question: Is there still a good reason to trust ourselves or should we defer to a computer's calculations?

The concern here, as Mike at TechDirt puts it, is "that even when humans do have the final word, do they become complacent in the face of computer generated information?"

This is related to the central concern of Martin Heidegger's The Question Concerning Technology . The danger of technology, Heidegger argues, lies in the transformation of the human being, by which human actions and aspirations are fundamentally distorted by what he calls a "technological understanding of being". Technology enters the inmost recesses of human existence, transforming the way we know and think. The "greatest danger" is that:

The approaching tide of technological revolution…could so captivate, bewitch, dazzle, and beguile man that calculative thinking may someday become so accepted and practiced as the only way of thinking.

Technology becomes, then, the primary mode of human existence, and Heidegger’s concern is the potential harm caused by this technological understanding of being.

Johnson understands this in terms of our potential reliance on the "calculative thinking" of autonomous fighting machines. He quotes John Searle, a philosopher at the University of California at Berkeley whose most recent book, "Mind: A Brief Introduction," came out last year. Searle argues that the brain is not just a computer strung together from neurons. Johnson writes:

Whatever is happening in the head - and nobody really knows - it is not computation. Confuse reason with calculation, he argues, and disaster lies ahead.

By substituting human reason with computer calculations, and by placing our faith in the calculative thinking of the machine, we are in danger of embracing a new technological understanding of being - that of the infallability of computational logic. The becomes especially dangerous when we're talking not about TiVo's ability to predict your viewing patterns, but about battlefield technology.

Johnson wisely closes with this nagging suspicion: "The machine will have been designed by the imperfect species called homo sapiens. What if we got something wrong?"