Albert Wenger writes that concerns about “black box” algorithms are overwrought. (See here and here for more about these concerns.) It’s okay, Wenger says, if we can’t follow or audit the logic of the machines, even in life-and-death contexts like healthcare or policing. We often have that same lack of insight into the way humans make decisions, he says—and so perhaps we can adapt our current error prevention to the machines:
It all comes down to understanding failure modes and guarding against them.
For instance, human doctors make wrong diagnoses. One way we guard against that is by getting a second opinion. Turns out we have used the same technique in complex software systems. Get multiple systems to compute something and act only if their outputs agree. This approach is immediately and easily applicable to neural networks.
Other failure modes include hidden biases and malicious attacks (manipulation). Again these are no different than for humans and for existing software systems. And we have developed mechanisms for avoiding and/or detecting these issues, such as statistical analysis across systems.
