It’s that it simply does not work.

I have written several times about how existing Artificial Intelligence (AI) software can discriminate its targets. This time, I’d like to point you to a great summary of the real, “upstream” problem:

no matter how much they are marketed as THE solution to ALL sorts of problems, AI and machine learning simply cannot work in some domains

This is well explained, with plenty of examples that you really should read, in a slideshow titled “How to recognize AI snake oil”, whose main conclusions are that:

• Companies advertising AI as the solution to all problems have been helped along by credulous media [and policy makers]
• Certain AI systems, for example those screening job candidates are " essentially elaborate random number generators"
• the most “fundamentally dubious” AI applications are indeed those made to “predict social outcomes” from criminal recidivism to job performance

If AI and machine learning just cannot do certain jobs

This has two consequences. One is that, if algorithmic decision is unsuited to certain tasks to begin with, finding when it discriminates is certainly possible, as discussed in this paper, but at best it only makes the algorithms FAIL… more fairly than before.

The other obvious consequence is that there is little, possibly very little substance in the belief that human oversight may significantly correct and improve algorithmic decision-making, once it has been implemented. Assuming all the stakeholders know that it was implemented, of course, which too often is NOT the case.

The only solution, discussed in the last link, is to limit the damage by “incorporating an algorithm into decision-making” only when:

• first, it can be demonstrated that it is appropriate (and actually useful)
• and only after those proofs HAVE been carefully reviewed and approved before the algorithms are actually adopted

Luckily, those are decisions that should be fairly easy to take without any algorithm. Right?