- In January 2020, Robert Julian Borchak Williams was handcuffed and arrested in front of his family for shoplifting after being identified by facial recognition used by the Detroit Police Department. The system was wrong and he wasn’t a criminal but, because a machine said so, Williams spent 30 hours in jail.
- Williams has the distinction of being the first person arrested and jailed after being falsely identified by facial recognition – or, at least, the first person that we the public have been told about. The Detroit police chief said at a meeting following the reports of Williams’ arrest that the system misidentified suspects 96% of the time1. Given the wider discussion around reforming policing in the US following the killing of George Floyd2 by Minneapolis officers, it's no wonder calls for bans of the tech are starting to be heard.
- Amazon, Microsoft and IBM soon paused sales of facial-recognition systems to police, although it’s worth noting that there are plenty of specialist companies that still sell to authorities. Politicians are calling for a blanket ban until the technology is better understood and proven safe. "There should probably be some kind of restrictions." Jim Jordan, a Republican representative, said in a committee hearing. "It seems to be it's time for a timeout.”
- That’s in the US. In the UK, police continue to use the controversial technology. The Met Police used it at the Notting Hill Carnival and outside Stratford station in London, but the tech is also used by police in South Wales. "Facial recognition has been creeping across the UK in public spaces.” Garfield Benjamin, researcher at Solent University, told PC Pro. "It is used by the police, shopping centres, and events such as concerts and festivals. It appears in most major cities and increasingly other places, but is particularly prevalent across London where the Met Police and private developers have been actively widening its use.”
- That's despite a growing body of evidence that suggests the systems aren’t accurate, with research from activist group Big Brother Watch3 claiming that 93% of people stopped4 by the Met Police using the tech were incorrectly identified. A further study by the University of Essex showed the Met Police's system was accurate 19% of the time5.
- Can facial recognition ever work? Is a temporary ban enough? Or is this a technology that should forever be relegated to photo apps rather than serious use? The answers to these questions will decide the future of facial recognition - but the road forward isn't clear.
- The problems with facial recognition tech
- The problems with facial recognition aren’t limited to a few instances or uses - it's across the entire industry. A study by the US National Institute of Standards and Technology tested 189 systems from 99 companies, finding that black and Asian faces were between ten to 100 times more likely to be falsely identified6 than people from white backgrounds.
- What causes such problems? Sometimes the results are due to poor quality training data, which could be too limited or biased - some datasets don't have as many pictures of black people as other racial groups, for example, meaning the system has less to go on. In other instances, the algorithms are flawed, again perhaps because of human bias, meaning good data is misinterpreted.
- That could be solved by having a “human in the loop”, when a person uses data from an Al but still makes the final decision - what you would expect to happen with policing, with a facial-recognition system flagging a suspect for officers to investigate, not blindly arrest. But we humans too easily put our faith in machines, says Birgit Schippers. a senior lecturer at St Mary’s University College Belfast. "There’s also concern over automation bias, where human operators trust, perhaps blindly, decisions proposed by a facial-recognition technology," she said. "Trained human operators should in fact take decisions that are based in law.”
- Even a sound system trained well on a solid dataset can have downsides. “It has a profound impact on our fundamental human rights, beginning with the potential for blanket surveillance that creates a chill factor, which impacts negatively on our freedom of expression, and perhaps our willingness to display nonconformist behaviour in public places.” Schippers explained. "Another fundamental concern is lack of informed consent7 ... we do not know what is going to happen with our data.”
- Then there's the other side of human intervention: misuse. " Another key concern is the way that facial-recognition technology can be used to target marginalised, vulnerable, perhaps already over-policed communities," she said.
- Whether we allow the tech in policing or elsewhere should depend on whether the benefits outweigh the downsides, argues Kentaro Toyama, a computer scientist at the University of Michigan. "The technology provides some kinds of convenience – you no longer have to hand label all your friends in Facebook photos, and law enforcement can sometimes find criminal suspects quicker.” Toyama said. "But. all technology is a double-edged sword. You now also have less privacy-, and law enforcement sometimes goes after the wrong people." And it's worth remembering, added Toyama, that facial recognition isn't a necessity. “There was no such technology - at least, none that was very accurate - until five to ten years ago, and there were no major disasters and no geopolitical crises because of the lack8."
- Fixing facial recognition
- New technologies don't arrive fully formed and perfect. They need to be trialled and tested to spot flaws and bugs and knock-on effects before being rolled out more widely. Arguably, facial recognition has been rolled out too quickly because we're still clearly in the phase of finding the problems with and caused by this technology. So, now that we know about bias, inaccuracy and misuse, can we fix those problems to make this technology viable? “If people are seeking to make them fairer, then on a technical level we need to address bias in training data which leads to misidentification of women and ethnic minorities," said Solent University’s Benjamin. But, he added, you must be vigilant: "The audit data that is used to test these systems as biased audits often conceal deeper flaws9. If your training data is mostly white males, and your audit data is mostly white males, then the tests won't see any problem with only being able to correctly identify white males.”
- There have been efforts to build more diverse facial recognition training sets, but that only addresses one part of the problem. The systems themselves can have flaws with their decision-making, and once a machine-learning algorithm is fully trained, we don’t necessarily know what it’s looking for10 when it examines an image of a person.
- There are ways around this. A system could share its workings11, telling users why it thinks two images are of the same person. But this comes back to the automation bias: humans learn quickly to lean on machine-made decisions. The police in the Williams case should have taken the facial recognition system as a single witness, and further investigated - had they asked, they would have learned Williams had an alibi for the time of the theft. In short, even with a perfect system, we humans can still be a problem12.
- Regulation, regulation, regulation
- Given those challenges and the serious consequences of inaccuracy and misuse, it’s clear that facial recognition should be carefully monitored. That means regulators need to step in.
- However, regulators aren’t always up to the job. "Facial recognition crosses at least three major regulators in the UK: the CCTV Commissioner, Biometrics Commissioner and Information Commissioner,” said Benjamin. "All three have logged major concerns about the use of these technologies but have so far been unable to come together to properly regulate their use. The Biometrics Commissioner even had to complain to the Met Police when they misrepresented his views, making it seem like he was in favour of its use. We need more inter-regulator mechanisms, resources and empowerment to tackle these bigger and more systemic issues.”
- Beyond that, there is no specific regulation that addresses these concerns with facial recognition in the UK, noted St Mary’s University College Belfast’s Schippers, but there is currently a private members bill working its way through parliament, seeking to ban the use of facial recognition in public places13. In Scotland, MSPs have already made such a recommendation, but plans by Police Scotland to use the technology had already been put on hold.
- Such a pause could let regulators assess how and when to use the technology. “As the pros and cons become clearer, we should gradually allow certain applications, at progressively larger scales, taking each step with accompanying research and oversight, so we can understand the impacts.” said the University of Michigan’s Toyama.
- That’s worked for other potentially dangerous, but useful, advanced technologies. "The most effective form of this is the development of nuclear energy and weapons - not just anyone can experiment with it, sell it, or use it,” Toyama added. “It’s tightly regulated everywhere, as it should be."
- Time for a ban?
- Facial recognition is flawed, has the potential for serious negative repercussions, and regulators are struggling to control its use. Until those challenges can be overcome, many experts believe the technology should be banned from any serious use. "There is little to no evidence that the technologies provide any real benefit, particularly compared to their cost, and the rights violations are too great to continue their deployment.” Benjamin said. Toyama agrees that a moratorium is necessary until the potential impacts are better understood. “Personally, I think that many uses can be allowed as long as they are narrowly circumscribed and have careful oversight … though, I would only say that in the context of institutions I trust on the whole,” Toyama explained.
- Schippers would also like to see a ban - but not only on facial recognition technology's use by police forces, but by private companies too. “Retailers, bars, airports, building sites. leisure centres all use facial-rejection technology,” Schippers said. “It's becoming impossible to ignore.”
- What’s next?
- Facial recognition is quickly becoming a case study in how not to test and roll out a new idea - but other future technologies could also see the same mistakes.
- Look at driverless cars or drones: both are being pushed hard by companies and governments as necessary solutions to societal problems, despite the technologies remaining unproven, regulation not yet being in place, and the potential downsides not being fully considered.
- That said, facial recognition seems more alarming14 than its fellow startup technologies. “There’s something about facial recognition that many people feel to be particularly creepy - but facial recognition is just another arrow in the quiver of technologies that corporations and governments use to erode our privacy, and therefore, our ability to be effective, democratic citizens.” said Toyama.
- Due to that, and the inaccuracies, missteps and misuse, facial recognition faces a reckoning - and it’s coming fast.
- “I think the next five years will see a strong tipping point either way for facial recognition,” said Benjamin. “With some companies ceasing to sell the technologies to the police, and some regulatory success, we could see them fall out of favour. But the government and many police forces are very keen on expanding their use, so it will be a matter of whether rights and equality or security and oppression15 win out.”
- The pace of technology development continues to accelerate, but we should control its pace. We need to either slow it down via regulatory approval and testing, or speed up our own understanding of how it works and what could go wrong - or we risk more people like Williams being made victims of so-called progress.
- Sub-title: “Regulation against the future tech is looming amid concerns about its accuracy for policing and other public uses. Nicole Kobie reveals the future of facial recognition. ”
- PC Pro 312, October 2020
Footnote 2: Footnote 3: Footnote 4:
- This is an extraordinary failure rate, and I’d initially thought it a typo for “identified”, but apparently not.
- It’s not clear, however, how the “identification” takes place. Presumably there’s not a national database of mug-shots, so does the AI just make its best guess from a database of ex-cons?
- Maybe there’s a national database of ID cards, driver’s licenses or passport photos?
- See Wikipedia: Identity documents in the United States.
Footnote 5: Footnote 6:
- So, marginally better that the US, but still terrible.
- But – again – it’s not explained how the technology is used.
- Also, as this is a pressure group, how confident can we be of their statistics?
- I can well believe it, given the likely training algorithms.
- But how does this stack up with the dire success rates already reported?
- Well, for any system to work, there could be no question of “consent” – informed or otherwise – at the individual level (or all those that needed to be surveyed would opt out.
- This is something that would need to be voted on (hopefully not by a referendum) as a general policy.
- That is a very broad claim!
- Covid-19 contact tracing … wouldn’t facial recognition help? Is it being used in the Far East? Maybe use of mobile-phone tracking instead? Just as invasive?
- Well, yes, but these flaws can be fixed as well. Stop whining. We’re only arguing here about whether the tech can be got to work.
- This is a general problem with machine learning – we don’t know how it does it.
- But this is completely different from – say – credit rating. It either gets faces right or it doesn’t – if it does, we don’t care how it does it (like we don’t care how AlphaZero beats Stockfish). But a credit-rating isn’t a “fact” in the same way. A human that checks the algorithm might suffer from the same prejudices that are implicated in the training program. But you can’t be “prejudiced” about facial recognition, can you? You – and the AI – might both think you’ve identified someone – but you might both be wrong, and this is a fact that’s easily checked.
- Really? Not in neural networks.
- Well, yes – but that’s true of any technology, and it doesn’t suggest we all be Luddites.
- Given the technology doesn’t yet seem to work, it shouldn’t be used to inform any decisions without human oversight.
- But, like driverless cars, if you ban them from public spaces they will never improve.
- You must be joking! Driverless cars and drones can directly lead to significant loss of life.
- This is a very tendentious way of putting things. Face recognition – provided it works – is ethically neutral. It’s how it’s used and regulated that matters.
Text Colour Conventions (see disclaimer)
- Blue: Text by me; © Theo Todman, 2020
- Mauve: Text by correspondent(s) or other author(s); © the author(s)