How Facial Recognition Actually Works — And Where It Is Watching You

A face match is a probability, not a fact. Here is how the tech works, where it is really deployed, where it fails, and what you can do about it.

There is a particular sinking feeling that comes when a machine decides who you are and gets it wrong. It has happened to real people: someone arrested at home in front of their kids because an algorithm flagged their driver's license photo as a match for grainy store footage they had never been near. By the time a human checks the work, the squad car is already there. The thing most people do not realize is that the "match" that started it all was never a yes. It was a percentage. Someone decided that percentage was close enough.

That gap — between a probability and a person's life — is the whole story of facial recognition. Here is the record on how it works, where it is actually running, and what you can do.

How a face becomes a number

Facial recognition does not "see" your face the way you do. It converts it into math. A neural network trained on millions of faces measures the geometry and texture of yours — distances and relationships between features — and outputs a list of numbers called a faceprint or embedding (often a few hundred values). Two photos of the same person produce embeddings that sit close together in that mathematical space; two different people sit farther apart.

There are two very different jobs people lump together:

• Verification (1-to-1): Are you the person in this one stored photo? This is what unlocks your phone or checks your passport. It is comparatively easy because the system only compares against a single reference, and you are cooperating.
• Identification (1-to-many): Who is this, out of a database of millions? This is the law-enforcement and surveillance use. It is far harder and far riskier, because the system returns a ranked list of candidates and someone has to decide which "close enough" hit to believe.

The output is always a similarity score, never a verdict. A human or a policy sets the threshold for what counts as a match — and where that line sits decides how many innocent people get flagged.

Where it is actually deployed

This is not science fiction. It is in service today, in three broad arenas.

Airports and borders. This is the most widespread and most normalized use in the United States. Customs and Border Protection runs face comparison at airport exits and entries, and the TSA has been expanding camera-based ID checks at security lines, matching your live photo to your ID document. At the airport it is mostly verification (1-to-1), which is the more accurate mode. Crucially, for many of these checkpoints you can decline the face scan and ask for a standard manual ID check — signage often buries this, but the opt-out has generally been offered for travelers.

Police. Departments and agencies compare probe images (surveillance stills, social media photos) against mugshot databases, driver's license photos, or commercial scrapers. Private vendors have assembled enormous searchable databases by scraping public photos from the open web and social media without consent. This is the highest-stakes use and the least transparent — defendants frequently are not told that a face search generated the lead against them.

Retail and private spaces. Stores, stadiums, and casinos use it for loss prevention and to enforce banned-person lists. The notable pattern here is using it to exclude people — and at least one large venue operator drew scrutiny for using face recognition to identify and bar people connected to firms in litigation against it. Your local grocery store could be running it with nothing more than a sign by the door, if that.

Where it fails — and who pays

The accuracy claims you hear ("99%+") usually come from the easy case: high-quality, well-lit, 1-to-1 verification. Real-world identification off a blurry camera at a bad angle is a different animal, and the errors are not evenly distributed.

Large-scale government testing of commercial algorithms has repeatedly found that error rates vary by demographic group — with higher false-match rates for some groups, and women, older adults, children, and people with darker skin tones often faring worse, depending on the algorithm. That is not a rounding error when the consequence is handcuffs. The known cases of people wrongfully arrested after a bad facial-recognition lead have disproportionately involved Black Americans.

Two structural problems make this worse than a simple accuracy stat suggests:

1. Automation bias. Once a screen shows a ranked candidate with a confidence number, humans tend to trust it and stop digging. The tool is sold as "just an investigative lead," but in practice the lead becomes the case.
2. Garbage in. A low-quality probe image, an edited photo, or a forensic sketch run through these systems still returns confident-looking matches. The system does not refuse. It always answers.

Kooky till proven — and your rights

Treat any face match as a claim to be verified, not a fact. The burden is on the system to prove it, the same way NU ranks by the record and not by who is most confident.

What you can actually do:

• At the airport, you can ask for a manual ID check instead of the face scan. Say it plainly at the podium; you are entitled to ask, and declining should not cost you your flight.
• Know your state. A handful of states regulate biometric data. Illinois's Biometric Information Privacy Act (BIPA) is the strongest — it requires consent before a private company captures your faceprint and lets individuals sue, which has produced major settlements. Texas and Washington have their own biometric laws. Several cities have banned government and police use outright. Where you live changes your rights dramatically.
• If you are charged, and a face search may have generated the lead, that is something for a defense lawyer to demand in discovery. Whether the technology was used — and whether it was disclosed — matters.
• As a customer or tenant, you can ask a business directly whether it uses facial recognition and what it does with the data. Silence is an answer too.
• Lower your footprint by checking the privacy settings on photo-sharing accounts; scrapers feed on public images.

The core principle is simple. A face match is a probability dressed up as certainty. Anyone using it on you should be able to show their work — the image quality, the threshold, the second source that confirms it. Until they do, it is a lead, not a verdict. Make them prove it.