This is one of the few topics on hn that I actually have some expertise in, after a ten year career consulting for the three big biometric systems; involved in standards work, accuracy testing, and R&D program management. Note that I have been out of the game for 7-8 years, but things do not change fast. There are several issues, some of which the biometrics community is aware of and some of which are outside their worldview.

The first is that the face matching algorithms are inherently less accurate on people of color given the sensors and algorithms in use. This has been an issue going back to color film being developed for Caucasian consumers - the range on a lot of sensors simply doesn't provide enough contrasting tones for brown and black people. The face detection, extraction, and matching algorithms are often not developed using datasets that match the demographics they will be used on, and at least 7-8 years ago there wasn't much focus on providing ethnicity-specific error rates. The biometrics community largely knows about this and understands it.

The second issue is that even the flagship programs don't measure Precision and Recall. They operate purely off metrics like False Match Rate and False Positive Identification Rate. Those are closed-set (i.e. "everyone is enrolled"), tech-only metrics that can be measured by NIST, that don't consider system usage or CONOPs. Program management asks for Precision and Recall metrics (not by name, because they don't know what they are), and tech responds with True Match Rate and True Positive Identification Rate. This is an entire worldview where whole-system accuracy has been placed fully on algorithmic error rates. Leadership generally DOES NOT KNOW that error rates are just as dependent on usage - i.e. that if you search for someone who isn't in the database, 100% of the matches will be false. They do not understand which part of the confusion matrix they are being told about.

The third issue is that, while these massive federal programs are run by ethical and mission-oriented professionals who have hill-climbed their way to a pretty accurate system with stable usage, the talent and professionalism isn't there across the state and local level (there are a couple bright spots where at least the tech is well understood). Face matching systems are generally not lights-out like fingerprint or iris systems (fingerprint systems do generally require a manual verification step, but that's driven more by policy and history than the accuracy metrics require); FR requires a second step where a trained face matching expert looks at the proposed matches in the consideration set and uses several techniques to assess if they are correct - it's a notable difference compared to fingerprint that candidate faces are not supposed to be presumed matches when they come out of the automated system, and even after the human expert they are generally supposed to be evidence/leads that need to be corroborated through other means. As far as I know (again, several years out of date now), no one has checked whether the accuracy of the human experts varies by ethnicity of the subject.

Even the FBI's professionalism broke down with the Madrid Train Bombing, resulting in the arrest of a random Brown lawyer in Seattle (again, no one considered that looking for an international political terrorist in a domestic criminal database meant a much lower prior probability than IAFIS generally operates with, before you even start talking about stitching together multiple partial fingerprints, and you shouldn't be considering anything in the person's file while making an objective matching decision). A local police department is going to be much worse in terms of policy and system usage.

And finally, there is the reality that once you are enrolled in a booking database, you are accused of every crime that happens in any linked jurisdiction until the system goes offline. The probability that you will have an encounter with the police is not uniformly distributed across the population, nor is the probability that such an encounter will lead to a booking where your photo is taken and enrolled in a database forever. This is not something the biometrics community really discusses at all, but every enrollment in one of these systems is hundreds of thousands of opportunities for a false match per year, hanging over your head like bad debt.

The end result is that it's not surprising that just under one in six people in America are Black, but all six of these false arrests were Black. It's also not easily fixed in one place by "giving everyone implicit bias training."