FRTE 1:N Identification

These comments provided by Neurotechnology are based on NIST FRTE 1:N Identification ongoing evaluation results reviewed on November 28, 2023.

What is FRTE 1:N Identification evaluation?

NIST FRTE 1:N Identification tests the suitability of biometric facial recognition algorithms for civil and law enforcement applications with two sets of experiments. The "Identification" set of experiments covers civilian use by setting criteria to minimize errors and assist in automated decisions. The "Investigation" set of experiments aims for law enforcement use by using the evaluation parameters to produce results for further human review to validate or dismiss returned candidates. Both sets of experiments include the the same scenarios and datasets.

See FRTE official page for more information.

The Face Recognition Technology Evaluation (FRTE) started in July, 2023, as a successor of the previous FRVT Ongoing evaluation of face verification algorithms. The FRTE 1:N Identification track allows vendors to submit their algorithms once every three months.

Neurotechnology algorithm has been ranked in the top 4% of most accurate results matching frontal and profile mugshots scenarios among 452 submissions by 128 different vendors. The algorithm showed top results among border control supervised (Visa vs Border, Border vs Border ΔT ≥ 10 YRS) and unsupervised (Visa vs Kiosk) scenarios. Also, considering the template size, the algorithm showed the best results among all other submissions with the same template size.

Eight scenarios with six datasets were tested. Two experiments were performed in each scenario:

• Identification – intended to test civilian use cases. This experiment used a threshold set to limit the False Positive Identification Rate (FPIR) to 0.3%. FPIR is the proportion of non-mated searches producing one or more candidates above threshold. The False Negative Identification Rates (FNIR) is the proportion of mated searches failing to return the mate above threshold. The use of thresholding supports use of face recognition in making mostly automated decisions e.g. for access into facility.
• Investigation – intended to test law enforcement use cases. This experiment used a threshold set to zero FPIR, and the algorithm should return a fixed number of candidates. FNIR is the proportion of mated searches for which the algorithm does not place the correct candidate at rank 1. The use of face recognition without a threshold supports investigating uses where it is assumed and necessary that a human expert will have to review the candidates returned from each search. For mated-searches the human is tasked with finding the correct mate; for non-mated searches the reviewer must reject all the candidates.

Identification experiments

1. Mugshot vs Mugshot, 12 million gallery
2. Mugshot vs Mugshot, 1.6 million gallery
3. Mugshot vs Webcam, 1.6 million gallery
4. Mugshot vs Profile, 1.6 million gallery
5. Visa vs Border, 1.6 million gallery
6. Visa vs Kiosk, 1.6 million gallery
7. Border vs Border ΔT ≥ 10 years, 1.6 million gallery
8. Mugshot vs Mughsot ΔT ≥ 12 years, 3 million gallery

Investigation experiments

9. Mugshot vs Mugshot, 12 million gallery
10. Mugshot vs Mugshot, 1.6 million gallery
11. Mugshot vs Webcam, 1.6 million gallery
12. Mugshot vs Profile, 1.6 million gallery
13. Visa vs Border, 1.6 million gallery
14. Visa vs Kiosk, 1.6 million gallery
15. Border vs Border ΔT ≥ 10 years, 1.6 million gallery
16. Mugshot vs Mughsot ΔT ≥ 12 years, 3 million gallery

Identification – Neurotechnology algorithm accuracy in this experiment was 0.65% FNIR at 0.3% FPIR. The most accurate contender showed 0.12% FNIR at the same FPIR.

Investigation – Neurotechnology algorithm accuracy in this experiment was 0.13% FNIR at Rank1. The most accurate contender showed 0.10% FNIR at Rank1.

Scenario overview

• 12 million images from the Mugshot dataset were used for persons' enrollment.
• In part of the cases more than one image corresponded a unique person.
• Each face template was compared against all other enrolled templates.
• The comparisons were fully zero-effort, meaning there was no pre-grouping by gender, age or other covariates.

Identification – Neurotechnology algorithm accuracy in this experiment was 0.26% FNIR at 0.3% FPIR. The most accurate contender showed 0.10% FNIR at the same FPIR.

Investigation – Neurotechnology algorithm accuracy in this experiment was 0.10% FNIR at Rank1. The most accurate contender showed 0.08% FNIR at Rank1.

Scenario overview

• 1.6 million images from the Mugshot dataset were used for persons' enrollment.
• In part of the cases more than one image corresponded a unique person.
• Each face template was compared against all other enrolled templates.
• The comparisons were fully zero-effort, meaning there was no pre-grouping by gender, age or other covariates.

Identification – Neurotechnology algorithm accuracy in this experiment was 1.82% FNIR at 0.3% FPIR. The most accurate contender showed 0.65% FNIR at the same FPIR.

Investigation – Neurotechnology algorithm accuracy in this experiment was 0.77% FNIR at Rank1. The most accurate contender showed 0.54% FNIR at Rank1.

Scenario overview

• Images from Mugshot and Webcam datasets were used for persons' enrollment. About 86% of the images were from the Mugshot dataset.
• 1.6 million images from the Mugshot dataset were used.
• In part of the cases more than one image corresponded a unique person.
• Each enrolled face template from the Webcam dataset was compared against all enrolled templates from the Mugshot dataset.
• The comparisons were fully zero-effort, meaning there was no pre-grouping by gender, age or other covariates.

Identification – Neurotechnology algorithm accuracy in this experiment was 16.00% FNIR at 0.3% FPIR. The most accurate contender showed 5.89% FNIR at the same FPIR.

Investigation – Neurotechnology algorithm accuracy in this experiment was 5.81% FNIR at Rank1. The most accurate contender showed 5.17% FNIR at Rank1.

Scenario overview

• Images from Mugshot and Profile datasets were used for persons' enrollment. About 86% of the images were from the Mugshot dataset.
• 1.6 million images from the Mugshot dataset were used.
• In part of the cases more than one image corresponded a unique person.
• Each enrolled face template from the Profile dataset was compared against all enrolled templates from the Mugshot dataset.
• The comparisons were fully zero-effort, meaning there was no pre-grouping by gender, age or other covariates.

Identification – Neurotechnology algorithm accuracy in this experiment was 0.72% FNIR at 0.3% FPIR. The most accurate contender showed 0.16% FNIR at the same FPIR.

Investigation – Neurotechnology algorithm accuracy in this experiment was 0.14% FNIR at Rank1. The most accurate contender showed 0.09% FNIR at Rank1.

Scenario overview

• Images from Visa and Border datasets were used for persons' enrollment.
• 1.6 million images from the Border dataset were used.
• In part of the cases more than one image corresponded a unique person.
• Each enrolled face template from the Border dataset was compared against all enrolled templates from the Visa dataset.
• The comparisons were fully zero-effort, meaning there was no pre-grouping by gender, age or other covariates.

Identification – Neurotechnology algorithm accuracy in this experiment was 24.12% FNIR at 0.3% FPIR. The most accurate contender showed 4.77% FNIR at the same FPIR.

Investigation – Neurotechnology algorithm accuracy in this experiment was 5.10% FNIR at Rank1. The most accurate contender showed 3.87% FNIR at Rank1.

Scenario overview

• Images from Visa and Kiosk datasets were used for persons' enrollment.
• 1.6 million images from the Visa dataset were used.
• In part of the cases more than one image corresponded a unique person.
• Each enrolled face template from the Kiosk dataset was compared against all enrolled templates from the Visa dataset.
• The comparisons were fully zero-effort, meaning there was no pre-grouping by gender, age or other covariates.

Identification – Neurotechnology algorithm accuracy in this experiment was 6.82% FNIR at 0.3% FPIR. The most accurate contender showed 1.23% FNIR at the same FPIR.

Investigation – Neurotechnology algorithm accuracy in this experiment was 1.22% FNIR at Rank1. The most accurate contender showed 0.45% FNIR at Rank1.

Scenario overview

• 1.6 million images from the Border dataset were used for persons' enrollment.
• In part of the cases more than one image corresponded a unique person.
• Age attribute was used to exclude template pairs, which represented same or different persons with age difference of less than 10 years at the capture moment, from the comparison.
• Each face template was compared against all other enrolled templates.

Identification – Neurotechnology algorithm accuracy in this experiment was 2.86% FNIR at 0.3% FPIR. The most accurate contender showed 0.20% FNIR at the same FPIR.

Investigation – Neurotechnology algorithm accuracy in this experiment was 0.39% FNIR at Rank1. The most accurate contender showed 0.23% FNIR at Rank1.

Scenario overview

• 3 million images from the Mugshot dataset were used for persons' enrollment.
• In part of the cases more than one image corresponded a unique person.
• Age attribute was used to exclude template pairs, which represented same or different persons with age difference of less than 12 years at the capture moment, from the comparison.
• Each face template was compared against all other enrolled templates.

These datasets were used for the experiments.