Reliability Tests

We present the testing results to show the template verification reliability evaluations for the MegaMatcher ID system. The tests are available below for each supported biometric modality:

• Face template verification reliability evaluation
• Voice template verification reliability evaluation
• Slap (contactless fingerprints) template verification reliability evaluation

The face liveness check algorithm was tested by iBeta and certified as compliant with ISO 30107-3 Biometric Presentation Attack Detection Standards.

University of Massachusetts Labeled Faces in the Wild (LFW) public dataset was used for testing the face template verification reliability of Neurotechnology MegaMatcher ID system. The following modifications were applied to the dataset:

• According to the original protocol, only 6,000 pairs (3,000 genuine and 3,000 impostor) should be used to report the results. But recent algorithms are "very close to the maximum achievable by a perfect classifier" [source]. Instead, as Neurotechnology algorithms were not trained on any image from this dataset, verification results on matching each pair of all 13,233 face images of 5,729 persons were chosen to be reported.
• All identity mistakes, which had been mentioned on the LFW website, were fixed. Also, several not mentioned issues were fixed.
• Some images from the LFW dataset contained multiple faces. The correct faces for assigned identities were chosen manually to solve these ambiguities.

Three experiments were performed with each dataset:

• Experiment 1 used face templates which were extracted with settings for mobile device usage optimization. The MegaMatcher ID 2024.1 algorithm reliability in this experiment is shown on the ROC chart as black curve.
• Experiment 2 used face templates which were extracted with settings for server platform usage optimization. The MegaMatcher ID 2024.1 algorithm reliability in this experiment is shown on the ROC chart as red curve.
• Experiment 3 used face templates from Experiments 1 and 2 to test interoperability between them. The MegaMatcher ID 2024.1 algorithm reliability in this experiment is shown on the ROC chart as green curve.

Receiver operation characteristic (ROC) curves are usually used to demonstrate the recognition quality of an algorithm. ROC curves show the dependence of false rejection rate (FRR) on the false acceptance rate (FAR). Equal error rate (EER) is the rate at which both FAR and FRR are equal.

Click to zoom

The MegaMatcher ID 2024.1 voice template matching algorithm has been tested with voice samples taken from the XM2VTS Database, as well as with voice samples from Neurotechnology's internal dataset. Two datasets included fixed phrases pronounced by all subjects within a particular dataset, and one dataset included unique phrases for each subject.

Gallery and probe were populated in this way:

• Gallery – each voice sample from each source dataset was truncated to 9 seconds by removing excessive part at the end.
• Probe – each voice sample from each source dataset was processed in this way:
  • Each voice sample was truncated to 9 seconds by removing excessive part at the end and added to the probe
  • The truncated 9-second samples were cut into three 3-second samples, thus adding three additional samples out of each original sample to the probe.
  • The truncated 9-second samples were truncated again to 6 seconds by removing excessive part at the end and added to the probe.
  • The truncated 9-second samples were truncated again to 6 seconds by removing excessive part at the beginning and added to the probe.

Two tests were performed during each experiment:

• Test 1 used compact template extraction model, designed for Android platform. The reliability of matching the extracted templates is shown as red curves on the ROC charts.
• Test 2 used large template extraction model, designed for Server platform. The reliability of matching the extracted templates is shown as blue curves on the ROC charts.

Experiment 1

Experiment 2

Experiment 3

Receiver operation characteristic (ROC) curves are usually used to demonstrate the recognition quality of an algorithm. ROC curves show the dependence of false rejection rate (FRR) on the false acceptance rate (FAR). Charts with ROC curves for each of the experiments are available above.

The MegaMatcher ID 2024.1 system's slap biometric engine has been tested using three Neurotechnology internal datasets, captured with different devices. All datasets contained biometric data belonging to the same persons, so one dataset could be used as a gallery, and another as a probe. These datasets were used:

• Dataset 1 – four-finger slaps from both left and right hands, captured in a contactless way with Google Puxel 4a (5G) smartphone camera as photos.
• Dataset 2 – left hand four-finger slaps, captured in a contact way with several different fingerprint readers and saved as images
• Dataset 3 – right hand four-finger slaps, captured in a contact way with several different fingerprint readers and saved as images

Four experiments were performed during rhe testing:

• Experiment 1 – all slaps from the Dataset 1 were compared between each other.
• Experiment 2 – the Dataset 1 was used as probe, and the Dataset 2 was used as gallery.
• Experiment 3 – the Dataset 1 was used as probe, and the Dataset 3 was used as gallery.
• Experiment 4 – the Dataset 1 was used as probe, and both Dataset 2 and Dataset 3 were used as gallery.

Receiver operation characteristic (ROC) curves are usually used to demonstrate the recognition quality of an algorithm. ROC curves show the dependence of false rejection rate (FRR) on the false acceptance rate (FAR). The charts below compare the Experiment 1 results with the results of other experiments.

Experiment 1 vs. Experiment 2

Experiment 1 vs. Experiment 3

Experiment 1 vs. Experiment 4