Towards Knowledge Uncertainty Estimation for Open Set Recognition

Research output: Contribution to journalArticlepeer-review

Abstract

Uncertainty is ubiquitous and happens in every single prediction of Machine Learning models. The ability to estimate and quantify the uncertainty of individual predictions is arguably relevant, all the more in safety-critical applications. Real-world recognition poses multiple challenges since a model's knowledge about physical phenomenon is not complete, and observations are incomplete by definition. However, Machine Learning algorithms often assume that train and test data distributions are the same and that all testing classes are present during training. A more realistic scenario is the Open Set Recognition, where unknown classes can be submitted to an algorithm during testing. In this paper, we propose a Knowledge Uncertainty Estimation (KUE) method to quantify knowledge uncertainty and reject out-of-distribution inputs. Additionally, we quantify and distinguish aleatoric and epistemic uncertainty with the classical information-theoretical measures of entropy by means of ensemble techniques. We performed experiments on four datasets with different data modalities and compared our results with distance-based classifiers, SVM-based approaches and ensemble techniques using entropy measures. Overall, the effectiveness of KUE in distinguishing in- and out-distribution inputs obtained better results in most cases and was at least comparable in others. Furthermore, a classification with rejection option based on a proposed combination strategy between different measures of uncertainty is an application of uncertainty with proven results.
Original languageEnglish
Pages (from-to)505-532
JournalMACHINE LEARNING AND KNOWLEDGE EXTRACTION
Volume2
Issue number4
DOIs
Publication statusPublished - Dec 2020

Keywords

  • uncertainty
  • machine learning
  • open set recognition
  • entropy
  • out-of-distribution

Fingerprint

Dive into the research topics of 'Towards Knowledge Uncertainty Estimation for Open Set Recognition'. Together they form a unique fingerprint.

Cite this