Marco Zaffalon, Giorgio Corani, Denis Mauá

---

Utility-Based Accuracy Measures to Empirically Evaluate Credal Classifiers

Abstract

Predictions made by imprecise-probability models are often indeterminate (that is, set-valued). Measuring the quality of an indeterminate prediction by a single number is important to fairly compare different models, but a principled approach to this problem is currently missing. In this paper we derive a measure to evaluate the predictions of credal classifiers from a set of assumptions. The measure turns out to be made of an objective component, and another that is related to the decision-maker's degree of risk-aversion. We discuss when the measure can be rendered independent of such a degree, and provide insights as to how the comparison of classifiers based on the new measure changes with the number of predictions to be made. Finally, we empirically study the behavior of the proposed measure.

Keywords

Credal classification, indeterminacy, empirical evaluations, discounted accuracy, utility, risk-aversion.

---

Download area

The paper is available in the following formats:

• Gzipped pdf file
• pdf file

---

Authors’ addresses

Marco Zaffalon
Galleria 2
CH-6928 Manno
Switzerland

Giorgio Corani
IDSIA
CH-6928 Manno
Lugano
Switzerland

Denis Mauá
Galleria 2, CH-6928 Manno.

E-mail addresses

Marco Zaffalon		zaffalon@idsia.ch
Giorgio Corani		giorgio@idsia.ch
Denis Mauá		denis@idsia.ch