[2010.03039] Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Proceduresopen searchopen navigation menucontact arXivsubscribe to arXiv mailings

Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model's uncertainty is evaluated using point-prediction metrics such as negative log-likelihood or the Brier score on heldout data. In this study, we provide the first large scale evaluation of the empirical frequentist coverage properties of well known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and establish coverage as an important metric in developing models for real-world applications.

1 mentions: @KyleCranmer
Keywords: deep learning
Date: 2020/11/18 21:51

Related Entries

Read more [1805.12244] Mining gold from implicit models to improve likelihood-free inferencecontact arXivarXiv...
0 users, 1 mentions 2020/01/22 00:51
Read more Data Science @ LHC 2015 Workshop (9-13 November 2015): Overview · Indico
0 users, 1 mentions 2020/05/13 05:20
Read more Towards fairness in ML with adversarial networks - GoDataDriven
0 users, 1 mentions 2020/06/10 21:51
Read more Publications | Institute for Research and Innovation in Software for High Energy Physics
0 users, 1 mentions 2020/06/15 20:21
Read more Provably exact artificial intelligence for nuclear and particle physics - ScienceBlog.com
0 users, 3 mentions 2020/09/24 18:19