ICML2022: How Faithful is your Synthetic Data?

Описание: This is a quick intro to our ICML 2022 paper “How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models” by Ahmed M. Alaa, Boris van Breugel, Evgeny S. Saveliev and Mihaela van der Schaar.

Abstract:
Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric,(alpha-Precision, beta-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data—a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low quality samples and hence improving the overall model performance in a post-hoc manner.

Paper: http://link.vanderschaar-lab.com/icml...
Code:
https://github.com/vanderschaarlab/ev...
https://github.com/ahmedmalaa/evaluat...
ArXiv: https://arxiv.org/abs/2102.08921

The van der Schaar Lab:
Website: https://www.vanderschaar-lab.com/
LinkedIn:   / mihaela-van-der-schaar  
Twitter:   / mihaelavds  
YouTube:    / vanderschaarlab  
GitHub: https://github.com/vanderschaarlab

Authors:

Ahmed M. Alaa
Twitter:   / _ahmedmalaa  
LinkedIn:   / ahmed-m-alaa-b1007614  
GitHub: https://github.com/ahmedmalaa
Google Scholar: https://scholar.google.com/citations?...

Boris van Breugel
LinkedIn:   / boris-van-breugel-728405124  
GitHub: https://github.com/bvanbreugel
Google Scholar: https://scholar.google.com/citations?...

Evgeny S. Saveliev
Twitter:   / essaveliev  
LinkedIn:   / e-s-saveliev  
GitHub: https://github.com/DrShushen
Google Scholar: https://scholar.google.com/citations?...

Mihaela van der Schaar
Twitter:   / mihaelavds  
LinkedIn:   / mihaela-van-der-schaar  
GitHub: https://github.com/vanderschaarlab
Google Scholar: https://scholar.google.com/citations?...

#ICML2022 #machinelearning #generativemodels #syntheticdata #artificialintelligence