Evaluating generative networks using Gaussian mixtures of image features

• Main Authors: Luzi, Lorenzo, Marrero, Carlos Ortiz, Wynar, Nile, Baraniuk, Richard G, Henry, Michael J
• Format: Journal Article
• Language: English
• Published: 08-10-2021
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Learning
• DOI: 10.48550/arxiv.2110.05240

We develop a measure for evaluating the performance of generative networks
given two sets of images. A popular performance measure currently used to do
this is the Fr\'echet Inception Distance (FID). FID assumes that images
featurized using the penultimate layer of Inception-v3 follow a Gaussian
distribution, an assumption which cannot be violated if we wish to use FID as a
metric. However, we show that Inception-v3 features of the ImageNet dataset are
not Gaussian; in particular, every single marginal is not Gaussian. To remedy
this problem, we model the featurized images using Gaussian mixture models
(GMMs) and compute the 2-Wasserstein distance restricted to GMMs. We define a
performance measure, which we call WaM, on two sets of images by using
Inception-v3 (or another classifier) to featurize the images, estimate two
GMMs, and use the restricted $2$-Wasserstein distance to compare the GMMs. We
experimentally show the advantages of WaM over FID, including how FID is more
sensitive than WaM to imperceptible image perturbations. By modelling the
non-Gaussian features obtained from Inception-v3 as GMMs and using a GMM
metric, we can more accurately evaluate generative network performance.