The Elusive Pursuit of Replicating PATE-GAN: Benchmarking, Auditing, Debugging

• Main Authors: Ganev, Georgi, Annamalai, Meenatchi Sundaram Muthu Selva, De Cristofaro, Emiliano
• Format: Journal Article
• Language: English
• Published: 20-06-2024
• Subjects: Computer Science - Cryptography and Security; Computer Science - Learning
• DOI: 10.48550/arxiv.2406.13985

Synthetic data created by differentially private (DP) generative models is
increasingly used in real-world settings. In this context, PATE-GAN has emerged
as a popular algorithm, combining Generative Adversarial Networks (GANs) with
the private training approach of PATE (Private Aggregation of Teacher
Ensembles). In this paper, we analyze and benchmark six open-source PATE-GAN
implementations, including three by (a subset of) the original authors. First,
we shed light on architecture deviations and empirically demonstrate that none
replicate the utility performance reported in the original paper. Then, we
present an in-depth privacy evaluation, including DP auditing, showing that all
implementations leak more privacy than intended and uncovering 17 privacy
violations and 5 other bugs. Our codebase is available from
https://github.com/spalabucr/pategan-audit.