Beyond Accuracy: Automated De-Identification of Large Real-World Clinical Text Datasets

• Main Authors: Kocaman, Veysel, Haq, Hasham Ul, Talby, David
• Format: Journal Article
• Language: English
• Published: 13-12-2023
• Subjects: Computer Science - Computation and Language; Computer Science - Cryptography and Security; Computer Science - Learning
• DOI: 10.48550/arxiv.2312.08495

Recent research advances achieve human-level accuracy for de-identifying
free-text clinical notes on research datasets, but gaps remain in reproducing
this in large real-world settings. This paper summarizes lessons learned from
building a system used to de-identify over one billion real clinical notes, in
a fully automated way, that was independently certified by multiple
organizations for production use. A fully automated solution requires a very
high level of accuracy that does not require manual review. A hybrid
context-based model architecture is described, which outperforms a Named Entity
Recogniton (NER) - only model by 10% on the i2b2-2014 benchmark. The proposed
system makes 50%, 475%, and 575% fewer errors than the comparable AWS, Azure,
and GCP services respectively while also outperforming ChatGPT by 33%. It
exceeds 98% coverage of sensitive data across 7 European languages, without a
need for fine tuning. A second set of described models enable data obfuscation
-- replacing sensitive data with random surrogates -- while retaining name,
date, gender, clinical, and format consistency. Both the practical need and the
solution architecture that provides for reliable & linked anonymized documents
are described.