Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention

• Main Authors: Lee, Kwanhyung, Lee, Soojeong, Hahn, Sangchul, Hyun, Heejung, Choi, Edward, Ahn, Byungeun, Lee, Joohyung
• Format: Journal Article
• Language: English
• Published: 03-05-2023
• Subjects: Computer Science - Learning
• DOI: 10.48550/arxiv.2305.02504

Electronic Health Record (EHR) provides abundant information through various
modalities. However, learning multi-modal EHR is currently facing two major
challenges, namely, 1) data embedding and 2) cases with missing modality. A
lack of shared embedding function across modalities can discard the temporal
relationship between different EHR modalities. On the other hand, most EHR
studies are limited to relying only on EHR Times-series, and therefore, missing
modality in EHR has not been well-explored. Therefore, in this study, we
introduce a Unified Multi-modal Set Embedding (UMSE) and Modality-Aware
Attention (MAA) with Skip Bottleneck (SB). UMSE treats all EHR modalities
without a separate imputation module or error-prone carry-forward, whereas MAA
with SB learns missing modal EHR with effective modality-aware attention. Our
model outperforms other baseline models in mortality, vasopressor need, and
intubation need prediction with the MIMIC-IV dataset.