MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

• Main Authors: Shen, Leyang, Chen, Gongwei, Shao, Rui, Guan, Weili, Nie, Liqiang
• Format: Journal Article
• Language: English
• Published: 17-07-2024
• Subjects: Computer Science - Computer Vision and Pattern Recognition
• DOI: 10.48550/arxiv.2407.12709

Multimodal large language models (MLLMs) have demonstrated impressive
capabilities across various vision-language tasks. However, a generalist MLLM
typically underperforms compared with a specialist MLLM on most VL tasks, which
can be attributed to task interference. In this paper, we propose a mixture of
multimodal experts (MoME) to mitigate task interference and obtain a generalist
MLLM. Our MoME is composed of two key components, a mixture of vision experts
(MoVE) and a mixture of language experts (MoLE). MoVE can adaptively modulate
the features transformed from various vision encoders, and has a strong
compatibility in transformation architecture. MoLE incorporates sparsely gated
experts into LLMs to achieve painless improvements with roughly unchanged
inference costs. In response to task interference, our MoME specializes in both
vision and language modality to adapt to task discrepancies. Extensive
experiments show that MoME significantly improves the performance of generalist
MLLMs across various VL tasks. The source code is released at
https://github.com/JiuTian-VL/MoME