Anna: A KVS for Any Scale

• Published in: 2018 IEEE 34th International Conference on Data Engineering (ICDE) pp. 401 - 412
• Main Authors: Chenggang Wu, Faleiro, Jose, Yihan Lin, Hellerstein, Joseph
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-04-2018
• Subjects: consistency; coordination free; distributed storage system; Hardware; high performance; key value store; lattice; Lattices; Message systems; Multicore processing; partitioning; Programming; replication; Scalability; Servers
• ISSN: 2375-026X
• DOI: 10.1109/ICDE.2018.00044

Modern cloud providers offer dense hardware with multiple cores and large memories, hosted in global platforms. This raises the challenge of implementing high-performance software systems that can effectively scale from a single core to multicore to the globe. Conventional wisdom says that software designed for one scale point needs to be rewritten when scaling up by 10-100X. In contrast, we explore how a system can be architected to scale across many orders of magnitude by design. We explore this challenge in the context of a new key-value store system called Anna: a partitioned, multi-mastered system that achieves high performance and elasticity via wait-free execution and coordination-free consistency. Our design rests on a simple architecture of coordination-free actors that perform state update via merge of lattice-based composite data structures. We demonstrate that a wide variety of consistency models can be elegantly implemented in this architecture with unprecedented consistency, smooth fine-grained elasticity, and performance that far exceeds the state of the art.