Data transformation and migration in polystores

• Published in: 2016 IEEE High Performance Extreme Computing Conference (HPEC) pp. 1 - 6
• Main Authors: Dziedzic, Adam, Elmore, Aaron J., Stonebraker, Michael
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-09-2016
• Subjects: Arrays; Data mining; Distributed databases; Engines; Relational databases; Transforms
• DOI: 10.1109/HPEC.2016.7761594

Ever increasing data size and new requirements in data processing has fostered the development of many new database systems. The result is that many data-intensive applications are underpinned by different engines. To enable data mobility there is a need to transfer data between systems easily and efficiently. We analyze the state-of-the-art of data migration and outline research opportunities for a rapid data transfer. Our experiments explore data migration between a diverse set of databases, including PostgreSQL, SciDB, S-Store and Accumulo. Each of the systems excels at specific application requirements, such as transactional processing, numerical computation, streaming data, and large scale text processing. Providing an efficient data migration tool is essential to take advantage of superior processing from that specialized databases. Our goal is to build such a data migration framework that will take advantage of recent advancement in hardware and software.