Enhancing water system models by integrating big data

• Published in: Sustainable cities and society Vol. 37; pp. 485 - 491
• Main Authors: Shafiee, M. Ehsan, Barker, Zachary, Rasekh, Amin
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01-02-2018
• Subjects: Apache Spark; Automation; Big data; Cloud computing; Hadoop; Modeling; Water systems; Cloud computing; Automation; Hadoop; Water systems; Big data; Modeling; Apache Spark
• ISSN: 2210-6707; 2210-6715
• DOI: 10.1016/j.scs.2017.11.042

•Describe, highlight the benefit of big data, and study some applications of big data.•Identify the challenges of integrating big data with water computer models.•Propose a generic model for integration of water computer models with the big data.•Support the study with previous applications of big data.
The past quarter century has witnessed development of advanced modeling approaches, such as stochastic and agent-based modeling, to sustainably manage water systems in the presence of deep uncertainty and complexity. However, all too often data inputs for these powerful models are sparse and outdated, yielding unreliable results. Advancements in sensor and communication technologies have allowed for the ubiquitous deployment of sensors in water resources systems and beyond, providing high-frequency data. Processing the large amount of heterogeneous data collected is non-trivial and exceeds the capacity of traditional data warehousing and processing approaches. In the past decade, significant advances have been made in the storage, distribution, querying, and analysis of big data. Many tools have been developed by computer and data scientists to facilitate the manipulation of large datasets and create pipelines to transmit the data from data warehouses to computational analytic tools. A generic framework is presented to complete the data cycle for a water system. The data cycle presents an approach for integrating high-frequency data into existing water-related models and analyses, while highlighting some of the more helpful data management tools. The data tools are helpful to make sustainable decisions, which satisfy the objectives of a society. Data analytics distribution tool Spark is introduced through the illustrative application of coupling high-frequency demand metering data with a water distribution model. By updating the model in near real-time, the analysis is more accurate and can expose serious misinterpretations.