A Big Data Cleaning Method for Drinking-Water Streaming Data

Abstract

A HA_Cart_AdaBoost model is proposed to clean the data in drinking-water-quality data. First, the data that do not follow the normal distribution are regarded as outliers and eliminated. Next, the optimal control theory of nonlinear partial differential equations (PDEs) is introduced into the cart decision tree, and the cart decision with the specified depth is used. As a weak classifier of AdaBoost, the tree uses the HA_Cart_AdaBoost model to compensate for the eliminated data, then it fits and predicts the missing values of the data stream, realizes the cleaning of drinking-water-quality data, and finally uses the big data Hadoop architecture for real-time storage and analysing streaming data. The experimental results show that compared with the most advanced data cleaning methods, after the optimal control theory of nonlinear PDEs is introduced into the cart decision tree, the stability and accuracy of the HA_Cart_AdaBoost model for water quality data cleaning are greatly improved. Taking pH as an example, the HA_Cart_AdaBoost model shows a minimum improvement of 2.25% and a maximum improvement of 53.33% in terms of RMSE, and a minimum improvement of 13.51% and a maximum improvement of 78.08% in terms of MAE.

Keywords:
CART decision tree; partial differential equation; water data cleaning; AdaBoost algorithm; Hadoop architecture.

HIGHLIGHTS

• Proposing a drinking-water data cleaning model with the combination of nonlinear partial differential equations and the CART decision tree.

• Preprocessing the data by removing outlier elements not following the normal distribution.

• Preprocessing the missing data by a modified classifier of AdaBoost.

• Implementing the proposed model with Big Data technology based on Hadoop architecture.

• Overall performance of the proposed method competes with other similar cutting-edge methods.