Indicators for evaluation of model performance: irrigation hydraulics applications

ABSTRACT.

Several mathematical models have been developed for applications in the hydraulics of irrigation systems and several performance indicators of these models are used and suggested by the literature. Thus, the objective of this work was to investigate the performance of statistical indicators for the evaluation of models in irrigation hydraulics. For this, three case studies which represent typical irrigation hydraulics modeling were used to assess the indicators. A set of indicators were analyzed: a) difference-based: mean absolute error, mean square error, root mean square error, scaled root mean square error, and percent mean absolute error; b) efficiency-based: Nash-Sutcliffe and Legates-McCabe; c) correlation coefficient (r); d) coefficient of determination (R²); e) index of agreement index (d); f) Camargo and Sentelhas index (c); and g) graphical methods: regression error characteristic curve based on relative absolute error and 1:1 scatter plot. For the evaluated cases, which are physical phenomena, differentiable indicators are similar measures and it is appropriate to report either or both indices. The assessment of models must also be supported by graphical analysis, which shows the real scenario of errors in the model evaluation processes. Efficiency-based indicators, r, R², c, and d are not recommended and should be avoided in modeling of irrigation hydraulics.

Keywords:
accuracy; engineering; error; physical systems; prediction

Introduction

In irrigation engineering, several mathematical models have been developed to assist in the sizing and decision support of hydraulic design of irrigation systems. The evaluation or assessment of models’ performance is an important step when developing mathematical models. The evaluation of a model aims to quantify the deviation between observed and predicted values within the validation limits of the model. A model is suitable when the accuracy of its predictions complies with the application requirements.

Model calibration and validation are fundamental processes for establishing the credibility of models and simulations (Chatterjee & Simonoff, 2013). Quantitative and graphical methods are useful for the correct parameterization and validation of models.

Several statistical indicators and methods have been suggested to assess models’ performance (Nash & Sutcliffe, 1970; Fox, 1981; Willmott, 1981; Ali & Abustan, 2014). Among the statistical indicators, Fox (1981) recommended that the difference-based measures mean absolute error (MAE), mean square error (MSE), and root mean square error (RMSE) should be calculated and reported. Ali and Abustan (2014) also proposed a new difference-based indicator which can be used to evaluate model performance, the percent mean relative absolute error (PMRAE). Willmott (1981) demonstrated that Pearson’s correlation coefficient (r) and determination coefficient (R²) can be misleading and proposed an index of agreement (d). Regarding efficiency-based indicators, the Nash-Sutcliffe (NS_E) and Legates and McCabe (1999) (LM_E) indices are widely used for evaluation of model performance that investigate hydraulic irrigation problems, such as orifice discharge (Zhang, Chai, Li, Xu, & Li, 2019), friction losses in polyethylene pipes (Provenzano, Alagna, Autovino, Juarez, & Rallo, 2016), perforation geometry of drainage pipes (Gaj & Madramootoo, 2020) and channel stability hydraulics (Thompson, Hathaway, & Schwartz, 2018).

Moriasi et al. (2007) pointed out that a model can be assessed as suitable based on one statistic but may present poor performance when evaluated according to another statistic. Furthermore, Alexandrov et al. (2011) emphasized the need for standardized evaluation tools for specific fields. For example, Bellochi, Acuit, Fila, and Donatelli (2002) suggested the use of Pearson’s correlation coefficient, the relative root mean square error, efficiency-based indicators, and t-student probability for solar radiation modeling.

Modeling has been widely used in irrigation hydraulics. However, the literature does not specify the most appropriate statistical indicators to evaluate models capable of representing the physical phenomena in this area. In this paper, statistical indicators were investigated to evaluate the performance of models in irrigation hydraulics, as well as to verify their limitations to help make decisions about the accuracy of the models.

Material and methods

Indicators for model performance evaluation

Table 1 lists the indicators evaluated, as well as their corresponding formulas, range of output values, and basic interpretation. The equations use the following notation: observed values (O_i), predicted values (P_i), number of observations (n), average of observed values (O ̅), and average of predicted values (P ̅).

Table 1
Indicators for model performance evaluation.

Indicator	Formula	Range	Interpretation
Mean Absolute Error (MAE)	$\mathrm{M}\mathrm{A}\mathrm{E}=\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\left|{\mathrm{O}}_i-{\mathrm{P}}_i\right|$	0 to +∞	The lower, the better
Mean Square Error (MSE)	$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{O}}_i-{\mathrm{P}}_i\right)}^2$	0 to +∞	The lower, the better
Root Mean Square Error (RMSE)	$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{O}}_i-{\mathrm{P}}_i\right)}^2}$	0 to +∞	The lower, the better
Scaled Root Mean Square Error (SRMSE)	$\mathrm{S}\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{\stackrel{-}{\mathrm{O}}}\sqrt{\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{O}}_i-{\mathrm{P}}_i\right)}^2}$	0 to +∞	The lower, the better
Percent Mean Absolute Relative Error (PMARE)	$\mathrm{P}\mathrm{M}\mathrm{A}\mathrm{R}\mathrm{E}=\frac{100}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\frac{\left|{\mathrm{O}}_i-{\mathrm{P}}_i\right|}{{\mathrm{O}}_i}$	0% to +∞	The lower, the better
Relative absolute error (δ)	${\mathrm{\delta }}_{\mathrm{i}}=\left|\frac{{\mathrm{O}}_{\mathrm{i}}-{\mathrm{P}}_{\mathrm{i}}}{{\mathrm{O}}_{\mathrm{i}}}\right|$	0 to +∞	The lower, the better
Nash-Sutcliffe (NSE)	x${\mathrm{N}\mathrm{S}}_{\mathrm{E}}=1-\frac{\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{O}}_i-{\mathrm{P}}_i\right)}^2}{\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{O}}_i-\stackrel{-}{\mathrm{O}}\right)}^2}$	-∞ to 1	Near 1 is better
Legates and McCabe (LME)	$\mathrm{L}{\mathrm{M}}_{\mathrm{E}}=1-\frac{\sum _{i=1}^n\left|{\mathrm{O}}_i-{\mathrm{P}}_i\right|}{\sum _{i=1}^n\left|{\mathrm{O}}_i-\stackrel{-}{\mathrm{O}}\right|}$	-∞ to 1	Near 1 is better
Correlation coefficient (r)	$\mathrm{r}=\frac{\sum _{\mathrm{i}=1}^{\mathrm{n}}\left({\mathrm{O}}_i-\stackrel{-}{\mathrm{O}}\right)\left({\mathrm{P}}_i-\stackrel{-}{\mathrm{P}}\right)}{\sqrt{\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{O}}_i-\stackrel{-}{\mathrm{O}}\right)}^2\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{P}}_i-\stackrel{-}{\mathrm{P}}\right)}^2}}$	-1 to 1	-1: perfectly linearly related with a negative slope; 0: no linear dependence; +1: perfectly linearly related with a positive slope.
Coefficient of determination (R²)	${\mathrm{R}}^2=\frac{\mathrm{n}\left(\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{O}}_{\mathrm{i}}{\mathrm{P}}_{\mathrm{i}}\right)-\left(\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{O}}_{\mathrm{i}}\right)(\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{P}}_{\mathrm{i}})}{\sqrt{\left[\mathrm{n}\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{O}}_{\mathrm{i}}^2-{\left(\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{O}}_{\mathrm{i}}\right)}^2\right]\left[\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{P}}_{\mathrm{i}}^2-{\left(\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{P}}_{\mathrm{i}}\right)}^2\right]}}$	0 to 1	The higher, the better
Index of agreement (d)	$\mathrm{d}=1-\frac{\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{O}}_i-{\mathrm{P}}_i\right)}^2}{\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left(\left|{\mathrm{O}}_i-\stackrel{-}{\mathrm{O}}\right|+\left|{\mathrm{P}}_{\mathrm{i}}-\stackrel{-}{\mathrm{O}}\right|\right)}^2}$	0 to 1	The higher, the better
Camargo and Sentelhas index (c)	c - d r	0 to 1	The higher, the better

The first set of indicators shown in Table 1 comprises difference-based indicators, which measure the deviation between observed and predicted values in a data set: MAE, MSE, RMSE, SRMSE, and PMARE. The values of all these indicators range from 0 to +∞, and the lower the values, the better. The smallest value corresponds to the hypothetical situation of no deviation between the predicted and observed data. All these indicators compute absolute or squared deviations between observed and predicted data and do not consider the deviation signal.

The relative absolute error (δ) is applied to analyze deviations between pairs of observed and predicted values. This indicator is useful to draw the regression error characteristic (REC) curve, which is a useful graphical tool to quantify prediction errors associated with their cumulative frequency of occurrence (Sobenko, Bombardelli, Camargo, Frizzone, & Duarte, 2020). δ can be expressed in decimal or percentage units. For any cumulative frequency, the smaller the value of δ, the better. In addition, a scatter plot with a straight line (1:1) and pairs of observed and predicted values is also useful to identify data dispersion, bias, and outliers in the evaluated dataset.

Table 1 also has efficiency-based indicators (NS_E and LM_E), which measure how well a model fits the observed values. Efficiency indicators have a numerator that represents the deviation between observed and predicted values and a denominator that represents the variation of observed values from the average of observed values. The numerator refers to the variation not explained by the model, while the denominator expresses the total variability in the observed data. If the predictions of a linear model are unbiased, then the results of NS_E will lie in the interval from 0 to 1, but it may provide negative values for biased models. For nonlinear models, negative values can be obtained even when the model is unbiased (McCuen, Knight, & Cutter, 2006).

The correlation coefficient (r) is a dimensionless measure of the linear dependence between two data sets. If the two variables are perfectly linearly related, r is 1 (positive slope) or -1 (negative slope). If no linear relationship between the two variables exists, then r is zero.

The coefficient of determination (R²) represents the amount of variability in the data explained by the regression model. R² values near unity do not necessarily imply that the regression model will provide accurate predictions of future observations (Montgomery & Runger, 2013). In general, R² increases when more variables are added to a model, but this does not necessarily imply that increasing the number of variables improves the model performance.

The index of agreement (d) is a measure of the degree to which a model’s predictions are error free and it ranges from 0 to 1. Values near to 1 indicate better agreement between the observed and estimated variables.

The c index is the product of d and r (Camargo & Sentelhas, 1997). Pimenta et al. (2018) proposed a criterion of interpretation and classification of d, r, and c, which will be used in this study.

Data for comparison of indicators

Datasets from three typical problems of irrigation hydraulics were used as case studies to assess the indicators. Methodologies and particularities from each case study are fully described in Pimenta et al. (2018), Katsurayama et al. (2020), and Cano et al. (2021).

Pimenta et al. (2018) used the Colebrook and White (1937) equation (Equation 1) to obtained reference values of the friction factor (f) for pressurized conduits and compared these reference values with values predicted by the equations of Swamee and Jain (1976 - Equation 2) and Shaikh, Massan, and Wagan (2015 - Equation 3) for turbulent flow conditions (4000 ≤R_e≤ 108).

$\frac{1}{\sqrt{\mathrm{f}}}=-2\mathrm{l}\mathrm{o}\mathrm{g}\left(\frac{\mathrm{\epsilon }}{3.7\mathrm{D}}+\frac{2.51}{\mathrm{R}\mathrm{e}\sqrt{\mathrm{f}}}\right)$(1)

$\frac{1}{\sqrt{\mathrm{f}}}=\mathrm{}-2\log \left(\frac{\mathrm{\epsilon }}{3.7\mathrm{D}}+\mathrm{}\frac{5.74}{{\mathrm{R}\mathrm{e}}^{\mathrm{0,9}}}\right)$(2)

$\mathrm{f}=0.25{\left[\mathrm{l}\mathrm{o}\mathrm{g}\left(\frac{\mathrm{\epsilon }}{3.7\mathrm{D}}+\frac{2.51}{\mathrm{\alpha }\mathrm{R}\mathrm{e}}\right)\right]}^{-2}$

$\mathrm{\alpha }={\left[\mathrm{1,14}-2\mathrm{l}\mathrm{o}\mathrm{g}\left(\frac{\mathrm{\epsilon }}{\mathrm{D}}\right)\right]}^{-2}$(3)

where f is the coefficient of head loss of the Darcy-Weisbach formulation (dimensionless), Ɛ/D is the relative roughness of the pipe (m), and Re is the Reynolds number (dimensionless).

For the second case study, Katsurayama et al. (2020) modeled flow characteristics in microtube emitters using experimental data and dimensional analysis. They proposed the model shown in Equation 4 and compared the results with a theoretical model proposed by Souza and Botrel (2004) (Equation 5).

$\mathrm{H}=19.883\mathrm{}{\mathrm{\rho }}^{0.2095}{\mathrm{\mu }}^{0.7905}\frac{{\mathrm{L}}_{\mathrm{m}}^{0.9408}{\mathrm{Q}}_{\mathrm{m}}^{1.2095}}{{\mathrm{D}}_{\mathrm{i}\mathrm{n}}^{1.7314}}$(4)

$\mathrm{H}=\frac{64\mathrm{}\mathrm{\upsilon }\mathrm{}4}{\mathrm{\pi }\mathrm{}2\mathrm{}\mathrm{g}}\left(\frac{{\mathrm{L}}_{\mathrm{m}}\mathrm{}{\mathrm{Q}}_{\mathrm{m}}}{{\mathrm{D}}_{\mathrm{i}\mathrm{n}}^4}\right)+\frac{16}{{\mathrm{\pi }}^2\mathrm{}2\mathrm{}\mathrm{g}}\left(\frac{{\mathrm{Q}}_{\mathrm{m}}^2}{{\mathrm{D}}_{\mathrm{i}\mathrm{n}}^4}\right)+\left[{\mathrm{\alpha }}_{\mathrm{K}}\ln \left(\mathrm{R}\mathrm{e}\right)+{\mathrm{\beta }}_{\mathrm{K}}\frac{16}{{\mathrm{\pi }}^2\mathrm{}2\mathrm{}\mathrm{g}}\left(\frac{{\mathrm{Q}}_{\mathrm{m}}^2}{{\mathrm{D}}_{\mathrm{i}\mathrm{n}}^4}\right)\right]$(5)

where H is the pressure head (m), ρ is the water density (kg m^-3), μ is the water dynamic viscosity (Pa s), L_m is the microtube length (m), Q_m is the microtube flow rate (m³ s^-1), D_in is the microtube internal diameter (m), υ is the water kinematic viscosity (m² s^-1), g is the gravitational acceleration (m s^-2), and α_K and β_K are the empirical coefficients which represent the minor loss coefficients as a function of Re (dimensionless).

The last case evaluated was based on the study carried out by Cano et al. (2021) in the modeling of corner taps’ orifice plates to determine the flow rate in pipes. Based on experimental data, the authors adjusted an empirical equation for orifice plates with an internal diameter of 150 mm (Equation 6). The results were compared with the theoretical equation with the discharge coefficient (C_d - Equation 7) obtained by the Reader-Harris/Gallagher equation for corner taps’ orifice plates (ISO 5167-2, 2003) (Equation 8).

${\mathrm{Q}}_{\mathrm{o}\mathrm{p}}=37.903\mathrm{\Delta }{\mathrm{h}}^{0.606}$(6)

${\mathrm{Q}}_{\mathrm{o}\mathrm{p}}={\mathrm{C}}_{\mathrm{d}}\frac{\mathrm{\pi }{\mathrm{d}}^2}{4}\sqrt{\frac{2\mathrm{g}\mathrm{\Delta }\mathrm{h}}{1-{\mathrm{\beta }}^4}}$(7)

${\text{C}}_{\text{d}}\text{= 0.5961 + 0.0261}{\text{β}}^{\text{2}}\mathrm{}\text{- 0.216}{\text{β}}^{\text{8}}\text{+ 0.000521}{\left(\frac{{\text{10}}^{\text{6}}\text{β}}{\text{Re}}\right)}^{\text{0.7}}$

$\left(\text{0.0188 + 0.0063 A}\right)\mathrm{}{\text{β}}^{\text{3.5}}{\left(\frac{{\text{10}}^{\text{6}}}{\text{Re}}\right)}^{\text{0.3}}$(8)

Where Q_op is the flow through the orifice plate (m³ s^-1), Δh is the differential pressure head on the orifice plate (m), C_d is the orifice plate discharge coefficient (dimensionless), d is the orifice plate internal diameter (m), β is the ratio between the orifice plate and pipe diameters (dimensionless), and A is the coefficient depending on the Reynolds number (dimensionless).

In this way, datasets of 480, 615, and 2,000 records from Pimenta et al. (2018), Katsurayama et al. (2020), and Cano et al. (2021), respectively, were used to test the statistical indicators described above. Both indicators were calculated using an electronic spreadsheet following the equations of the respective indices.

Results and discussion

Table 2 shows the values obtained from the indicators for the predictions of the coefficient of head loss (f - case study 1), microtube length (L_m - case study 2), and flow rate through the orifice plate (Q_op - case study 3). The data points of each case study are, respectively, graphically illustrated in Figures 1, 2, and 3 along with 1:1 lines and REC curves.

Table 2
Indicators to assess the performance of equations in case studies 1, 2, and 3.

Indicator	Case study 1		Case study 2		Case study 3
	Swamee and Jain (1976)	Shaik et al. (2015)	Katsurayama et al. (2020)	Souza and Botrel (2004)	Cano et al. (2021)	ISO 5167-2 (2003)
MAE	0.001	0.003	0.0245	0.0774	1.5425	3.6032
MSE	0.000	0.000	0.0013	0.0085	3.4515	18.0160
RMSE	0.001	0.005	0.0355	0.0923	1.8578	4.2445
SRMSE	0.020	0.149	0.0359	0.0935	0.0217	0.0497
PMARE	0.009	0.102	0.0236	0.1052	0.0198	0.0533
δ(95%)	0.038	0.367	0.0620	0.3840	0.0497	0.1989
NSE	0.999	0.938	0.9907	0.9513	0.9961	0.9705
LME	0.978	0.836	0.9219	0.7514	0.9386	0.8566
r	0.999	0.977	0.9954	0.9844	0.9983	0.9994
R2	0.999	0.954	0.9907	0.9691	0.9967	0.9988
d	0.999	0.971	0.9977	0.9859	0.9990	0.9937
c	0.999	0.949	0.9930	0.9706	0.9973	0.9931

Case study 1 - Coefficient of head loss(f)

For f predictions, Table 2 shows the following: (a) according to the difference-based indicators (MAE, MSE, RMSE, SRMSE, and PMARE), the equation of Swamee and Jain (1976) presented better performance in their predictions, with values closer to zero; (b) the efficiency indicators (NS_E and LM_E) also showed better predictive performance of the same equation as the difference-based, with values closer to unity; (c) composite indicators (d, r, and c) classified both f predictions as "excellent", according to the criterion proposed by Pimenta et al. (2018); (d) through the determination coefficient (R²), it is possible to observe strong correlations between the data of each equation (i.e. values higher than 0.95).

This can be explained by the scatter plot, also called “1:1”, shown in Figure 1a, which illustrates the relationship between observed and predicted values. They also allow for the interpretation of the prediction fitting as over- or underestimates of the observed values. It can be observed that Swamee and Jain's equation overestimated the standard values by only 1.0 ± 1% on average, while the equation of Shaikh et al. (2015) underestimated 67.5% of the observed f data, with an average of 21.2 ± 22% of the observed values. The graph shown in Figure 1b illustrates the relative error (δ) associated with its frequency of occurrence (i.e. regression error characteristic curve). This type of graph can be interpreted in several ways and can present us with some very significant information regarding predictions (Sobenko et al., 2020). Taking as an example the predictions made by the equation of Shaikh et al. (2015), it can be seen that 95% of his predictions had a relative error of up to 36.7% (δ_(95%) in Table 1).

Figure 1
Comparison between equations in predicting the coefficient of head loss (f): (a) standard versus estimated values of f using the models of Swamee and Jain (1976) and Shaikh et al. (2015) and (b) regression error characteristic curve presenting relative errors (δ) versus frequency of error occurrence.

Case study 2 - Modeling of microtube emitters

In the estimation of L_m, the equation proposed by Katsurayama et al. (2020) performed better according to the difference-based and efficiency-based indicators. The composite indicators classified both models evaluated in this case study as "excellent", and the R² also showed strong correlations between the predicted and observed data (R² > 0.95) (Table 1). However, the equation proposed by Souza and Botrel (2004) underestimated 65.4% of the observed values, with underestimations ranging from 0.3 to 341%, and δ_(95%)of 38.4% (Figure 2). Also, from the regression error characteristic curve presented in Figure 2b, it can be observed that 99.2% of the predictions made by the equations of Katsurayama et al. (2020) had a relative error of up to 10%.

Figure 2
Comparison between equations in predicting the microtube length (L_m): (a) observed versus estimated values of Lm using the models of Katsurayama et al. (2020) and Souza and Botrel (2004); and (b) regression error characteristic curve presenting relative errors (δ) versus frequency of error occurrence.

Case study 3 - Orifice plates

In this case, the composite indicators (d, r, and c) and R² suggested that the evaluated equations had the same performance in Q_op predictions (Table 1). The 1:1 graph showed that the methodology of ISO 5167-2 (2003) overestimates 47.5% of the data by 8.0 ± 0.2% and underestimates 52.5% of the data by 3.1 ± 0.1% in relation to the values observed experimentally (Figure 3a). Even more precisely, 95.2 and 73.0% of the predictions made by the equations of Cano et al. (2021) and ISO 5167-2 (2003), respectively, had a relative error of up to 5% (Figure 3b). Thus, this graph shows us how far those points outside the 1:1 line in Figure 3a can be accepted.

Figure 3
Comparison between equations in predicting the flow rate through the orifice plate (Q_op): (a) observed versus estimated values of Q_op using the models of Cano et al. (2021) and ISO 5167-2 (2003); and (b) regression error characteristic curve presenting relative errors (δ) versus frequency of error occurrence.

In essence, MAE presented a large magnitude in the evaluations because it describes the true mean of the deviations but can vary with different data patterns/sets, and MSE is similar to the MAE but more sensitive to large errors, as it squares individual differences (Hallak & Pereira Filho, 2011; Ali & Abustan, 2014). According to Willmott (1981), MAE and RMSE are similar measures which provide estimates of the average error, but neither measure provides information about the relative size of the average difference or the nature (type) of the differences comprising MAE or RMSE. Also, the authors pointed out that MSE and RMSE are generally amenable to more in-depth mathematical or statistical analyses than MAE. Moreover, Ali and Abustan (2014) proposed the PMARE indicator, pointing out that it is capable of directly indicating the accuracy or the pitfalls of the prediction in any field of observation, regardless of the units and ranges of values.

The Nash-Sutcliffe index is based on the squares of differences, while the Legates-McCabe equation is based on the absolute values of differences. From the equations of NS_E and LM_E, it can be observed that they are more dependent on the observation range (O_i and O ̅) than the difference between the observed and predicted values, being more sensitive to the observed range/fluctuation (Willmott, Robseon, & Matsuura, 2011). Thus, in irrigation hydraulics studies, which involve physical phenomena that often do not show dispersion in the observed values, the use of these indicators for model calibration, validation, or testing is not recommended.

The c index offers precision and accuracy from index d and coefficient r, respectively. From Table 1, it can be observed that all predictions made by the evaluated models showed “excellent” classifications for composite indicators and a strong correlation with the observed values, that is, high accuracy (R² > 0.95). Furthermore, the R² does not indicate whether a model provides an adequate fit to the observed data, because it just evaluates the scatter of the data points around the fitted regression line. Due to the ambiguity of these indicators, they are also not recommended for use individually in irrigation hydraulics problems (i.e. in physical processes) and should be avoided. These indicators, as well as efficiency indicators, are widely used in other areas of irrigation, such as irrigation management, which involves studies of evapotranspiration, hydrological modeling, and soil water storage (Shiri & Kisi, 2011; Bachou, Walker, Ticlavilca, & McKee, 2014; Hatiye, Prasad, & Ojha, 2018).

While the indicators give us quantitative measures, graphical methods give the overall and real scenarios, and can be regarded as exploratory tools as well. Indicator sets such as those associated with 1:1 and/or regression error curve analysis graphs have been used to calibrate, validate, and test several empirical, semi-empirical, or deterministic models of different situations in the area of irrigation engineering: dimensional analysis and artificial neural networks (ANN) approaches applied to estimate minor losses due to start connectors in micro-irrigation laterals were assessed with RMSE, MAE, 1:1 graphs, and regression error characteristic curves (Sobenko et al., 2020); linear modeling and ANN techniques used to estimate losses by wind drift and evaporation in sprinkler systems were evaluated by MAE, MSE,RMSE, r, R² and 1:1 graphs (Al-Ghobari, El-Marazky, Dewidar, & Mattar, 2018; Sarwar, Peters, & Mohamed, 2019); RMSE, r, R² and 1:1 graphs were used to evaluate predictions of head loss in micro-irrigation sand filters by ANN techniques (García Nieto et al., 2017); R² and 1:1 graphs were used to assess an empirical local losses estimation model for lay-flat drip laterals (Elbana, Ramírez de Cartagena, & Puig-Bargués, 2013; Provenzano, Di Dio, & Leone, 2014); RMSE, NS_E, and 1:1 graphs were used to evaluate and examine appropriate equations for continuous head loss calculation in real field operating center-pivot systems (Alazba, Mattar, ElNesr, & Amin, 2012); δ, RMSE, d, r, c, and 1:1 graphs were used to update or analyze the performance of equations that estimate the coefficient of head loss (Oke, Ojo, & Adeosun, 2015; Najafzadeh, Shiri, Sadeghi, & Ghaemi, 2018; Pimenta et al., 2018).

In essence, for irrigation hydraulics problems, difference-based indicators are similar measures and it is appropriate, in many cases, to report either or both indices. The diagnosis of a model’s performance must be supported by the quantitative measures and graphical analysis. For this, the statistical indicators should be consistent in their results, just as was reported in this study. Otherwise, the particular indicator is not suitable for model comparison and should be avoided as a model performance measure.

Conclusion

In the process of evaluation of mathematical models in irrigation hydraulics the difference-based indicators assessed are similar measures and can be used individually or as a set. Graphical analyses are essential to identify the magnitude of the error and perform more accurate assessments of the models. Efficiency-based indicators, the correlation coefficient, the coefficient of determination, the index of agreement, and the Camargo and Sentelhas index are not recommended for consideration individually and must be supported by graphical analysis and difference-based indicators.

Acknowledgements

The authors are grateful to the following Brazilian institutions for their financial support: the Federal Department of Science and Technology (MCT), the National Scientific and Technological Development Council (CNPq), and the National Institute of Science and Technology in Irrigation Engineering (INCTEI)

References

• Alazba, A. A., Mattar, M. A., ElNesr, M. N., & Amin, M. T. (2012). Field assessment of friction head loss and friction correction factor equations. Journal of Irrigation and Drainage Engineering, 138(2), 166-176. DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0000387
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0000387
• Alexandrov, G. A., Ames, D., Bellocchi, G., Michael, B., Crout, N., Erechtchoukova, M., … Samaniego, L. (2011). Technical assessment and evaluation of environmental models and software: Letter to the Editor. Environmental Modelling & Software, 26(3), 328-336. DOI: https://doi.org/10.1016/j.envsoft.2010.08.004
  » https://doi.org/https://doi.org/10.1016/j.envsoft.2010.08.004
• Al-Ghobari, H. M., El-Marazky, M. S., Dewidar, A. Z., & Mattar, M. A. (2018). Prediction of wind drift and evaporation losses from sprinkler irrigation using neural network and multiple regression techniques. Agricultural Water Management, 195, 211-221. DOI: https://doi.org/10.1016/j.agwat.2017.10.005
• Ali, M. H., & Abustan, I. (2014). A new novel index for evaluating model performance. Journal of Natural Resources and Development, 4, 1-9. DOI: https://doi.org/10.5027/jnrd.v4i0.01
  » https://doi.org/https://doi.org/10.5027/jnrd.v4i0.01
• Bachour, R., Walker, W. R., Ticlavilca, A. M., & McKee, M. (2014). Estimation of spatially distributed evapotranspiration using remote sensing and a relevance vector machine. Journal of Irrigation and Drainage Engineering , 140(8). DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0000754
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0000754
• Bellocchi, G., Acuit, M., Fila, G., & Donatelli, M. (2002). An indicator of solar radiation model performance based on a fuzzy expert system. Agronomy Journal, 94(6), 1222-1233. DOI: https://doi.org/10.2134/agronj2002.1222
  » https://doi.org/https://doi.org/10.2134/agronj2002.1222
• Camargo, A. P., & Sentelhas, P. C. (1997). Avaliação do desempenho de diferentes métodos de estimativa da evapotranspiração potencial no estado de São Paulo, Brasil. Revista Brasileira Meteorologia, 5(1), 89-97.
• Cano, N. D., Camargo, A. P., Muniz, G. K., Oliveira, J., Dalfré Filho, J. G., & Frizzone, J. A. (2021). Performance of models to determine flowrate using orifice plates. Revista Brasileira de Engenharia Agrícola e Ambiental, 25(1), 10-16. DOI: https://doi.org/10.1590/1807-1929/agriambi.v25n1p10-16
  » https://doi.org/https://doi.org/10.1590/1807-1929/agriambi.v25n1p10-16
• Chatterjee, S., & Simonoff, J. (2013). Handbook of regression analysis Hoboken, NJ: John Wiley and Sons.
• Colebrook, C. F., & White, C. M. (1937). Experiments with fluid friction in roughened pipes. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 161(906), 367-381. DOI: https://doi.org/10.1098/rspa.1937.0150
  » https://doi.org/https://doi.org/10.1098/rspa.1937.0150
• Elbana, M., Ramírez de Cartagena, F., & Puig-Bargués, J. (2013). New mathematical model for computing head loss across sand media filter for microirrigation systems. Irrigation Science, 31, 343-349. DOI: https://doi.org/10.1007/s00271-011-0310-4
  » https://doi.org/https://doi.org/10.1007/s00271-011-0310-4
• Fox, D. G. (1981). Judging air quality model performance. Bulletin of the American Meteorological Society, 62(5), 599-609. DOI: https://doi.org/10.1175/1520-0477(1981)062<0599:JAQMP>2.0.CO;2
  » https://doi.org/https://doi.org/10.1175/1520-0477(1981)062<0599:JAQMP>2.0.CO;2
• Gaj, N., & Madramootoo, C. A. (2020). Effects of perforation geometry on pipe drainage in agricultural lands. Journal of Irrigation and Drainage Engineering , 146(7). DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0001482
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0001482
• García Nieto, P. J., García-Gonzalo, E., Bové, J., Arbat, G., Duran-Ros, M., & Puig-Bargués, J. (2017). Modeling pressure drop produced by different filtering media in microirrigation sand filters using the hybrid ABC-MARS-based approach, MLP neural network and M5 model tree. Computers and Electronics in Agriculture, 139, 65-74. DOI: https://doi.org/10.1016/j.compag.2017.05.008
  » https://doi.org/https://doi.org/10.1016/j.compag.2017.05.008
• Hallak, R., & Pereira Filho, A. J. (2011). Metodologia para análise de desempenho de simulações de sistemas convectivos na região metropolitana de São Paulo com o modelo ARPS: sensibilidade a variações com os esquemas de advecção e assimilação de dados. Revista Brasileira de Meteorologia, 24(4), 591-608. DOI: https://doi.org/10.1590/S0102-77862011000400009
  » https://doi.org/https://doi.org/10.1590/S0102-77862011000400009
• Hatiye, S. D., Prasad, K. S. H., & Ojha, C. S. P. (2018). Deep percolation under irrigated water-intensive crops. Journal of Irrigation and Drainage Engineering , 144(8), 1-13. DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0001326
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0001326
• International Organization for Standardization [ISO]. (2003). ISO 5167-2 - Measurement of fluid flow by means of pressure differential devices inserted in circular-cross section conduits running full - Part 2: Orifice plates Genebra, SW: ISO.
• Katsurayama, G. T., Sobenko, L. R., Camargo, A. P., Botrel, T. A., Frizzone, J. A., & Duarte, S. N. 2020. A mathematical model for hydraulic characterization of microtube emitters using dimensional analysis. Journal of Agricultural Science and Technology, 22(4), 1123-1135.
• Legates, D. R., & McCabe, G. J. (1999). Evaluating the use of “goodness-of fit” measures in hydrologic and hydroclimatic model validation. Water Resources Research, 35(1), 233-241. DOI: https://doi.org/10.1029/1998WR900018
  » https://doi.org/https://doi.org/10.1029/1998WR900018
• McCuen, R. H., Knight, Z., & Cutter, A. G. (2006). Evaluation of the Nash-Sutcliffe efficiency index. Journal of Hydrologic Engineering, 11(6), 597-602. DOI: https://doi.org/10.1061/(ASCE)1084-0699(2006)11:6(597)
  » https://doi.org/https://doi.org/10.1061/(ASCE)1084-0699(2006)11:6(597)
• Montgomery, D. C., & Runger, G. C. (2013). Applied statistics and probability for engineers (6th ed.). Hoboken, NJ: Wiley.
• Moriasi, D. N., Arnold, J. G., Van Liew, M. W., Bingner, R. L., Harmel, R. D., & Veith, T. L. (2007). Model evaluation guidelines for systematic quantification of accuracy in watershed simulations. Transactions of ASABE, 50(3), 885-900. DOI: https://doi.org/10.13031/2013.23153
  » https://doi.org/https://doi.org/10.13031/2013.23153
• Nash, J. E., & Sutcliffe, J. V. (1970). River flow forecasting through conceptual models part I - A discussion of principles. Journal of Hydrology, 10(3), 282-290. DOI: https://doi.org/10.1016/0022-1694(70)90255-6
  » https://doi.org/https://doi.org/10.1016/0022-1694(70)90255-6
• Najafzadeh, M., Shiri, J., Sadeghi, G., & Ghaemi, A. (2018). Prediction of the friction factor in pipes using model tree. Journal of Hydraulic Engineering, 24(1), 9-15. DOI: https://doi.org/10.1080/09715010.2017.1333926
  » https://doi.org/https://doi.org/10.1080/09715010.2017.1333926
• Oke, I. A., Ojo, S. O., & Adeosun, O. O. (2015). Performance evaluation for Darcy friction factor formulae using Colebrook-White as reference. Ife Journal of Science, 17(1), 75-86.
• Pimenta, B. D., Robaina, A. D., Peiter, M. X., Mezzomo, W., Kirchner, J. H., & Ben, L. H. B. (2018). Performance of explicit approximations of the coefficient of head loss for pressurized conduits. Revista Brasileira de Engenharia Agrícola e Ambiental , 22(5), 301-307. DOI: https://doi.org/10.1590/18071929/agriambi.v22n5p301-307
  » https://doi.org/https://doi.org/10.1590/18071929/agriambi.v22n5p301-307
• Provenzano, G., Di Dio, P. M., & Leone, R. (2014). Assessing a local losses evaluation procedure for low-pressure lay-flat drip laterals. Journal of Irrigation and Drainage Engineering , 140(6). DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0000731
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0000731
• Provenzano, G., Alagna, V., Autovino, D., Juarez, J. M., & Rallo, G. (2016). Analysis of geometrical relationships and friction losses in small-diameter lay-flat polyethylene pipes. Journal of Irrigation and Drainage Engineering , 142(2), 1-9. DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0000958
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0000958
• Sarwar, A., Peters, R. T., & Mohamed, A. Z. (2019). Linear mixed modeling and artifcial neural network techniques for predicting wind drift and evaporation losses under moving sprinkler irrigation systems. Irrigation Science , 38, 177-188. DOI: https://doi.org/10.1007/s00271-019-00659-x
  » https://doi.org/https://doi.org/10.1007/s00271-019-00659-x
• Shaikh, M. M., Massan, S. R., & Wagan, A. I. (2015). A new explicit approximation to Colebrook’s friction factor in rough pipes under highly turbulent cases. International Journal of Heat and Mass Transfer, 88, 538-543. DOI: https://doi.org/10.1016/j.ijheatmasstransfer.2015.05.006
  » https://doi.org/https://doi.org/10.1016/j.ijheatmasstransfer.2015.05.006
• Shiri, J., & Kisi, Ö. (2011). Application of artificial intelligence to estimate daily pan evaporation using available and estimated climatic data in the Khozestan Province (South Western Iran). Journal of Irrigation and Drainage Engineering , 137, 412-425. DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0000315
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0000315
• Sobenko, L. R., Bombardelli, W. W. A., Camargo, A. P., Frizzone, J. A., & Duarte, S. N. (2020). Minor losses through start connectors in microirrigation laterals: dimensional analysis and artificial neural networks approaches. Journal of Irrigation and Drainage Engineering , 146(5), 1-13. DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0001466
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0001466
• Souza, R. O. R. M., & Botrel, T. A. (2004). Modeling for the design of microtubes in trickle irrigation. Revista Brasileira de Engenharia Agrícola e Ambiental , 8(1), 16-22. DOI: https://doi.org/10.1590/S1415-43662004000100003
  » https://doi.org/https://doi.org/10.1590/S1415-43662004000100003
• Swamee, P. K., & Jain, A. K. (1976). Explicit equations for pipe flow problems. Journal of the Hydraulics Division, 102, 657-664.
• Thompson, M. J., Hathaway, J. M., & Schwartz, J. S. (2018). Three-dimensional modeling of the hydraulic function and channel stability of regenerative stormwater conveyances. Journal of Sustainable Water in the Built Environment, 4(3). DOI: https://doi.org/10.1061/ JSWBAY.0000861
  » https://doi.org/https://doi.org/10.1061/ JSWBAY.0000861
• Willmott, C. J. (1981). On the validation of models. Physical Geography, 2, 184-194. DOI: https://doi.org/10.1080/02723646.1981.10642213
  » https://doi.org/https://doi.org/10.1080/02723646.1981.10642213
• Willmott, C. J., Robeson, S. M., & Matsuura, K. (2011). A refined index of model performance. International Journal of Climatology, 32(13), 2088-2094. DOI: https://doi.org/10.1002/joc.2419
  » https://doi.org/https://doi.org/10.1002/joc.2419
• Zhang, Z., Chai, J., Li, Z., Xu, Z., & Li, P. (2019). Discharge coefficient of a spillway with a riser perforated by rectangular orifices. Journal of Irrigation and Drainage Engineering , 145(11). DOI: https://doi.org/10.1061/(ASCE)IR.1943-4774.0001425
  » https://doi.org/https://doi.org/10.1061/(ASCE)IR.1943-4774.0001425

Publication Dates

• Publication in this collection
  28 Oct 2022
• Date of issue
  2023

History

• Received
  21 Oct 2020
• Accepted
  10 Feb 2021