metrics
Compute surrogate quality metrics
Topics
surrogate_models
Specification
Alias: diagnostics
Arguments: STRINGLIST
Default: No diagnostics
Child Keywords:
Required/Optional |
Description of Group |
Dakota Keyword |
Dakota Keyword Description |
---|---|---|---|
Optional |
Perform k-fold cross validation |
||
Optional |
Leave-one-out cross validation |
Description
Diagnostic metrics assess the goodness of fit of a global surrogate to its training data.
The default diagnostics are:
root_mean_squared
mean_abs
rsquared
Additional available diagnostics include
sum_squared
mean_squared
sum_abs
max_abs
The keywords press
and cross_validation
further specify
leave-one-out or k-fold cross validation, respectively, for all of the
active metrics from above.
Usage Tips
When specified, the metrics
keyword must be followed by a list of
quoted strings, each of which activates a metric.
Examples
This example input fragment constructs a quadratic polynomial surrogate and computes four metrics on the fit, both with and without 5-fold cross validation. (Also see dakota/share/dakota/test/dakota_surrogate_import.in for additional examples.)
model
surrogate global
polynomial quadratic
metrics = "root_mean_squared" "sum_abs" "mean_abs" "max_abs"
cross_validation folds = 5
Theory
Most of these diagnostics refer to some operation on the residuals (the difference between the surrogate model and the truth model at the data points upon which the surrogate is built).
For example, sum_squared
refers to the sum of the squared
residuals, and mean_abs
refers to the mean of the absolute value of
the residuals. rsquared
refers to the R-squared value typically
used in regression analysis (the proportion of the variability in the
response that can be accounted for by the surrogate model). Care
should be taken when interpreting metrics, for example, errors may be
near zero for interpolatory models or rsquared may not be applicable
for non-polynomial models.