.. _model-surrogate-global-neural_network:

""""""""""""""
neural_network
""""""""""""""


Artificial neural network model


.. toctree::
   :hidden:
   :maxdepth: 1

   model-surrogate-global-neural_network-max_nodes
   model-surrogate-global-neural_network-range
   model-surrogate-global-neural_network-random_weight
   model-surrogate-global-neural_network-export_model
   model-surrogate-global-neural_network-import_model


**Specification**

- *Alias:* None

- *Arguments:* None


**Child Keywords:**

+-------------------------+--------------------+--------------------+-----------------------------------------------+
| Required/Optional       | Description of     | Dakota Keyword     | Dakota Keyword Description                    |
|                         | Group              |                    |                                               |
+=========================+====================+====================+===============================================+
| Optional                                     | `max_nodes`__      | Maximum number of hidden layer nodes          |
+----------------------------------------------+--------------------+-----------------------------------------------+
| Optional                                     | `range`__          | Range for neural network random weights       |
+----------------------------------------------+--------------------+-----------------------------------------------+
| Optional                                     | `random_weight`__  | (Inactive) Random weight control              |
+----------------------------------------------+--------------------+-----------------------------------------------+
| Optional                                     | `export_model`__   | Exports surrogate model in user-specified     |
|                                              |                    | format(s)                                     |
+----------------------------------------------+--------------------+-----------------------------------------------+
| Optional                                     | `import_model`__   | Import surrogate model from archive file      |
+----------------------------------------------+--------------------+-----------------------------------------------+

.. __: model-surrogate-global-neural_network-max_nodes.html
__ model-surrogate-global-neural_network-range.html
__ model-surrogate-global-neural_network-random_weight.html
__ model-surrogate-global-neural_network-export_model.html
__ model-surrogate-global-neural_network-import_model.html



**Description**


Dakota's artificial neural network surrogate is a stochastic layered
perceptron network, with a single hidden layer.  Weights for the input
layer are chosen randomly, while those in the hidden layer are
estimated from data using a variant of the Zimmerman direct training
approach :cite:p:`Zim96`.

This typically yields lower training cost than traditional neural
networks, yet good out-of-sample performance.  This is helpful in
surrogate-based optimization and optimization under uncertainty, where
multiple surrogates may be repeatedly constructed during the
optimization process, e.g., a surrogate per response function, and a
new surrogate for each optimization iteration.

The neural network is a non parametric surface fitting method. Thus,
along with Kriging (Gaussian Process) and MARS, it can be used to
model data trends that have slope discontinuities as well as multiple
maxima and minima. However, unlike Kriging, the neural network
surrogate is not guaranteed to interpolate the data from which it was
constructed.

This surrogate can be constructed from fewer than :math:`n_{c_{quad}}` 
data points, however, it is a good rule of thumb to use at least
:math:`n_{c_{quad}}`  data points when possible.

*Known Issue: When using discrete variables, there have been
sometimes significant differences in surrogate behavior observed
across computing platforms in some cases.  The cause has not yet been
fully diagnosed and is currently under investigation.  In addition,
guidance on appropriate construction and use of surrogates with
discrete variables is under development.  In the meantime, users
should therefore be aware that there is a risk of inaccurate results
when using surrogates with discrete variables.*



**Theory**


The form of the neural network model is


.. math:: 
   \hat{f}(\mathbf{x}) \approx
   \tanh\left\{ \mathbf{A}_{1} \tanh\left( \mathbf{A}_{0}^{T} \mathbf{x} +\theta_{0}^T \right)+\theta_{1} \right\}

where :math:`\mathbf{x}`  is the evaluation point in :math:`n` -dimensional
parameter space; the terms :math:`\mathbf{A}_{0}, \theta_{0}`  are the
random input layer weight matrix and bias vector, respectively; and
:math:`\mathbf{A}_{1}, \theta_{1}`  are a weight vector and bias scalar,
respectively, estimated from training data.  These coefficients are
analogous to the polynomial coefficients obtained from regression to
training data.  The neural network uses a cross validation-based
orthogonal matching pursuit solver to determine the optimal number of
nodes and to solve for the weights and offsets.