.. _`theory:nls:ci`: Confidence Intervals for Nonlinear Least Squares ================================================ When a calibration is performed with one of Dakota's tailored nonlinear least squares solvers (see :ref:`nls`), Dakota reports a confidence interval for each calibrated parameter at the solution. This chapter describes the statistical model behind these intervals, how they are computed, and the assumptions that limit their validity. The approach is the classical linearization (asymptotic) theory of nonlinear regression; see :cite:p:`Seb03` for a thorough treatment and :cite:p:`Vug07` for a comparison with alternative confidence region techniques. .. _`theory:nls:ci:model`: Statistical Model ----------------- Consider the least squares problem .. math:: \hat{\mathbf{x}} = \arg\min_{\mathbf{x} \in \Re^p} f(\mathbf{x}) = \sum_{i=1}^{n} T_i(\mathbf{x})^2 = \mathbf{T}(\mathbf{x})^T \mathbf{T}(\mathbf{x}), where :math:`\mathbf{x}` is the vector of :math:`p` calibration parameters and :math:`\mathbf{T}(\mathbf{x}) = [T_1(\mathbf{x}), \ldots, T_n(\mathbf{x})]^T` is the vector of :math:`n` least squares terms. In the most common case each term is a residual between a model prediction and an observation, .. math:: T_i(\mathbf{x}) = R_i(\mathbf{x}) - \bar{R}_i . To make statistical statements about :math:`\hat{\mathbf{x}}`, the observations are assumed to be generated by the model at some unknown true parameter value :math:`\mathbf{x}^*`, corrupted by additive noise: .. math:: \bar{R}_i = R_i(\mathbf{x}^*) + \epsilon_i, \qquad \epsilon_i \overset{iid}{\sim} \mathcal{N}(0, \sigma^2), \qquad i = 1, \ldots, n, with common but unknown error variance :math:`\sigma^2`. Under this model the least squares estimate :math:`\hat{\mathbf{x}}` is also the maximum likelihood estimate of :math:`\mathbf{x}^*`. .. _`theory:nls:ci:linearization`: Linearization ------------- Let .. math:: \mathbf{J} = \left[ \frac{\partial T_i}{\partial x_j} \right]_{\mathbf{x} = \hat{\mathbf{x}}} \in \Re^{n \times p} denote the Jacobian matrix of the least squares terms with respect to the parameters, evaluated at the solution. Near :math:`\hat{\mathbf{x}}` the model is approximated by its first-order Taylor expansion, .. math:: \mathbf{T}(\mathbf{x}) \approx \mathbf{T}(\hat{\mathbf{x}}) + \mathbf{J} \, (\mathbf{x} - \hat{\mathbf{x}}), so locally the problem is a *linear* regression with design matrix :math:`\mathbf{J}`. Applying linear regression theory to this approximation gives the approximate sampling distribution of the estimator, .. math:: \hat{\mathbf{x}} \;\dot\sim\; \mathcal{N}\left( \mathbf{x}^*, \; \sigma^2 \left( \mathbf{J}^T \mathbf{J} \right)^{-1} \right). The matrix :math:`(\mathbf{J}^T \mathbf{J})^{-1}` is the inverse of the Gauss-Newton approximation to the Hessian of :math:`f/2`, the same matrix used by the Gauss-Newton solution methods (see :ref:`nls:formulations`). The approximation becomes exact as the noise level goes to zero or the number of observations grows, and it is exact for any :math:`n` if the model is linear in the parameters. .. _`theory:nls:ci:variance`: Estimating the Error Variance ----------------------------- The error variance :math:`\sigma^2` is estimated from the residual sum of squares at the solution: .. math:: \hat{\sigma}^2 = \frac{1}{N_{dof}} \sum_{i=1}^{n} T_i(\hat{\mathbf{x}})^2, \qquad N_{dof} = n - p . Dividing by the residual degrees of freedom :math:`N_{dof}` instead of :math:`n` accounts for the :math:`p` parameters fitted to the data, and makes :math:`\hat{\sigma}^2` unbiased in the linear case. (If :math:`n = p`, Dakota uses :math:`N_{dof} = 1` to avoid dividing by zero. Intervals computed this way have essentially no statistical meaning.) .. _`theory:nls:ci:intervals`: Standard Errors and Confidence Intervals ---------------------------------------- The estimated covariance of the parameter estimates is :math:`\hat{\sigma}^2 (\mathbf{J}^T\mathbf{J})^{-1}`, so the standard error of the :math:`j`-th parameter is .. math:: SE_j = \hat{\sigma} \sqrt{ \left( \mathbf{J}^T \mathbf{J} \right)^{-1}_{jj} } . Because :math:`\sigma^2` is estimated rather than known, the standardized quantity :math:`(\hat{x}_j - x_j^*)/SE_j` approximately follows a Student's t-distribution with :math:`N_{dof}` degrees of freedom. The two-sided :math:`100(1-\alpha)\%` confidence interval for parameter :math:`x_j` is therefore .. math:: \hat{x}_j \pm t_{1-\alpha/2, \, N_{dof}} \cdot SE_j , where :math:`t_{1-\alpha/2, \, N_{dof}}` is the :math:`1-\alpha/2` quantile of the t-distribution. Dakota reports 95% intervals (:math:`\alpha = 0.05`, so the quantile :math:`t_{0.975, \, N_{dof}}` is used). These are *individual* (marginal) intervals: each covers its own parameter with approximately 95% probability, but the probability that all :math:`p` intervals cover their parameters at once is generally lower. The intervals also ignore correlation between parameter estimates, which is described by the off-diagonal entries of :math:`(\mathbf{J}^T\mathbf{J})^{-1}`. Joint regions, such as the linearized ellipsoid .. math:: \left\{ \mathbf{x} : (\mathbf{x} - \hat{\mathbf{x}})^T \mathbf{J}^T \mathbf{J} (\mathbf{x} - \hat{\mathbf{x}}) \le p \, \hat{\sigma}^2 F_{1-\alpha;\, p, \, N_{dof}} \right\}, or simultaneous Bonferroni intervals (which replace :math:`\alpha` by :math:`\alpha/p`), are alternatives that Dakota does not currently report. .. _`theory:nls:ci:qr`: Numerical Computation --------------------- Forming :math:`\mathbf{J}^T\mathbf{J}` explicitly squares the condition number of :math:`\mathbf{J}`, which can cause significant loss of precision when parameters are poorly identified. Instead, Dakota takes a thin QR factorization .. math:: \mathbf{J} = \mathbf{Q} \mathbf{R}, \qquad \mathbf{Q} \in \Re^{n \times p}, \; \mathbf{Q}^T\mathbf{Q} = \mathbf{I}, \; \mathbf{R} \in \Re^{p \times p} \text{ upper triangular}. Then .. math:: \left( \mathbf{J}^T \mathbf{J} \right)^{-1} = \left( \mathbf{R}^T \mathbf{Q}^T \mathbf{Q} \mathbf{R} \right)^{-1} = \mathbf{R}^{-1} \mathbf{R}^{-T}, so the needed diagonal entries are the squared Euclidean norms of the rows of the triangular inverse :math:`\mathbf{R}^{-1}`: .. math:: \left( \mathbf{J}^T \mathbf{J} \right)^{-1}_{jj} = \sum_{k=j}^{p} \left( \mathbf{R}^{-1} \right)_{jk}^2 . Only :math:`\mathbf{R}` must be computed and inverted. If :math:`\mathbf{J}` is rank deficient (for example, a parameter has no effect on any residual, or two parameters have the same effect), :math:`\mathbf{R}` is singular and the parameters cannot be identified from the data. Dakota detects this only when a diagonal entry of :math:`\mathbf{R}` is exactly zero, in which case it issues a warning and reports no intervals. The QR factorization does not use column pivoting or a rank tolerance, so in floating point arithmetic a rank deficient or nearly rank deficient :math:`\mathbf{J}` usually yields small but nonzero diagonal entries instead. Dakota then reports intervals, but they are extremely wide or numerically meaningless. Very large intervals should be taken as a sign that the corresponding parameters are poorly identified, not as a reliable measure of uncertainty. .. _`theory:nls:ci:weighting`: Weighted, Error-Scaled, and Scaled Residuals -------------------------------------------- The intervals are computed from the least squares terms that the solver actually minimized. When Dakota transforms the residuals before handing them to the solver, :math:`\mathbf{T}` and :math:`\mathbf{J}` above refer to the transformed terms. The parameters, however, are always in their original (native, unscaled) units, so the intervals are reported in the user's parameter space. **Observation error covariance.** When observation errors with covariance :math:`\boldsymbol{\Sigma}` are given with calibration data, Dakota computes the Cholesky factorization :math:`\boldsymbol{\Sigma} = \mathbf{L}\mathbf{L}^T` and whitens the raw residuals :math:`\mathbf{r} = \mathbf{R}(\mathbf{x}) - \bar{\mathbf{R}}`: .. math:: \mathbf{T}(\mathbf{x}) = \mathbf{L}^{-1} \mathbf{r}(\mathbf{x}), \qquad \mathbf{J} = \mathbf{L}^{-1} \frac{\partial \mathbf{r}}{\partial \mathbf{x}} . For a diagonal covariance this is the familiar variance-weighted form :math:`T_i = r_i / \sigma_i`. The minimized objective is the generalized least squares misfit :math:`\mathbf{r}^T\boldsymbol{\Sigma}^{-1}\mathbf{r}`, and :math:`(\mathbf{J}^T\mathbf{J})^{-1}` becomes :math:`(\mathbf{G}^T\boldsymbol{\Sigma}^{-1}\mathbf{G})^{-1}`, where :math:`\mathbf{G} = \partial\mathbf{r}/\partial\mathbf{x}`. Dakota still estimates :math:`\hat{\sigma}^2` from the data rather than fixing it to one. The specified covariance therefore sets the *relative* weighting of the observations, and :math:`\hat{\sigma}^2` acts as an overall scale factor fitted to the data. A value of :math:`\hat{\sigma}^2` far from one suggests that the specified error magnitudes are inconsistent with the observed misfit. **Residual weights.** User-specified nonnegative weights :math:`w_i` define the objective :math:`\sum_i w_i r_i^2`, which is equivalent to least squares terms :math:`T_i = \sqrt{w_i} \, r_i` with Jacobian rows scaled by :math:`\sqrt{w_i}`. Under the statistical model above, this corresponds to assuming :math:`\mathrm{Var}(\epsilon_i) \propto 1/w_i`. A term with zero weight contributes nothing to the residual sum of squares or to :math:`\mathbf{J}`, but it is still counted in :math:`n` when computing :math:`N_{dof} = n - p`. Zero weights therefore inflate :math:`N_{dof}`, which makes :math:`\hat{\sigma}^2` and the t quantile too small and the intervals too narrow. **Response scaling.** Scaling each residual by a constant :math:`c_i` has the same effect on the intervals as a weight :math:`w_i = c_i^2`. If every residual is scaled by the same constant :math:`c`, :math:`\hat{\sigma}^2` is multiplied by :math:`c^2` and :math:`(\mathbf{J}^T\mathbf{J})^{-1}` by :math:`c^{-2}`, so the intervals do not change. This equivalence applies only to value (multiplicative) scaling. With logarithmic response scaling the solver minimizes the sum of squares of the log-transformed terms, and the intervals are those of that transformed problem: :math:`\mathbf{T}` and :math:`\mathbf{J}` are the log-scaled terms and their derivatives, and the error model of :ref:`theory:nls:ci:model` is assumed to hold for the log-scaled terms rather than for the original residuals. **Parameter scaling.** If the solver works with scaled parameters :math:`\mathbf{u} = \mathbf{s}(\mathbf{x})`, the chain rule converts the Jacobian back to native parameters before the intervals are formed: .. math:: \frac{\partial \mathbf{T}}{\partial \mathbf{x}} = \frac{\partial \mathbf{T}}{\partial \mathbf{u}} \frac{\partial \mathbf{u}}{\partial \mathbf{x}} . .. _`theory:nls:ci:limitations`: Assumptions and Limitations --------------------------- The intervals are only as reliable as the assumptions behind them: - **Linearity.** The intervals come from a local linear approximation at :math:`\hat{\mathbf{x}}`. For strongly nonlinear models, or when the noise is large compared with the curvature of the model, the true confidence region can be far from the symmetric interval, and its actual coverage can differ greatly from 95%. Profile likelihood or Bayesian calibration (see :ref:`uq:bayesian`) gives a more faithful picture in such cases. - **Error model.** Independent, identically distributed Gaussian errors (after any weighting or whitening) are assumed. Model form error, correlated errors that are not described by the covariance, or heavy-tailed noise all invalidate the nominal coverage. - **Interior optimum.** The derivation assumes :math:`\hat{\mathbf{x}}` is an unconstrained local minimum. The intervals ignore parameter bounds and other constraints. They are not truncated to the bounds, and they may be meaningless when a constraint is active at the solution. - **Derivative accuracy.** :math:`\mathbf{J}` must be accurate. Intervals are not computed when the solver's internal (vendor) finite differencing is used. Inaccurate gradients give inaccurate intervals. - **Problem size.** At least as many least squares terms as parameters (:math:`n \ge p`) are required, and :math:`\mathbf{J}` must have full column rank. - **Multiple experiments.** With replicate experiments, :math:`n` counts the residuals from all experiments together, and the formulas above are applied to the stacked system. Dakota warns that intervals may be inaccurate in this case.