As of Version 7.0, nonlinear regression functionality is built into

*Mathematica*.

»
# Nonlinear Regression Package

The built-in function

FindFit allows you to perform nonlinear least squares fitting. The function

NonlinearRegress gives a number of regression diagnostics and allows you to specify exactly what will be included in the output.

NonlinearRegress is similar to the

Linear Regression Package function

Regress, which gives diagnostics for linear least squares fitting.

NonlinearRegress[data,expr,pars,vars] | find numerical values of the parameters pars that make the model expr give a best fit to data as a function of vars and provide diagnostics for the fitting |

NonlinearRegress[data,{expr,cons},pars,vars] | fit the data to the model subject to the given constraints, find a best fit and provide diagnostics subject to the constraints cons |

The NonlinearRegress function.

The

expr argument to

NonlinearRegress must be completely specified by the symbols in the

vars argument and the symbols in the

pars argument. The

vars argument specifies the independent variables represented in

data. The

pars argument specifies the parameters to be fitted. For constrained models, the

cons argument can contain equations, inequalities, or logical combinations of equations and inequalities.

The

data argument can be a list of vectors, each vector consisting of the values of the independent variables

,

,

..., followed by the observed value of the associated response

y. The data argument can also be a vector, in which case it is assumed that the vector represents the observed values of the response variable, with the independent variable(s) for the

i response equal to

i.

{y_{1},y_{2},...} | data points specified by a list of response values, where i is the value of the independent variable associated with the i response, |

{{x_{11},x_{12},...,y_{1}},{x_{21},x_{22},...,y_{2}},...} |

| data points specified by a matrix, where is the value of the i case of the k independent variable, and is the i response |

Data specifications for NonlinearRegress.

The estimates of the model parameters are chosen to minimize the

merit function given by the sum of squared residuals

. The optimization methods used by

NonlinearRegress are iterative so starting values are required for the parameter estimate search. Careful choice of starting values may be necessary, as the parameter estimates found by

NonlinearRegress may represent a local minimum in the

merit function.

Parameters are specified using the same syntax as in

FindFit. Parameters can be specified as

symbol,

,

,

,

, where

start,

,

,

min, and

max are real numbers specifying starting values and minimum and maximum values for symbol. If the method is

NMinimize,

NMinimize automatically finds a feasible starting point. If parameters are specified as symbol and the method is not

NMinimize, the starting values for all parameters are taken to be 1.

If a parameter is specified as

, then the search for parameter estimates uses

and

as the first two values of

symbol. This form must be used if symbolic derivatives of

with respect to the parameters cannot be found.

This data set due to Roth and Meyer describes a chemical reaction.

This fits the model

and gives diagnostics for the fit.

Out[3]= | |

Options for NonlinearRegress.

NonlinearRegress uses

FindFit to find parameter estimates, thus it accepts the

FindFit options

AccuracyGoal,

Compiled,

,

MaxIterations,

Method,

PrecisionGoal, and

WorkingPrecision. The

Method option allows you to choose between algorithms for performing the minimization of the

merit function. Possible settings for

Method are

,

(steepest descent),

,

,

NMinimize, and

Automatic. Method options can be passed to methods by specifying

Method.

The default

RegressionReport setting

SummaryReport gives a list including values for

BestFitParameters,

ParameterCITable,

EstimatedVariance,

ANOVATable,

AsymptoticCorrelationMatrix, and

FitCurvatureTable. The option

RegressionReport can be used to specify a single object or a list of objects so that more (or less) than the default set of results is included in the output.

RegressionReportValues[NonlinearRegress] gives the objects that may be included in the

RegressionReport list for the

NonlinearRegress function.

The

Weights option allows you to implement weighted least squares by specifying a list of weights, one for each data point. The default

Weights->Automatic implies a weight of unity for each data point. When

Weights, the parameter estimates are chosen to minimize the weighted sum of squared residuals

.

Weights can also specify a pure function of the response. For example, to choose parameter estimates to minimize

, set

Weights->(Sqrt[#]&).

NonlinearRegress will also accept any option that can be specified for

StudentTCI. In particular,

ConfidenceLevel specifies the confidence level for the reference parameter confidence region (the relative curvature of which is given in

FitCurvatureTable) and the individual parameter confidence intervals given in

ParameterCITable.

Here just the parameter estimates are returned.

Out[4]= | |

Results can be extracted from the report and used in further computations.

The residuals are extracted and plotted here.

Out[5]= | |

Out[6]= | |

Basic RegressionReport values for NonlinearRegress.

ANOVATable provides a partitioning of the sum of squares. Both the uncorrected total sum of squares (without intercept term) and the corrected total sum of squares (with intercept term) are included. The corrected total sum of squares is useful for calculating a value for

comparable to that produced by

Regress, so that the fit of a nonlinear model can be compared to the fit of a linear model. For example, one possible definition of

for a nonlinear regression is the ratio of the difference between the corrected total sum of squares and the residual sum of squares to the corrected total sum of squares. The other objects in this table have meanings similar to those of analogous

RegressionReport values for the linear regression function

Regress.

Value describing the search for the least-squares fit.

Including

StartingParameters in the list of objects requested by

RegressionReport is helpful if the search for a least-squares fit is unsuccessful and it is necessary to restart the search at a new point.

HatDiagonal | diagonal of the hat matrix , where is the design matrix for the approximate linear model at the least-squares parameter estimates |

StandardizedResiduals | fit residuals scaled by their asymptotic standard errors, computed using the estimated error variance |

Diagnostics for detecting outliers.

If the nonlinear model is approximately linear at the least-squares fit,

HatDiagonal and

StandardizedResiduals are useful for detecting outliers in the data.

HatDiagonal gives the leverage of each point, measuring whether each observation of the predictor variables is unusual. A leverage of zero indicates no influence, while a leverage of one indicates that a degree of freedom has been lost to fitting that point.

FitCurvatureTable | maximum relative intrinsic curvature, maximum relative parameter-effects curvature, and relative curvature of the least-squares solution confidence region (where confidence level is given by ConfidenceLevel) |

ParameterBias | bias in the least-squares estimates of the parameters in a nonlinear model |

Diagnostics for evaluating the validity of a linear model approximation to the nonlinear model.

FitCurvatureTable expresses the relative curvature of the solution locus at the least-squares estimate in terms of two components, intrinsic and parameter-effects. For the combination of data and model given above,

p=3 and

n=5. The solution locus is a three-dimensional subspace of

given by {

,

,

,

,

},

.

Intrinsic curvature describes the normal component of the solution locus curvature at the least-squares estimate. Parameter-effects curvature describes the tangential component of the solution locus curvature at the least-squares estimate. Standardizing curvature to be response-invariant gives relative curvature. Maximizing the relative intrinsic curvature over the

-dimensional subspace normal to the locus gives the maximum relative intrinsic curvature. Similarly, maximizing the relative parameter-effects curvature over the

p-dimensional subspace tangential to the locus gives the maximum relative parameter-effects curvature.

Both of these quantities can be compared to the relative curvature of the confidence region centered on the least-squares parameter estimates. If the maximum relative intrinsic curvature is small compared to the confidence region relative curvature, the solution locus is approximately planar over the confidence region. If the maximum relative parameter-effects curvature is small compared to the confidence region relative curvature, the parameter coordinates projected onto the tangential plane are approximately parallel and uniformly spaced over the confidence region.

ParameterBias is based on the average curvature of the solution locus tangential to the least-squares estimate.

Here the same data is fit to a reparametrized model,

.

Out[7]= | |

Note that reparametrizing the model does not affect the intrinsic curvature. However, here reparametrization has reduced the parameter-effects curvature below the critical value of

. This indicates that the least-squares estimates of the parameters

,

, and

have nearly linear behavior. However, now the asymptotic correlation between parameters is high for all pairs, indicating that a model with fewer parameters should be considered.

The maximum relative curvature diagnostics are useful if you wish to make inferences based on the linear approximation to the nonlinear model. For example, diagnostics based on the asymptotic standard errors of the parameters, such as

ParameterCITable and

ParameterConfidenceRegion, are of questionable validity when the linear approximation to the model is poor at the least-squares parameter estimates. The linear approximation is based on the planar assumption, tested by maximum relative intrinsic curvature, and the uniform coordinate assumption, tested by maximum relative parameter-effects curvature.

For constrained models,

RegressionReport values based on approximate normality assumptions may not be valid. If such values are listed in the

RegressionReport option, the values will be generated along with a warning message.