Nonlinear Regression Package

The built-in function FindFit allows you to perform nonlinear least squares fitting. The function NonlinearRegress gives a number of regression diagnostics and allows you to specify exactly what will be included in the output. NonlinearRegress is similar to the Linear Regression Package function Regress, which gives diagnostics for linear least squares fitting.

The NonlinearRegress function.

The expr argument to NonlinearRegress must be completely specified by the symbols in the vars argument and the symbols in the pars argument. The vars argument specifies the independent variables represented in data. The pars argument specifies the parameters to be fitted. For constrained models, the cons argument can contain equations, inequalities, or logical combinations of equations and inequalities.

The data argument can be a list of vectors, each vector consisting of the values of the independent variables x₁, x₂, …, followed by the observed value of the associated response y. The data argument can also be a vector, in which case it is assumed that the vector represents the observed values of the response variable, with the independent variable(s) for the i ^th response equal to i.

Data specifications for NonlinearRegress.

The estimates of the model parameters are chosen to minimize the merit function given by the sum of squared residuals . The optimization methods used by NonlinearRegress are iterative so starting values are required for the parameter estimate search. Careful choice of starting values may be necessary, as the parameter estimates found by NonlinearRegress may represent a local minimum in the merit function.

Parameters are specified using the same syntax as in FindFit. Parameters can be specified as symbol, {symbol,start}, {symbol,start₀,start₁}, {symbol,start,min,max}, {symbol,start₀,start₁,min,max}, where start, start₀, start₁, min, and max are real numbers specifying starting values and minimum and maximum values for symbol. If the method is NMinimize, NMinimize automatically finds a feasible starting point. If parameters are specified as symbol and the method is not NMinimize, the starting values for all parameters are taken to be 1.

If a parameter is specified as {symbol,start₀,start₁}, then the search for parameter estimates uses start₀ and start₁ as the first two values of symbol. This form must be used if symbolic derivatives of with respect to the parameters cannot be found.

This fits the model and gives diagnostics for the fit.

Options for NonlinearRegress.

NonlinearRegress uses FindFit to find parameter estimates, thus it accepts the FindFit options AccuracyGoal, Compiled, Gradient, MaxIterations, Method, PrecisionGoal, and WorkingPrecision. The Method option allows you to choose between algorithms for performing the minimization of the merit function. Possible settings for Method are "LevenbergMarquardt", "Gradient" (steepest descent), "Newton", "QuasiNewton", NMinimize, and Automatic. Method options can be passed to methods by specifying Method->{method,options}.

The default RegressionReport setting SummaryReport gives a list including values for BestFitParameters, ParameterCITable, EstimatedVariance, ANOVATable, AsymptoticCorrelationMatrix, and FitCurvatureTable. The option RegressionReport can be used to specify a single object or a list of objects so that more (or less) than the default set of results is included in the output. RegressionReportValues[NonlinearRegress] gives the objects that may be included in the RegressionReport list for the NonlinearRegress function.

The Weights option allows you to implement weighted least squares by specifying a list of weights, one for each data point. The default Weights->Automatic implies a weight of unity for each data point. When Weights->{w₁,…,w_n}, the parameter estimates are chosen to minimize the weighted sum of squared residuals .

Weights can also specify a pure function of the response. For example, to choose parameter estimates to minimize , set Weights->(Sqrt[#]&).

NonlinearRegress will also accept any option that can be specified for StudentTCI. In particular, ConfidenceLevel specifies the confidence level for the reference parameter confidence region (the relative curvature of which is given in FitCurvatureTable) and the individual parameter confidence intervals given in ParameterCITable.

Results can be extracted from the report and used in further computations.

Basic RegressionReport values for NonlinearRegress.

ANOVATable provides a partitioning of the sum of squares. Both the uncorrected total sum of squares (without intercept term) and the corrected total sum of squares (with intercept term) are included. The corrected total sum of squares is useful for calculating a value for R² comparable to that produced by Regress, so that the fit of a nonlinear model can be compared to the fit of a linear model. For example, one possible definition of R² for a nonlinear regression is the ratio of the difference between the corrected total sum of squares and the residual sum of squares to the corrected total sum of squares. The other objects in this table have meanings similar to those of analogous RegressionReport values for the linear regression function Regress.

Value describing the search for the least-squares fit.

Including StartingParameters in the list of objects requested by RegressionReport is helpful if the search for a least-squares fit is unsuccessful and it is necessary to restart the search at a new point.

Diagnostics for detecting outliers.

If the nonlinear model is approximately linear at the least-squares fit, HatDiagonal and StandardizedResiduals are useful for detecting outliers in the data. HatDiagonal gives the leverage of each point, measuring whether each observation of the predictor variables is unusual. A leverage of zero indicates no influence, while a leverage of one indicates that a degree of freedom has been lost to fitting that point.

Diagnostics for evaluating the validity of a linear model approximation to the nonlinear model.

FitCurvatureTable expresses the relative curvature of the solution locus at the least-squares estimate in terms of two components, intrinsic and parameter-effects. For the combination of data and model given above, p=3 and n=5. The solution locus is a three-dimensional subspace of R⁵ given by {, , , , }, .

Intrinsic curvature describes the normal component of the solution locus curvature at the least-squares estimate. Parameter-effects curvature describes the tangential component of the solution locus curvature at the least-squares estimate. Standardizing curvature to be response-invariant gives relative curvature. Maximizing the relative intrinsic curvature over the (n-p)-dimensional subspace normal to the locus gives the maximum relative intrinsic curvature. Similarly, maximizing the relative parameter-effects curvature over the p-dimensional subspace tangential to the locus gives the maximum relative parameter-effects curvature.

Both of these quantities can be compared to the relative curvature of the confidence region centered on the least-squares parameter estimates. If the maximum relative intrinsic curvature is small compared to the confidence region relative curvature, the solution locus is approximately planar over the confidence region. If the maximum relative parameter-effects curvature is small compared to the confidence region relative curvature, the parameter coordinates projected onto the tangential plane are approximately parallel and uniformly spaced over the confidence region.

ParameterBias is based on the average curvature of the solution locus tangential to the least-squares estimate.

Here the same data is fit to a reparametrized model, .

Note that reparametrizing the model does not affect the intrinsic curvature. However, here reparametrization has reduced the parameter-effects curvature below the critical value of 0.22843. This indicates that the least-squares estimates of the parameters , , and have nearly linear behavior. However, now the asymptotic correlation between parameters is high for all pairs, indicating that a model with fewer parameters should be considered.

The maximum relative curvature diagnostics are useful if you wish to make inferences based on the linear approximation to the nonlinear model. For example, diagnostics based on the asymptotic standard errors of the parameters, such as ParameterCITable and ParameterConfidenceRegion, are of questionable validity when the linear approximation to the model is poor at the least-squares parameter estimates. The linear approximation is based on the planar assumption, tested by maximum relative intrinsic curvature, and the uniform coordinate assumption, tested by maximum relative parameter-effects curvature.

For constrained models, RegressionReport values based on approximate normality assumptions may not be valid. If such values are listed in the RegressionReport option, the values will be generated along with a warning message.