As of Version 7.0, nonlinear regression functionality is built into the Wolfram Language. »

Nonlinear Regression Package

The built-in function FindFit allows you to perform nonlinear least squares fitting. The function NonlinearRegress gives a number of regression diagnostics and allows you to specify exactly what will be included in the output. NonlinearRegress is similar to the Linear Regression Package function Regress, which gives diagnostics for linear least squares fitting.

NonlinearRegress[data,expr,pars,vars]find numerical values of the parameters pars that make the model expr give a best fit to data as a function of vars and provide diagnostics for the fitting
NonlinearRegress[data,{expr,cons},pars,vars] fit the data to the model subject to the given constraints, find a best fit and provide diagnostics subject to the constraints cons

The NonlinearRegress function.

The expr argument to NonlinearRegress must be completely specified by the symbols in the vars argument and the symbols in the pars argument. The vars argument specifies the independent variables represented in data. The pars argument specifies the parameters to be fitted. For constrained models, the cons argument can contain equations, inequalities, or logical combinations of equations and inequalities.

The data argument can be a list of vectors, each vector consisting of the values of the independent variables , , , followed by the observed value of the associated response y. The data argument can also be a vector, in which case it is assumed that the vector represents the observed values of the response variable, with the independent variable(s) for the i^(th) response equal to i.

{y1,y2,}data points specified by a list of response values, where i is the value of the independent variable associated with the i^(th) response,
{{x11,x12,,y1},{x21,x22,,y2},}
data points specified by a matrix, where is the value of the i^(th) case of the k^(th) independent variable, and is the i^(th) response

Data specifications for NonlinearRegress.

The estimates of the model parameters are chosen to minimize the merit function given by the sum of squared residuals . The optimization methods used by NonlinearRegress are iterative so starting values are required for the parameter estimate search. Careful choice of starting values may be necessary, as the parameter estimates found by NonlinearRegress may represent a local minimum in the merit function.

Parameters are specified using the same syntax as in FindFit. Parameters can be specified as symbol, , , , , where start, , , min, and max are real numbers specifying starting values and minimum and maximum values for symbol. If the method is NMinimize, NMinimize automatically finds a feasible starting point. If parameters are specified as symbol and the method is not NMinimize, the starting values for all parameters are taken to be 1.

If a parameter is specified as , then the search for parameter estimates uses and as the first two values of symbol. This form must be used if symbolic derivatives of with respect to the parameters cannot be found.

This loads the package.
In[1]:=
Click for copyable input
This data set due to Roth and Meyer describes a chemical reaction.
In[2]:=
Click for copyable input
This fits the model and gives diagnostics for the fit.
In[3]:=
Click for copyable input
Out[3]=
option name
default value
AccuracyGoalAutomaticthe accuracy sought
CompiledAutomaticwhether to compile model
GradientAutomaticlist of gradient functions
MaxIterations100maximum number of iterations to use
MethodAutomaticmethod to use
PrecisionGoalAutomaticthe precision sought
RegressionReportSummaryReportfit diagnostics to include
ToleranceAutomaticnumerical tolerance for matrix operations
WeightsAutomaticlist of weights for each point or pure function
WorkingPrecisionMachinePrecisionthe precision used in internal computations

Options for NonlinearRegress.

NonlinearRegress uses FindFit to find parameter estimates, thus it accepts the FindFit options AccuracyGoal, Compiled, , MaxIterations, Method, PrecisionGoal, and WorkingPrecision. The Method option allows you to choose between algorithms for performing the minimization of the merit function. Possible settings for Method are , (steepest descent), , , NMinimize, and Automatic. Method options can be passed to methods by specifying Method->{method,options}.

The default RegressionReport setting SummaryReport gives a list including values for BestFitParameters, ParameterCITable, EstimatedVariance, ANOVATable, AsymptoticCorrelationMatrix, and FitCurvatureTable. The option RegressionReport can be used to specify a single object or a list of objects so that more (or less) than the default set of results is included in the output. RegressionReportValues[NonlinearRegress] gives the objects that may be included in the RegressionReport list for the NonlinearRegress function.

The Weights option allows you to implement weighted least squares by specifying a list of weights, one for each data point. The default Weights->Automatic implies a weight of unity for each data point. When Weights->{w1,,wn}, the parameter estimates are chosen to minimize the weighted sum of squared residuals .

Weights can also specify a pure function of the response. For example, to choose parameter estimates to minimize , set Weights->(Sqrt[#]&).

NonlinearRegress will also accept any option that can be specified for StudentTCI. In particular, ConfidenceLevel specifies the confidence level for the reference parameter confidence region (the relative curvature of which is given in FitCurvatureTable) and the individual parameter confidence intervals given in ParameterCITable.

Here just the parameter estimates are returned.
In[4]:=
Click for copyable input
Out[4]=

Results can be extracted from the report and used in further computations.

The residuals are extracted and plotted here.
In[5]:=
Click for copyable input
Out[5]=
In[6]:=
Click for copyable input
Out[6]=
BestFitbest fit function
BestFitParametersbest fit parameter estimates
ANOVATableanalysis of variance table
EstimatedVarianceestimated error variance
ParameterTabletable of parameter information including asymptotic standard errors and test statistics
ParameterCITabletable of asymptotic confidence intervals for the parameters
ParameterConfidenceRegionasymptotic ellipsoidal joint confidence region for the parameters
ParameterConfidenceRegion[{pari 1,pari 2,}]
asymptotic ellipsoidal conditional joint confidence region for the parameters
FitResidualsdifferences between the observed responses and the predicted responses
PredictedResponsefitted values obtained by evaluating the best fit function at the observed values of the independent variables
SinglePredictionCITabletable of asymptotic confidence intervals for predicting a single observation of the response variable
MeanPredictionCITabletable of asymptotic confidence intervals for predicting the expected value of the response variable
AsymptoticCovarianceMatrixasymptotic covariance matrix of the parameters
AsymptoticCorrelationMatrixasymptotic correlation matrix of the parameters

Basic RegressionReport values for NonlinearRegress.

ANOVATable provides a partitioning of the sum of squares. Both the uncorrected total sum of squares (without intercept term) and the corrected total sum of squares (with intercept term) are included. The corrected total sum of squares is useful for calculating a value for comparable to that produced by Regress, so that the fit of a nonlinear model can be compared to the fit of a linear model. For example, one possible definition of for a nonlinear regression is the ratio of the difference between the corrected total sum of squares and the residual sum of squares to the corrected total sum of squares. The other objects in this table have meanings similar to those of analogous RegressionReport values for the linear regression function Regress.

StartingParametersstarting values for the parameters

Value describing the search for the least-squares fit.

Including StartingParameters in the list of objects requested by RegressionReport is helpful if the search for a least-squares fit is unsuccessful and it is necessary to restart the search at a new point.

HatDiagonaldiagonal of the hat matrix , where is the design matrix for the approximate linear model at the least-squares parameter estimates
StandardizedResidualsfit residuals scaled by their asymptotic standard errors, computed using the estimated error variance

Diagnostics for detecting outliers.

If the nonlinear model is approximately linear at the least-squares fit, HatDiagonal and StandardizedResiduals are useful for detecting outliers in the data. HatDiagonal gives the leverage of each point, measuring whether each observation of the predictor variables is unusual. A leverage of zero indicates no influence, while a leverage of one indicates that a degree of freedom has been lost to fitting that point.

FitCurvatureTablemaximum relative intrinsic curvature, maximum relative parameter-effects curvature, and relative curvature of the least-squares solution confidence region (where confidence level is given by ConfidenceLevel)
ParameterBiasbias in the least-squares estimates of the parameters in a nonlinear model

Diagnostics for evaluating the validity of a linear model approximation to the nonlinear model.

FitCurvatureTable expresses the relative curvature of the solution locus at the least-squares estimate in terms of two components, intrinsic and parameter-effects. For the combination of data and model given above, p=3 and n=5. The solution locus is a three-dimensional subspace of given by {, , , , }, .

Intrinsic curvature describes the normal component of the solution locus curvature at the least-squares estimate. Parameter-effects curvature describes the tangential component of the solution locus curvature at the least-squares estimate. Standardizing curvature to be response-invariant gives relative curvature. Maximizing the relative intrinsic curvature over the -dimensional subspace normal to the locus gives the maximum relative intrinsic curvature. Similarly, maximizing the relative parameter-effects curvature over the p-dimensional subspace tangential to the locus gives the maximum relative parameter-effects curvature.

Both of these quantities can be compared to the relative curvature of the confidence region centered on the least-squares parameter estimates. If the maximum relative intrinsic curvature is small compared to the confidence region relative curvature, the solution locus is approximately planar over the confidence region. If the maximum relative parameter-effects curvature is small compared to the confidence region relative curvature, the parameter coordinates projected onto the tangential plane are approximately parallel and uniformly spaced over the confidence region.

ParameterBias is based on the average curvature of the solution locus tangential to the least-squares estimate.

Here the same data is fit to a reparametrized model, .
In[7]:=
Click for copyable input
Out[7]=

Note that reparametrizing the model does not affect the intrinsic curvature. However, here reparametrization has reduced the parameter-effects curvature below the critical value of . This indicates that the least-squares estimates of the parameters , , and have nearly linear behavior. However, now the asymptotic correlation between parameters is high for all pairs, indicating that a model with fewer parameters should be considered.

The maximum relative curvature diagnostics are useful if you wish to make inferences based on the linear approximation to the nonlinear model. For example, diagnostics based on the asymptotic standard errors of the parameters, such as ParameterCITable and ParameterConfidenceRegion, are of questionable validity when the linear approximation to the model is poor at the least-squares parameter estimates. The linear approximation is based on the planar assumption, tested by maximum relative intrinsic curvature, and the uniform coordinate assumption, tested by maximum relative parameter-effects curvature.

For constrained models, RegressionReport values based on approximate normality assumptions may not be valid. If such values are listed in the RegressionReport option, the values will be generated along with a warning message.