# NAG Library Routine Document

## e04fff (handle_solve_dfls)

Note: this routine uses optional parameters to define choices in the problem specification and in the details of the algorithm. If you wish to use default settings for all of the optional parameters, you need only read Sections 1 to 10 of this document. If, however, you wish to reset some or all of the settings please refer to Section 11 for a detailed description of the algorithm and to Section 12 for a detailed description of the specification of the optional parameters.

## 1Purpose

e04fff is a derivative-free solver from the NAG optimization modelling suite for small to medium-scale nonlinear least squares problems with bound constraints.

## 2Specification

Fortran Interface
 Subroutine e04fff ( mon, nvar, x, nres, rx,
 Integer, Intent (In) :: nvar, nres Integer, Intent (Inout) :: iuser(*), ifail Real (Kind=nag_wp), Intent (Inout) :: x(nvar), ruser(*) Real (Kind=nag_wp), Intent (Out) :: rx(nres), rinfo(100), stats(100) Type (c_ptr), Intent (In) :: handle, cpuser External :: objfun, mon
#include nagmk26.h
 void e04fff_ (void **handle, void (NAG_CALL *objfun)(const Integer *nvar, const double x[], const Integer *nres, double rx[], Integer *inform, Integer iuser[], double ruser[], void **cpuser),void (NAG_CALL *mon)(const Integer *nvar, const double x[], Integer *inform, const double rinfo[], const double stats[], Integer iuser[], double ruser[], void **cpuser),const Integer *nvar, double x[], const Integer *nres, double rx[], double rinfo[], double stats[], Integer iuser[], double ruser[], void **cpuser, Integer *ifail)

## 3Description

e04fff serves as a solver for compatible problems stored as a handle. The handle points to an internal data structure which defines the problem and serves as a means of communication for routines in the suite.
e04fff is aimed at minimizing a sum of a squares objective function subject to bound constraints:
 $minimize x∈ℝn ∑ i=1 mr ri x 2 (a) subject to lx≤x≤ux , (b)$ (1)
Here the ${r}_{i}\left(x\right)$ are smooth nonlinear functions called residuals and ${l}_{x}$ and ${u}_{x}$ are $n$-dimensional vectors defining bounds on the variables. Typically, in a calibration or data fitting context, the residuals will be defined as the difference between a data point and a nonlinear model (see Section 2.2.3 in the E04 Chapter Introduction)
To define a compatible problem handle, you must call e04raf followed by e04rmf to initialize it and optionally call e04rhf to define bounds on the variables. If e04rhf is not called, all the variables will be considered free by the solver. It should be noted that e04fff always assumes that the Jacobian of the residuals is dense, therefore defining a sparse structure for the residuals in the call to e04rmf will have no effect.
It is possible to fix some variables with the definition of the bounds. However, some constraints must be met in order to be able to call the solver:
• the number of non-fixed variables ${n}_{r}$ has to be at least $2$
• for all non-fixed variable ${x}_{i}$, the value of ${u}_{x}\left(i\right)-{l}_{x}\left(i\right)$ has to be at least twice the starting trust region radius (see the consistency constraint of the optional parameter DFLS Starting Trust Region).
The solver is based on a derivative-free trust region framework. This type of method is well suited for small to medium-scale problems (around 100 variables) for which the derivatives are unavailable or not easy to compute and/or for which the function evaluations are expensive or noisy. For a detailed description of the algorithm see Section 11. The algorithm behaviour and solver strategy can be modified by various optional parameters (see Section 12) which can be set by e04zmf and e04zpf anytime between the initialization of the handle by e04raf and a call to the solver. The default values for these optional parameters are chosen to work well in the general case but it is recommended to tune them to your particular problem. In particular, if the objective function is noisy, it is highly recommended to set the optional parameter DFLS Trust Region Update to SLOW to improve convergence. Once the solver has finished, options may be modified for the next solve. The solver may be called repeatedly with various starting points and/or optional parameters.

## 4References

Powell M J D (2009) The BOBYQA algorithm for bound constrained optimization without derivatives Report DAMTP 2009/NA06 University of Cambridge http://www.damtp.cam.ac.uk/user/na/NA_papers/NA2009_06.pdf
Zhang H, CONN A R and Scheinberg k (2010) A Derivative-Free Algorithm for Least-Squares Minimization SIAM J. Optim. 20(6) 3555–3576

## 5Arguments

1:     $\mathbf{handle}$ – Type (c_ptr)Input
On entry: the handle to the problem. It needs to be initialized by e04raf and the objective must be declared as nonlinear least squares by a call to the routine e04rmf. The routine e04rhf can optionally be called to define box bounds. It must not be changed between calls to the NAG optimization modelling suite.
2:     $\mathbf{objfun}$ – Subroutine, supplied by the user.External Procedure
objfun must evaluate the value of the nonlinear residuals ${r}_{i}\left(x\right)$ at a specified point $x$.
The specification of objfun is:
Fortran Interface
 Subroutine objfun ( nvar, x, nres, rx,
 Integer, Intent (In) :: nvar, nres Integer, Intent (Inout) :: inform, iuser(*) Real (Kind=nag_wp), Intent (In) :: x(nvar) Real (Kind=nag_wp), Intent (Inout) :: ruser(*) Real (Kind=nag_wp), Intent (Out) :: rx(nres) Type (c_ptr), Intent (In) :: cpuser
#include nagmk26.h
 void objfun (const Integer *nvar, const double x[], const Integer *nres, double rx[], Integer *inform, Integer iuser[], double ruser[], void **cpuser)
1:     $\mathbf{nvar}$ – IntegerInput
On entry: $n$, the number of variables in the problem, as set during the initialization of the handle by e04raf.
2:     $\mathbf{x}\left({\mathbf{nvar}}\right)$ – Real (Kind=nag_wp) arrayInput
On entry: $x$, the vector of variable values at which the residuals ${r}_{i}$ are to be evaluated.
3:     $\mathbf{nres}$ – IntegerInput
On entry: ${m}_{r}$, the number of residuals in the problem, as set during the initialization of the handle by e04rmf.
4:     $\mathbf{rx}\left({\mathbf{nres}}\right)$ – Real (Kind=nag_wp) arrayOutput
On exit: the value of the residuals ${r}_{i}\left(x\right)$ at $x$.
5:     $\mathbf{inform}$ – IntegerInput/Output
On entry: a non-negative value.
On exit: may be used to request the solver to stop immediately. Specifically, if ${\mathbf{inform}}<0$ then the value of rx will be discarded and the solver will terminate immediately with ${\mathbf{ifail}}={\mathbf{20}}$ otherwise, the solver will proceed normally.
6:     $\mathbf{iuser}\left(*\right)$ – Integer arrayUser Workspace
7:     $\mathbf{ruser}\left(*\right)$ – Real (Kind=nag_wp) arrayUser Workspace
8:     $\mathbf{cpuser}$ – Type (c_ptr)User Workspace
objfun is called with the arguments iuser, ruser and cpuser as supplied to e04fff. You should use the arrays iuser and ruser and the data handle cpuser to supply information to objfun.
objfun must either be a module subprogram USEd by, or declared as EXTERNAL in, the (sub)program from which e04fff is called. Arguments denoted as Input must not be changed by this procedure.
Note: objfun should not return floating-point NaN (Not a Number) or infinity values, since these are not handled by e04fff. If your code inadvertently does return any NaNs or infinities, e04fff is likely to produce unexpected results.
3:     $\mathbf{mon}$ – Subroutine, supplied by the NAG Library or the user.External Procedure
mon is provided to enable you to monitor the progress of the optimization and, if necessary, to halt the optimization process using argument inform.
If no monitoring is required, mon may be the dummy subroutine e04ffu supplied in the NAG Library.
mon is called at the end of every ${i}^{\mathrm{th}}$ step where $i$ is controlled by the optional parameter DFLS Monitor Frequency (default value $0$, mon is never called).
The specification of mon is:
Fortran Interface
 Subroutine mon ( nvar, x,
 Integer, Intent (In) :: nvar Integer, Intent (Inout) :: inform, iuser(*) Real (Kind=nag_wp), Intent (In) :: x(nvar), rinfo(100), stats(100) Real (Kind=nag_wp), Intent (Inout) :: ruser(*) Type (c_ptr), Intent (In) :: cpuser
#include nagmk26.h
 void mon (const Integer *nvar, const double x[], Integer *inform, const double rinfo[], const double stats[], Integer iuser[], double ruser[], void **cpuser)
1:     $\mathbf{nvar}$ – IntegerInput
On entry: $n$, the number of variables in the problem.
2:     $\mathbf{x}\left({\mathbf{nvar}}\right)$ – Real (Kind=nag_wp) arrayInput
On entry: the current best point.
3:     $\mathbf{inform}$ – IntegerInput/Output
On entry: a non-negative value.
On exit: may be used to request the solver to stop immediately. Specifically, if ${\mathbf{inform}}<0$ then the value of rx will be discarded and the solver will terminate immediately with ${\mathbf{ifail}}={\mathbf{20}}$ otherwise, the solver will proceed normally.
4:     $\mathbf{rinfo}\left(100\right)$ – Real (Kind=nag_wp) arrayInput
On entry: best objective value computed and various indicators (the values are as described in the main argument rinfo).
5:     $\mathbf{stats}\left(100\right)$ – Real (Kind=nag_wp) arrayInput
On entry: solver statistics at the end of the current iteration (the values are as described in the main argument stats).
6:     $\mathbf{iuser}\left(*\right)$ – Integer arrayUser Workspace
7:     $\mathbf{ruser}\left(*\right)$ – Real (Kind=nag_wp) arrayUser Workspace
8:     $\mathbf{cpuser}$ – Type (c_ptr)User Workspace
mon is called with the arguments iuser, ruser and cpuser as supplied to e04fff. You should use the arrays iuser and ruser and the data handle cpuser to supply information to mon.
mon must either be a module subprogram USEd by, or declared as EXTERNAL in, the (sub)program from which e04fff is called. Arguments denoted as Input must not be changed by this procedure.
4:     $\mathbf{nvar}$ – IntegerInput
On entry: $n$, the number of variables in the problem. It must be unchanged from the value set during the initialization of the handle by e04raf.
Constraint: ${\mathbf{nvar}}\ge 2$.
5:     $\mathbf{x}\left({\mathbf{nvar}}\right)$ – Real (Kind=nag_wp) arrayInput/Output
On entry: ${x}_{0}$, the initial estimates of the variables $x$.
On exit: the final values of the variables $x$.
6:     $\mathbf{nres}$ – IntegerInput
On entry: ${m}_{r}$, the number of residuals in the problem. It must be unchanged from the value set during the definition of the objective structure by e04rmf.
7:     $\mathbf{rx}\left({\mathbf{nres}}\right)$ – Real (Kind=nag_wp) arrayOutput
On exit: the values of the residuals at the final point given in x.
8:     $\mathbf{rinfo}\left(100\right)$ – Real (Kind=nag_wp) arrayOutput
On exit: optimal objective value and various indicators at the end of the final iteration as given in the table below:
 $1$ objective function value $f\left(x\right)$ (sum of the squared residuals); $2$ $\rho$, the size of trust region at the end of the algorithm; $3$ the number of interpolation points used by the solver. $4-100$ reserved for future use.
9:     $\mathbf{stats}\left(100\right)$ – Real (Kind=nag_wp) arrayOutput
On exit: solver statistics at the end of the final iteration as given in the table below:
 $1$ number of calls to the objective function; $2$ if Stats Time is activated, total time spent in the solver (including time spent evaluating the objective); $3$ if Stats Time is activated, total time spent evaluating the objective function; $4$ number of steps. $5-100$ reserved for future use.
10:   $\mathbf{iuser}\left(*\right)$ – Integer arrayUser Workspace
11:   $\mathbf{ruser}\left(*\right)$ – Real (Kind=nag_wp) arrayUser Workspace
12:   $\mathbf{cpuser}$ – Type (c_ptr)User Workspace
iuser, ruser and cpuser are not used by e04fff, but are passed directly to objfun and mon and may be used to pass information to these routines. If you do not need to reference cpuser, it should be initialized to c_null_ptr.
13:   $\mathbf{ifail}$ – IntegerInput/Output
On entry: ifail must be set to $0$, $-1\text{​ or ​}1$. If you are unfamiliar with this argument you should refer to Section 3.4 in How to Use the NAG Library and its Documentation for details.
For environments where it might be inappropriate to halt program execution when an error is detected, the value $-1\text{​ or ​}1$ is recommended. If the output of error messages is undesirable, then the value $1$ is recommended. Otherwise, because for this routine the values of the output arguments may be useful even if ${\mathbf{ifail}}\ne {\mathbf{0}}$ on exit, the recommended value is $-1$. When the value $-\mathbf{1}\text{​ or ​}1$ is used it is essential to test the value of ifail on exit.
On exit: ${\mathbf{ifail}}={\mathbf{0}}$ unless the routine detects an error or a warning has been flagged (see Section 6).

## 6Error Indicators and Warnings

If on entry ${\mathbf{ifail}}=0$ or $-1$, explanatory error messages are output on the current error message unit (as defined by x04aaf).
Note: e04fff may return useful information for one or more of the following detected errors or warnings.
Errors or warnings detected by the routine:
${\mathbf{ifail}}=1$
The supplied handle does not define a valid handle to the data structure for the NAG optimization modelling suite. It has not been initialized by e04raf or it has been corrupted.
${\mathbf{ifail}}=2$
The problem is already being solved.
The solver does not support the model defined in the handle.
It supports only nonlinear least squares problems with bound constraints.
${\mathbf{ifail}}=4$
The information supplied does not match with that previously stored.
On entry, ${\mathbf{nres}}=〈\mathit{\text{value}}〉$ must match that given during the definition of the objective in the handle, i.e., $〈\mathit{\text{value}}〉$.
The information supplied does not match with that previously stored.
On entry, ${\mathbf{nvar}}=〈\mathit{\text{value}}〉$ must match that given during initialization of the handle, i.e., $〈\mathit{\text{value}}〉$.
${\mathbf{ifail}}=5$
Inconsistent optional arguments DFLS Trust Region Tolerance ${\rho }_{\mathrm{end}}$ and DFLS Starting Trust Region ${\rho }_{\mathrm{beg}}$.
Constraint: ${\rho }_{\mathrm{end}}<{\rho }_{\mathrm{beg}}$.
Use e04zmf to set compatible option values.
Inconsistent optional arguments DFLS Trust Region Tolerance ${\rho }_{\mathrm{end}}$ and DFLS Trust Region Slow Tol ${\rho }_{\mathrm{tol}}$.
Constraint: ${\rho }_{\mathrm{end}}<{\rho }_{\mathrm{tol}}$.
Use e04zmf to set compatible option values.
Optional argument DFLS Starting Trust Region ${\rho }_{\mathrm{beg}}=〈\mathit{\text{value}}〉$, ${l}_{x}\left(i\right)=〈\mathit{\text{value}}〉$, ${u}_{x}\left(i\right)=〈\mathit{\text{value}}〉$ and $i=〈\mathit{\text{value}}〉$.
Constraint: if ${l}_{x}\left(i\right)\ne {u}_{x}\left(i\right)$ in coordinate $i$, then ${u}_{x}\left(i\right)-{l}_{x}\left(i\right)\ge 2×{\rho }_{\mathrm{beg}}$.
Use e04zmf to set compatible option values.
${\mathbf{ifail}}=6$
There were ${n}_{r}=〈\mathit{\text{value}}〉$ unequal bounds.
Constraint: ${n}_{r}\ge 2$.
There were ${n}_{r}=〈\mathit{\text{value}}〉$ unequal bounds and the optional argument DFLS Number Interp Points $\mathit{npt}=〈\mathit{\text{value}}〉$
Constraint: ${n}_{r}+2\le \mathit{npt}\le \frac{\left({n}_{r}+1\right)×\left({n}_{r}+2\right)}{2}$.
Use e04zmf to set compatible option values.
${\mathbf{ifail}}=18$
The predicted reduction in a trust region step was non-positive. Check your specification of objfun and whether the function needs rescaling. Try a different initial x.
${\mathbf{ifail}}=19$
A rescue procedure has been called in order to correct damage from rounding errors when computing an update to a quadratic approximation of $F$, but no further progress could be made. Check your specification of objfun and whether the function needs rescaling. Try a different initial x.
${\mathbf{ifail}}=20$
User requested termination after a call to the objective function. inform was set to a negative value within the user-supplied function objfun.
User requested termination during a monitoring step. inform was set to a negative value within the user-supplied function mon
${\mathbf{ifail}}=21$
Maximum number of function evaluations exceeded.
${\mathbf{ifail}}=23$
The solver terminated after the maximum time allowed was exceeded.
Maximum number of seconds exceeded. Use option Time Limit to reset the limit.
${\mathbf{ifail}}=24$
No progress, the solver was stopped after $〈\mathit{\text{value}}〉$ consecutive slow steps.
Use the optional argument DFLS Maximum Slow Steps to modify the maximum number of slow steps accepted.
The solver stopped after $5×{\mathbf{DFLS Maximum Slow Steps}}$ consecutive slow steps and a trust region above the tolerance set by DFLS Trust Region Slow Tol.
${\mathbf{ifail}}=50$
The problem was solved to an acceptable level after $〈\mathit{\text{value}}〉$ consecutive slow iterations.
Use the optional argument DFLS Maximum Slow Steps to modify the maximum number of slow steps accepted.
The solver stopped after DFLS Maximum Slow Steps consecutive slow steps and a trust region below the tolerance set by DFLS Trust Region Slow Tol.
${\mathbf{ifail}}=-99$
See Section 3.9 in How to Use the NAG Library and its Documentation for further information.
${\mathbf{ifail}}=-399$
Your licence key may have expired or may not have been installed correctly.
See Section 3.8 in How to Use the NAG Library and its Documentation for further information.
${\mathbf{ifail}}=-999$
Dynamic memory allocation failed.
See Section 3.7 in How to Use the NAG Library and its Documentation for further information.

## 7Accuracy

The solver can declare convergence on two conditions:
 (i) The trust region radius is below the tolerance ${\rho }_{\mathrm{end}}$ set by the optional parameter DFLS Trust Region Tolerance. When this condition is met, the corresponding solution will generally be at a distance lower than $10×{\rho }_{\mathrm{end}}$ of a local minimimum. (ii) The sum of the square of the residuals is below the tolerance set by the optional parameter DFLS Small Residuals Tol. In a data fitting context, this condition means that the error between the observed data and the model is smaller than the requested tolerance.

## 8Parallelism and Performance

e04fff makes calls to BLAS and/or LAPACK routines, which may be threaded within the vendor library used by this implementation. Consult the documentation for the vendor library for further information.
Please consult the X06 Chapter Introduction for information on how to control and interrogate the OpenMP environment used within this routine. Please also consult the Users' Note for your implementation for any additional implementation-specific information.

### 9.1Description of the Printed Output

The solver can print information to give an overview of the problem and of the progress of the computation. The output may be sent to two independent unit numbers which are set by optional parameters Print File and Monitoring File. Optional parameters Print Level, Print Options, Monitoring Level and Print Solution determine the exposed level of detail. This allows, for example, a detailed log file to be generated while the condensed information is displayed on the screen.
By default (${\mathbf{Print File}}=6$, ${\mathbf{Print Level}}=2$), four sections are printed to the standard output: a header, a list of options, an iteration log and a summary.
```  ----------------------------------------------
E04FF, Derivative-free solver for data fitting
(nonlinear least squares problems)
----------------------------------------------
Problem statistics
Number of variables                 10
Number of unconstrained variables   10
Number of fixed variables            0
Number of residuals                 10```
Optional parameters list
If Print Options is set to $\mathrm{YES}$, a list of the optional parameters and their values is printed. The list shows all options of the solver, each displayed on one line. The line contains the option name, its current value and an indicator for how it was set. The options left at their defaults are noted by (d) and the ones set by the user are noted by (U). Note that the output format is compatible with the file format expected by e04zpf. The output looks as follows:
```     Stats Time                    =                 Yes     * U
Dfls Trust Region Tolerance   =         1.00000E-07     * U
Dfls Max Objective Calls      =                 500     * d
Dfls Starting Trust Region    =         1.10000E-01     * U
Dfls Number Interp Points     =                   0     * d```
Iteration log
If ${\mathbf{Print Level}}\ge 2$, the solver will print a summary line for each step. An iteration is considered successful when it yields a decrease of the objective sufficiently close to the decrease predicted by the quadratic model. The line shows the step number, the value of the objective function, the radius of the trust region and the cumulative number of objective function evaluations. The output looks as follow:
```  ----------------------------------------
step |    obj       rho     |    nf   |
----------------------------------------
1 |  3.82E+01  1.10E-01  |    13   |
2 |  3.55E+01  1.10E-01  |    14   |
3 |  3.05E+01  1.10E-01  |    15   |
4 |  2.15E+01  1.10E-01  |    18   |```
Occasionally, the letter ‘s’ is printed at the end of the line indicating that the progress is considered slow by the slow convergence detection heuristic. After a certain number of consecutive slow steps, the solver is stopped. The limit on the number of slow iterations can be controlled by the optional parameter DFLS Maximum Slow Steps and the tolerance on the trust region radius before the solver can be stopped is driven by DFLS Trust Region Slow Tol.
Summary
Once the solver finishes, a summary is produced:
```  Status: Converged, small trust region size.

Value of the objective                    2.23746E-06
Number of objective function evaluations          107
Number of steps                                    51```
Optionally, if Stats Time is set to $\mathrm{YES}$, the timings are printed:
```  Timings
Total time spent in the solver            0.056
Time spent in the objective evaluation    0.012```
Additionally, if Print Solution is set to $\mathrm{YES}$, the solution is printed along with the bounds:
```  Computed Solution:
idx   Lower bound        Value      Upper bound
1       -inf       -1.00000E+00        inf
2       -inf       -1.00000E+00        inf
3       -inf       -1.00000E+00        inf
4       -inf       -1.00000E+00        inf```

## 10Example

In this example, we minimize the Kowalik and Osborne function with bounds on some of the variables. In this problem, the number of variables $n=4$ and the number of residuals ${m}_{r}=11$. The residuals ${r}_{i}$ are computed by
 $rix = zi- yi yi+ x2 yi yi+ x3 +x4 x1$ (2)
where
 $y = 4.0000,2.0000,1.0000,0.5000,0.2500,0.1670,0.1250,0.1000,0.0833,0.0714,0.0625 z = 0.1957,0.1947,0.1735,0.1600,0.0844,0.0627,0.0456,0.0342,0.0323,0.0235,0.0246$ (3)
The following bounds are defined on the variables
 $0.2 ≤ x2 ≤ 1.0 0.3 ≤ x4$ (4)
The initial guess is
 $x0= 0.25,0.39,0.415,0.39$ (5)
The expected solution is
 $x*= 0.1813,0.5901,0.2569,0.3000$ (6)

### 10.1Program Text

Program Text (e04fffe.f90)

None.

### 10.3Program Results

Program Results (e04fffe.r)

## 11Algorithmic Details

This section contains a short description of the algorithm used in e04fff which is based on Powell's method BOBYQA Powell (2009) and the work of Zhang et al. (2010). It is based on a model-based derivative-free trust region framework adapted to exploit least squares problems structure.

### 11.1Derivative-free trust region algorithm

In this section, we are interested in generic problems of the form
 $minimize x∈ℝn fx$ (7)
where the derivatives of the objective function $f$ are not easily available. A model-based derivative-free optimization (DFO) algorithm maintains a set of points ${Y}_{k}$ centred on an iterate ${x}_{k}$ to build quadratic interpolation models of the objective
 $fxk+s ≈ ϕks= fxk+ gkTs+ 12sTHks$ (8)
where ${g}_{k}$ and ${H}_{k}$ are built with the interpolation conditions
 $∀y∈Yk , ​ϕk y-xk =fy$ (9)
Note that if the number of interpolation points $\mathit{npt}$ is smaller than $\frac{\left({n}_{r}+1\right)×\left({n}_{r}+2\right)}{2}$, the model chosen is the one for which the hessian ${H}_{k}$ is the closest to ${H}_{k-1}$ in the Frobenius norm sense. This model is iteratively optimized over a trust region, updated and moved around the new computed points. More precisely, it can be described as:
• DFO Algorithm
• 1. Initialization
Choose an initial interpolation set ${Y}_{0}$, trust region radius ${\rho }_{\mathrm{beg}}$ and build the first quadratic model ${\varphi }_{0}$.
2. Iteration k  (i) Minimize the model in the trust region to obtain a step ${s}_{k}$. (ii) If the step is too small, adjust the geometry of the interpolation set and the trust region size ${\rho }_{k}$ and restart the iteration. (iii) Evaluate the objective at the new point ${x}_{k}+{s}_{k}$. (iv) Replace a far away point from ${Y}_{k}$ by ${x}_{k}+{s}_{k}$ to create ${Y}_{k+1}$. (v) If the decrease of the objective is sufficient (successful step), choose ${x}_{k+1}={x}_{k}+{s}_{k}$, else choose ${x}_{k+1}={x}_{k}$. (vi) Choose ${\rho }_{k+1}$ and adjust the geometry of ${Y}_{k+1}$, if necessary. (vii) Build ${\varphi }_{k+1}$ using the new interpolation set. (viii) Stop the algorithm if ${\rho }_{k+1}$is below the chosen tolerance ${\rho }_{\mathrm{end}}$.
In the rest of this documentation page, we call an iteration successful when the trial point ${x}_{k}+{s}_{k}$ is accepted as the next iterate.

### 11.2Bounds on the variables

The bounds on the variables are handled during the model optimization step (step 2(i) of DFO Algorithm) with an active set method. If a bound is hit, it is fixed and step 2(i) is restarted. The set of active constraints is kept throughout the optimization, progressively fixing the corresponding variables.

### 11.3Adaptation to nonlinear least squares problems

In the specific case where $f$ is a sum of square $f\left(x\right)=\sum _{i=1}^{{m}_{r}}{{r}_{i}\left(x\right)}^{2}$, a good approximation of the hessian of the objective can be
 $∇2fx ≈ JxTJx$ (10)
where $J$ is the ${m}_{r}$ by $n$ first derivative matrix of $f$. This approximation is the main idea behind the Gauss–Newton and Levenberg–Marquardt methods. Following the work of Zhang et al. (2010), it is possible to adapt it to the DFO framework. In e04fff, one quadratic model is built for each residual ${r}_{i}$
 $rix+s ≈ rix+ g(i)Ts+ 12sTH(i)s$ (11)
We call $J={\left({g}^{\left(1\right)},{g}^{\mathrm{\left(2\right)}},\mathrm{...}\right)}^{T}$. To build the model of the objective $f$, we then choose
 $fx+s ≈ ϕs= fx+ gfTs+ 12sTHfs$ (12)
where ${g}_{f}$ is chosen as
 $gf= JTfx$ (13)
and ${H}_{f}$ as
 $Hf= JTJ + 0 if gf≥κ1 κ3 fxI if gf<κ1 and 12 fx< κ2gf ∑ i=1 mr rix H(i) otherwise$ (14)
The constants ${\kappa }_{1}$, ${\kappa }_{2}$ and ${\kappa }_{3}$ are chosen as proposed in Zhang et al. (2010). The first expression amounts to making a Gauss–Newton approximation when we are far from a stationary point, the second to a Levenberg–Marquardt approximation when we are close to a stationary point with small residuals while the third takes the full hessian into account.
e04fff integrates this method of building models into the framework presented in the algorithm DFO Algorithm.

## 12Optional Parameters

Several optional parameters in e04fff define choices in the problem specification or the algorithm logic. In order to reduce the number of formal arguments of e04fff these optional parameters have associated default values that are appropriate for most problems. Therefore, you need only specify those optional parameters whose values are to be different from their default values.
The remainder of this section can be skipped if you wish to use the default values for all optional parameters.
The optional parameters can be changed by calling e04zmf anytime between the initialization of the handle by e04raf and the call to the solver. Modification of the arguments during intermediate monitoring stops is not allowed. Once the solver finishes, the optional parameters can be altered again for the next solve.
The option values may be retrieved by e04znf.
The following is a list of the optional parameters available. A full description of each optional parameter is provided in Section 12.1.

### 12.1Description of the Optional Parameters

For each option, we give a summary line, a description of the optional parameter and details of constraints.
The summary line contains:
• the keywords, where the minimum abbreviation of each keyword is underlined;
• a parameter value, where the letters $a$, $i$ and $r$ denote options that take character, integer and real values respectively.
• the default value, where the symbol $\epsilon$ is a generic notation for machine precision (see x02ajf).
All options accept the value $\mathrm{DEFAULT}$ to return single options to their default states.
Keywords and character values are case and white space insensitive.
 Defaults
This special keyword may be used to reset all optional parameters to their default values. Any argument value given with this keyword will be ignored.
 DFLS Maximum Slow Steps $i$ Default $=20$
If ${\mathbf{DFLS Maximum Slow Steps}}>0$, this argument defines the maximum number of consecutive slow iterations ${n}_{\mathrm{slow}}$ allowed. Set it to 0 to deactivate the slow iteration detection. The algorithm can stop in two situations:
${n}_{\mathrm{slow}}>{\mathbf{DFLS Maximum Slow Steps}}$ and $\rho <{\mathbf{DFLS Trust Region Slow Tol}}$ with ${\mathbf{ifail}}={\mathbf{50}}$
${n}_{\mathrm{slow}}>5×{\mathbf{DFLS Maximum Slow Steps}}$ with ${\mathbf{ifail}}={\mathbf{24}}$
Constraint: ${\mathbf{DFLS Maximum Slow Steps}}\ge 0$.
 DFLS Max Objective Calls $i$ Default $=500$
A limit on the number of objective function evaluations the solver is allowed to compute. If the limit is reached, the solver stops with ${\mathbf{ifail}}={\mathbf{21}}$.
Constraint: ${\mathbf{DFLS Max Objective Calls}}\ge 1$.
 DFLS Monitor Frequency $i$ Default $=0$
If ${\mathbf{DFLS Monitor Frequency}}>0$, the solver calls the user defined monitoring function mon at the end of every $i$th step.
Constraint: ${\mathbf{DFLS Monitor Frequency}}\ge 0$.
 DFLS Number Interp Points $i$ Default $=0$
The number of interpolation points in ${Y}_{k}$ (9) used to build the quadratic models. If ${\mathbf{DFLS Number Interp Points}}=0$, the number of points is chosen to be ${n}_{r}+2$ where ${n}_{r}$ is the number of non-fixed variables.
Constraint: ${\mathbf{DFLS Number Interp Points}}\ge 0$.
Consistency constraint, the solver stops with ${\mathbf{ifail}}={\mathbf{6}}$ if not met:
${n}_{r}+2\le {\mathbf{DFLS Number Interp Points}}\le \frac{\left({n}_{r}+1\right)×\left({n}_{r}+2\right)}{2}$.
 DFLS Print Frequency $i$ Default $=1$
If ${\mathbf{DFLS Print Frequency}}>0$, the solver prints the iteration log to the appropriate units at the end of every $i$th step.
Constraint: ${\mathbf{DFLS Print Frequency}}\ge 0$.
 DFLS Small Residuals Tol $r$ Default $={\epsilon }^{0.75}$
This option defines the tolerance on the value of the residuals. Namely, the solver declares convergence if
$f\left(x\right)=\sum _{i=1}^{{m}_{r}}{{r}_{i}\left(x\right)}^{2}<{\mathbf{DFLS Small Residuals Tol}}$.
Constraint: ${\mathbf{DFLS Small Residuals Tol}}>{\epsilon }^{2}$.
 DFLS Starting Trust Region $r$ Default $=0.1$
${\rho }_{\mathrm{beg}}$, the initial trust region radius. This argument should be set to about one tenth of the greatest expected overall change to a variable: the initial quadratic model will be constructed by taking steps from the initial x of length ${\rho }_{\mathrm{beg}}$ along each coordinate direction. The default value assumes that the variables have an order of magnitude 1.
Constraint: ${\mathbf{DFLS Starting Trust Region}}>\epsilon$.
Consistency constraints, the solver stops with ${\mathbf{ifail}}={\mathbf{5}}$ if not met:
${\mathbf{DFLS Starting Trust Region}}\le {\mathbf{DFLS Trust Region Tolerance}}$.
${\mathbf{DFLS Starting Trust Region}}\le \frac{1}{2}\underset{i}{\mathrm{min}}\phantom{\rule{0.25em}{0ex}}\left({u}_{x}\left(i\right)-{l}_{x}\left(i\right)\right)$
 DFLS Trust Region Tolerance $r$ Default $={\epsilon }^{0.37}$
${\rho }_{\mathrm{end}}$, the requested trust region radius. The algorithm declares convergence when the trust region radius reaches this limit. It should indicate the absolute accuracy that is required in the final values of the variables.
Constraint: ${\mathbf{DFLS Trust Region Tolerance}}>\epsilon$.
Consistency constraints, the solver stops with ${\mathbf{ifail}}={\mathbf{5}}$ if not met:
${\mathbf{DFLS Starting Trust Region}}>{\mathbf{DFLS Trust Region Tolerance}}$.
 DFLS Trust Region Slow Tol $r$ Default $\text{}={\epsilon }^{0.25}$
The minimal acceptable trust region radius for the solution to be declared as acceptable. The solver stops if:
${n}_{\mathrm{slow}}>{\mathbf{DFLS Maximum Slow Steps}}$ and ${\rho }_{k}<{\mathbf{DFLS Trust Region Slow Tol}}$
Constraint: ${\mathbf{DFLS Trust Region Slow Tol}}>\epsilon$.
Consistency constraints, the solver stops with ${\mathbf{ifail}}={\mathbf{5}}$ if not met:
${\mathbf{DFLS Trust Region Slow Tol}}>{\mathbf{DFLS Trust Region Tolerance}}$
 DFLS Trust Region Update $a$ Default $=\mathrm{FAST}$
Controls the speed at which the trust region is decreased after unsuccessful iterations. In smooth non-noisy cases, a fast decrease often leads to faster convergence. However, in noisy cases, a slow decrease is recommended to avoid premature stops.
Constraint: ${\mathbf{DFLS Trust Region Update}}=\mathrm{FAST}$ or $\mathrm{SLOW}$.
 Infinite Bound Size $r$ Default $\text{}={10}^{20}$
This defines the ‘infinite’ bound $\mathit{bigbnd}$ in the definition of the problem constraints. Any upper bound greater than or equal to $\mathit{bigbnd}$ will be regarded as $+\infty$ (and similarly any lower bound less than or equal to $-\mathit{bigbnd}$ will be regarded as $-\infty$). Note that a modification of this optional parameter does not influence constraints which have already been defined; only the constraints formulated after the change will be affected.
Constraint: ${\mathbf{Infinite Bound Size}}\ge 1000$.
 Monitoring File $i$ Default $=-1$
If $i\ge 0$, the unit number for the secondary (monitoring) output. If set to $-1$, no secondary output is provided. The information output to this unit is controlled by Monitoring Level.
Constraint: ${\mathbf{Monitoring File}}\ge -1$.
 Monitoring Level $i$ Default $=4$
This argument sets the amount of information detail that will be printed by the solver to the secondary output. The meaning of the levels is the same as with Print Level.
Constraint: $0\le {\mathbf{Monitoring Level}}\le 5$.
 Print File $i$ Default $=\mathrm{advisory message unit number}$
If $i\ge 0$, the unit number for the primary output of the solver. If ${\mathbf{Print File}}=-1$, the primary output is completely turned off independently of other settings. The default value is the advisory message unit number as defined by x04abf at the time of the optional parameters initialization, e.g., at the initialization of the handle. The information output to this unit is controlled by Print Level.
Constraint: ${\mathbf{Print File}}\ge -1$.
 Print Level $i$ Default $=2$
This argument defines how detailed information should be printed by the solver to the primary and secondary output.
 $i$ Output $0$ No output from the solver $1$ The Header and Summary. $2$,$3$,$4$,$5$ Additionally, the Iteration log.
Constraint: $0\le {\mathbf{Print Level}}\le 5$.
 Print Options $a$ Default $=\mathrm{YES}$
If ${\mathbf{Print Options}}=\mathrm{YES}$, a listing of optional parameters will be printed to the primary output. It is always printed to the secondary output.
Constraint: ${\mathbf{Print Options}}=\mathrm{YES}$ or $\mathrm{NO}$.
 Print Solution $a$ Default $=\mathrm{NO}$
If ${\mathbf{Print Solution}}=\mathrm{YES}$, the solution will be printed to the primary and secondary output.
Constraint: ${\mathbf{Print Solution}}=\mathrm{NO}$ or $\mathrm{YES}$.
 Stats Time $a$ Default $=\mathrm{NO}$
This argument turns on timings of various parts of the algorithm to give a better overview of where most of the time is spent. This might be helpful for a choice of different solving approaches. It is possible to choose between CPU and wall clock time. Choice $\mathrm{YES}$ is equivalent to wall clock.
Constraint: ${\mathbf{Stats Time}}=\mathrm{YES}$, $\mathrm{NO}$, $\mathrm{CPU}$ or $\mathrm{WALL CLOCK}$.
 Time Limit $r$ Default $\text{}={10}^{6}$
A limit on seconds that the solver can use to solve one problem. If during the convergence check this limit is exceeded, the solver will terminate with ${\mathbf{ifail}}={\mathbf{23}}$ error message.
Warning: the timings are not computed if Stats Time is set to $\mathrm{NO}$. The solver will therefore NOT be stopped if the time limit is exceeded in such a case.
Constraint: ${\mathbf{Time Limit}}>0$.
© The Numerical Algorithms Group Ltd, Oxford, UK. 2017