When the PDETAIL option is not specified, a standard table is produced displaying the iteration number,
the value of the objective function at that iteration, and
norm of the gradient vector
This table is useful for overall convergence behavior. However, unlike the table produced by the
In the case of sum of squared error, which results from specifying OBJECTIVE=DEV in the NETOPTS
statement and ERROR=NORMAL in the TARGET statement, the error function serves as the objective
function, and is given by
Candidate network weight values
that minimize the objective function
satisfy the first order
Hence, a natural convergence criteria is
for some small value . This is, in fact, convergence criteria for all prop methods. The value of is set
by the ABSGCONV= option in the NLOPTIONS statement, with a default value of
components of z.
The standard table prints the following quantities:
, using current network weight
Max Abs Gradient Element,
When the PDETAIL option is specified, this standard table is still printed. Each line of the standard
tables follows the detail lines for each of the weights at each iteration.
The Standard Backprop Algorithm
The standard backprop algorithm is a gradient descent with momentum.
At the nth iteration, the update is computed as
For TECH=BPROP, the PDETAIL option in the TRAIN statement results in the following quantities
is labeled "Previous Change"
is labeled "Gradient"
is labeled "Current Change"
is labeled "Previous Weight"
is labeled "Current Weight"
The learning rate and the momentum are printed at the beginning of the table. These quantities are
set by the LEARN= and MOMENTUM= options respectively, with the default values of
The RPROP Algorithm
use the magnitude of the gradient in calculating the weight update. Instead, the signs of the current and
previous gradient are used to determine a step size
at each iteration.
is bounded by
The value of
is set by the MAXLEARN= option with a default value of
. The value of
is set by the MINLEARN= option with a default value of
. Note that these values are
substantially different from the recommendations given in Schiffmann, Joost and Werner, (1994). These
new values improved stability and convergence over a wide range of problems.
For each connection weight, an initial stepsize
is given a small value. According to
Schiffmann, Joost and Werner, (1994), results are not typically dependent on the exact value given
. PROC NEURAL uses a default initial step size of 0.1 for all weights and is set by the
LEARN= option in the TRAIN statement.
At the nth iteration, adjust the step size by
The factors u and d are the acceleration and deceleration respectively. The values of these factors are set
by the ACCELERATE= and DECELERATE= options in the TRAIN statement. The default value for
ACCELERATE= 1.2; for DECELERATE= the default value is 0.5.
For TECH=RPROP, the PDETAIL option in the TRAIN statement results in the following quantities
is labeled "Previous Step Size"
is labeled "Previous Gradient"
is labeled "Current Gradient"
The Quickprop algorithm (Fahlman 1989) assumes that the error function behaves locally like a
parabola, and uses the method of false position to find the minimum of the approximating quadratic.
Variations are required to ensure the change is not uphill and to handle cases where the gradient does not
change between iterations (causing the false position method to fail).
The quickprop algorithm uses a modified gradient
related to the regular gradient by
At the nth iteration, the weight update is given by
For initialization at n=1, we set
, so the update step becomes a gradient descent:
At the second and subsequent iterations,
first involves evaluation
, the numerical estimate of the second
This second derivative can become large in absolute value or can signal a move "up" the gradient away
from a minimum. The following modifications are applied to account for these situations.
is set by the LEARN= option in the TRAIN statement, with a default value of
. The bound
is set by the MAXMOMENTUM= option in the TRAIN statement,
with a default value of
For TECH=QPROP, the PDETAIL option in the TRAIN statement results in the following quantities
is labeled "Previous Weight"
is labeled "Modified Gradient"