Training multilayer neural networks using fast global learning algorithm - least-squares and penalized optimization methods

Cho SY, Chow TWS
NEUROCOMPUTING 25 (1-3): 115-131 APR 1999

One of the major limitations of conventional learning algorithm is the slow convergence speed caused by the problem of local minima. This paper presents a novel heuristic approach for feedforward neural networks learning. The algorithm is based upon a hybrid type of the Least Squares (LS) and penalised optimisation methods. The LS method is used to provide a fast convergence speed, and a penalty approach is introduced to solve the problem of local minima. The penalty term, defined as discontinuous functions, such as Gaussian, Rayleigh and Laplace, superimposed into the specified error surface, provides a way of escape from the local minima when the convergence stalls. As a result, the learning performance are dramatically improved. The proper selection and adaptation for the penalty factor are also demonstrate the effect of the penalty terms and to ensure the convergence of the algorithm. In all the tested problems, the proposed algorithm yields excellent results in terms of convergence speed, avoidance of local minima and quality of solution.

Some of our obtained results are shown as below, but details can be referred to our paper published in Neurocomputing Journal.

Figure 1. Comparison results for the ability of converging to the global minimum to obtain the best solution for the different algorithms on the modified XOR problem.

a) error surface of a modified Ex-or problem (b) Contour plot and search trajectory by Gaussian approach

(c) Contour plot and search trajectory byRayleigh approach (d) Contour plot and search trajectory by Laplace approach

Figure 2. 3-D error plot and 2-D contour plots with searching trajectories of the proposed algorithm for the modified XOR problem. In these figures shown, the convergence is stuck in the local minimum and then the trajectories can be pulled out from the local minimum by our penalty approaches to converge to the global minimum.

Table 2. Simulation results for the two-spiral classification problem.

Figure 3a	Figure 3b
Figure 3c	Figure 3d

Figure 3. The output classification contour images for two-spiral problem created by (a) the SIMANN algorithm; (b) the Gaussian penalty approach; (c) the Rayleigh penalty approach; (d) the Laplace penalty approach.

Source Code : GNNv2-02. zip