Smooth fitting with a method for determining the regularization parameter under the genetic programming algorithm

Abstract

This paper deals with the smooth fitting problem under the genetic programming (GP) algorithm. To reduce the computational cost required for evaluating the fitness value of GP trees, numerical weights of GP trees are estimated by adopting both linear associative memories (LAM) and the Hook and Jeeves (HJ) method. The quality of smooth fitting is critically dependent on the choice of the regularization parameter. So, we present a novel method for choosing the regularization parameter. Two numerical examples are given with the comparison of generalized cross-validation (GCV) B-splines.

Introduction

We consider an univariate function approximation. The underlying model for approximations is

$$\text{y(t)=μ(t)+ε(t),}$$

where μ(t) is an unknown function, and ε(t) is noise such that E(ε(t))=0, and variance is σ². y(t) is observed for $\text{t=t}{}_0\text{,t}{}_1\text{,…,t}{}_{\mathrm{n}}\text{(t}{}_0\text{<t}{}_1\text{<⋯<t}{}_{\mathrm{n}}\text{)}$, and from this, a learning set L {(t_i,y_i)∣y_i≡y(y_i)}_i=0,…,n is constructed. The goal is to find $\text{μ}\text{̄}\text{(t)}$, which is the estimate of μ(t), using L. There may be a lot of methods for $\text{μ}\text{̄}\text{(t)}$. For instance, polynomials, locally weighted regressions, splines, and neural networks can be used.

In the paper, we try to use genetic programming (GP) [1], [2] for seeking the good estimate of μ(t). To do this, two major problems should be resolved.

First, numerical weights attached to nodes of the GP tree should be estimated in a computationally efficient way. The choice of an optimal GP tree requires the estimation of numerical parameters or weights that are attached to some nodes of a GP tree [3], [4]. GP trees with the poor estimation of its numerical weights could earn a very low grade, and be readily excluded in the next evolving process, although they are potentially good candidates for μ(t). The original GP algorithm mainly focuses on dynamically modifying the structure of GP trees, and thus suffers from a lack of estimation techniques for numerical weights. Provided that nonlinear optimization methods are applied to estimating weights of the GP tree [3], [4], [5], [6], [7], [8], [9], it is a very time-consuming process since usually the population consists of several hundreds or even thousands of GP trees. The approach taken in this paper is to use both linear associative memories (LAM) [10], [11], [12], [13], [14], [15] and the Hook and Jeeves (HJ) search method [16], [17] in a combined manner. This allows to reduce significant amounts of the computational cost.

Second, since L is corrupted by noise, the fitness function should contain the regularization term for smooth fitting. Thus, the fitness function consists of two terms; one that takes account into how well the GP tree is fitted against L, and another one, called the regularization term that represents the degree of smoothness [18]. Here, the most important task is to select a proper regularization parameter enabling to attain a solution that is near the data given in L and, at the same time, is as smooth as possible. If the parameter is too small, the solution shows unwanted oscillations, and if the parameter is too large, the solution shows oversmooths. Since, most popular parameter selection methods [19], [20], [21], [22], [23], [24] pose difficulties in being used under the GP algorithm with the estimation of numerical weight, so we have devised a simple heuristic method. This method is very computationally efficient, and sufficient for selecting good GP trees. As far as we know, our paper is the first reported one that concerns with smooth fitting with the choice of the adequate regularization parameter in the GP algorithm.

Numerical examples with the comparison of generalized cross-validation (GCV) B-spline [20], [25] are given in this paper. The results show that the GP tree outperforms GCV B-spline in most cases. Especially, the estimation of differentiation by the GP tree is far better than that of GCV B-spline.

The paper is organized as follows. In Section 2, we discuss the regularized fitness function and the efficient way of estimating numerical weights of GP trees. Also, in this section, the appropriate function set used in generating GP trees is considered. The method for choosing the regularization parameter is presented in Section 3. Section 4 contains brief descriptions for the overall framework of smooth fitting in the GP algorithm. Numerical examples are given in Section 5. Finally, Section 6 summarizes the results of the paper and offers concluding remarks.