Hoeffding bound based evolutionary algorithm for symbolic regression

https://doi.org/10.1016/j.engappai.2012.04.005Get rights and content

Abstract

In symbolic regression area, it is difficult for evolutionary algorithms to construct a regression model when the number of sample points is very large. Much time will be spent in calculating the fitness of the individuals and in selecting the best individuals within the population. Hoeffding bound is a probability bound for sums of independent random variables. As a statistical result, it can be used to exactly decide how many samples are necessary for choosing i individuals from a population in evolutionary algorithms without calculating the fitness completely. This paper presents a Hoeffding bound based evolutionary algorithm (HEA) for regression or approximation problems when the number of the given learning samples is very large. In HEA, the original fitness function is used in every k generations to update the approximate fitness obtained by Hoeffding bound. The parameter 1−δ is the probability of correctly selecting i best individuals from population P, which can be tuned to avoid an unstable evolution process caused by a large discrepancy between the approximate model and the original fitness function. The major advantage of the proposed HEA algorithm is that it can guarantee that the solution discovered has performance matching what would be discovered with a traditional genetic programming (GP) selection operator with a determinate probability and the running time can be reduced largely. We examine the performance of the proposed algorithm with several regression problems and the results indicate that with the similar accuracy, the HEA algorithm can find the solution more efficiently than tradition EA. It is very useful for regression problems with large number of training samples.

Introduction

Evolutionary algorithms have been applied successfully to a variety of complicated real-world applications. One main difficulty in these applications is that EAs usually need a large number of fitness evaluations before obtaining a satisfying result. However, when the evaluation of the fitness is computationally very expensive or an explicit fitness function does not exist, we have to construct an approximate model to estimate the fitness.

Fitness approximation has become an active area in evolutionary computation with many varying approaches and results (Jin, 2005). There are several reasons for utilizing fitness approximation through modeling. The first and most common reason is to reduce the computational complexity of expensive fitness evaluations. Many applications of evolutionary algorithms are in high-complexity or intractable domains, where the fitness calculation can be prohibitively time consuming (Ong et al., 2003, Jin et al., 2002, Regis and Shoemaker, 2004, Regis and Shoemaker, 2005) and the exact fitness objective is unnecessary for evolutionary progress. At the same time, approximation can be used advantageously in other problems, such as handling noisy fitness functions, smooth multimodal landscapes, and defining a continuous fitness in domains that lack an explicit fitness (Regis and Shoemaker, 2004, Regis and Shoemaker, 2005, Arnold,, Sano and Kita,, Audet et al.,).

Traditionally, there are several approaches in fitness approximation. Problem approximation tries to replace the original problem by another problem which is approximately the same to the original problem but easier to solve (Anderson., 1995, Bradshaw et al., 1976). In functional approximation, an alternate and explicit expression is constructed for the objective function. Several methods have been proposed for constructing approximate models. In (Johanson and Poli, 1998), a neural network has been used to model the behavior of the human to reduce human fatigue and evaluate the fitness. In Hart and Belew (1996), Liang et al. (2000), approximate models were used to smooth the rough fitness functions and prevent the evolutionary algorithm from stagnating in a local optimum. In Myers and Montgomery (1995), Sacks et al. (1989), a global polynomial approximation and a local Gaussian process were combined together to estimate the parameters used. Artificial neural networks, including multilayer perceptions and radial basis function networks have also been used to build approximate models for evolutionary optimization (Bartelemy and Haftka, 1993, Carpenter and Barthelemy, 1993, Shyy et al., 1999, Simpson et al., 1998). Evolutionary approximation is a type of approximation for evolutionary algorithms. Fitness inheritance is a popular evolutionary approximation method, in which fitness evaluations can be spared by estimating the fitness value of the offspring individuals from the value of their parents. In Kim and Cho (2001), the individuals are clustered into several groups and only the individual representing its cluster will be evaluated using the fitness function. The fitness value of other individuals in the same cluster will be estimated from the representative individual based a distance measure.

However, to get an accurate approximate model is a difficult task, the models achieved often have large approximation errors and always introduces a false optima. So, the evolutionary algorithm with approximate models for fitness evaluation always converges to an incorrectly solution (Kim and Cho, 2001).

How to guarantee that the solution discovered is optimal or near-optimal is a key problem we have to face. Most methods with approximate model for fitness evaluation assume that the approximate model is correct and the evolutionary algorithm will converge to a global or near-optimal solution (Redmond and Parker, 1996, Pierret, 1999). These methods can only succeed when the approximate model is globally correct. To guarantee to converge to the global solution, many frameworks for managing approximate models were proposed (More, 1983, Schramm and Zowe, 1992, Brooker et al., 1998). In Dennis and Torczon (1997), a framework was represented based on the classical trust-region methods, which ensures that the search process converges to a reasonable solution of original problem. In Ratle (1998), a heuristic convergence criterion is used to determine the updating time of the approximate model. In Ratle (1999), the original fitness function is used in every k generations to update the approximate model. However, if there is a large discrepancy between the approximate model and the original fitness function, the evolution process may become very unstable (Jin et al., 2000). In Bull (1999), an evolutionary algorithm with a neural network trained with some initial samples to approximate the model is proposed. During the evolutionary process, the fittest individual in the population is evaluated on the original fitness function once in every 50 generations. The individual then replaces the one with the lowest fitness in the training set and the neural network is retrained. However, when the complexity of the original fitness landscape is high, the neural network model will mislead the evolutionary algorithm to a default way. To improve the convergence of the evolutionary algorithm, a concept of evolution control, in which the evolution proceeds based on fitness evaluations using not only the approximate fitness model, but also the original fitness function, was proposed in Jin et al. (2002).

In this paper, we propose a new framework for evolutionary algorithm named Hoeffding evolutionary algorithm and it can be used with other traditional evolutionary algorithm together to deal with a kind of problems in which the evaluation of the fitness is computationally very expensive. The algorithm can exactly decide how many samples are necessary for choosing i best individuals from a population in evolutionary algorithms without calculating the fitness completely. The major advantage of the proposed algorithm is that it can guarantee that the solution discovered has performance matching what would be discovered with a traditional GP selection operator, with a determinate probability; and with the similar accuracy, the HEA algorithm can find the solution more efficiently than tradition EA.

The rest of this paper is organized as follows. We first review the related work on the Hoeffding bound, symbolic regression and genetic programming. Next, different kinds of fitness functions were introduced and two iterative fitness function models were built. Third, we represent the Hoeffding-EA algorithm (Hoeffding evolutionary framework with evolutionary algorithm) for symbolic regression and give some theorem analysis. Then we describe several experiments in which the parameters of the algorithm are examined and the algorithm is effectively applied to find a regression model. Finally, we conclude this paper by highlighting the key contributions of this work.

Section snippets

Hoeffding bound and relevant algorithms

The distribution function for the sum of independent random variables, x1+x2++xi++xn, when some information about the distribution of the xi is available, is very important for the modern theory of probability. Much work has been carried out on the asymptotic form of the distribution of such sums when the number of component random variables is large or when the component variables have identical distributions. If we test a given model on N samples, the average error of them can be expressed

Iterative fitness function

In general, an optimization problem requires a fitness function which is given by systems of arbitrary complexity. Different fitness functions in evolutionary optimization were mainly categorized into four classes: noise, time-varying, robustness and iteration.

The evaluation of noise fitness function is subject to noise which may come from many different sources such as sensory measurement errors or randomized simulations. The two common noises are Gaussian and Cauhy (Stroud, 2001). The

Hoeffding selection operation

Consider a real-valued random variable r whose range is B. Suppose we have made n independent observations of this variable, and computed their mean rmean. The Hoeffding bound states that, with probability 1−δ, the true mean of the variable is at least rmeanε, whereε=B2log(2/δ)2n

Hoeffding selection operation is operations that can be used to select individuals without entirely calculate the fitness value of them. We can solve the difficult problem of deciding exactly how many samples are

Performance evaluation

The experiments were conducted on a 3.0 GHz Pentium 4 with 1.0 GB of memory running Microsoft Windows XP. All code was compiled using Microsoft Visual C++6.0.

In fact, the proposed algorithm HEA/HGP is not a simply algorithm, but an evolution framework. Every kind of evolutionary algorithm can be inserted in it. In Sections 5.1–5.5, we inserted a traditional EA algorithm to this framework. We use the function “f(x)=min{2/x, sin(4x)+1}” to produce the set of sample points. When we searched for the

Discussions and conclusion

In fact, different evolutionary algorithms have been proposed to dealing with the problem of symbolic regression. For example, linear genetic programming (LGP), Cartesian genetic programming (CGP), gene expression programming (GEP), grammar guided genetic programming (GGGP), Tree adjunct grammar guided genetic programming (TAG3P), etc. But most of them pay more attention to the manner of representation, selection, crossover, and mutation. When the number of samples is very large, it is very

Acknowledgment

We are grateful to the anonymous referees for their invaluable suggestions to improve the paper. This work was supported by the National Natural Science Foundation of China (Grant nos. 60873035, 61073091, 61100009) and by Foundation of Excellent Doctoral Dissertation of Xi'an University of Technology (116-211102).

References (58)

  • R. Kohavi et al.

    Wrappers for feature subset selection

    Artif. Intell.

    (1997)
  • J.D. Anderson

    Computational Fluid Dynamics: The Basics with Applications

    (1995)
  • Arnold, D.V., 2001. Evolution strategies in noisy environments-a survey of existing work, in: Theoretical Aspects of...
  • Audet, C., Dennis, J.J.E., Moore, D.W., Booker, A., and Frank, P.D., 2000. Surrogate-model-based method for constrained...
  • J. Bartelemy et al.

    Approximation concepts for optimum structural design — a review

    Struct. Optim.

    (1993)
  • Birattari, M., Stutzle, T., Paquete, L., and Varrentrapp, K., 2002. A racing algorithm for configuring metaheuristics,...
  • B. Bollobas et al.

    The degree sequence of a scale-free random graph process

    Random Struct. Algorithms

    (2001)
  • P. Bradshaw et al.

    Calculation of compressible turbulent boundary layers on straight-tapered swept wings

    AIAA J.

    (1976)
  • A.J. Brooker et al.

    A rigorous framework for optimization of expensive functions by surrogates

    Struct. Optim.

    (1998)
  • L. Bull

    On model-based evolutionary computation

    Soft Comput.

    (1999)
  • W. Carpenter et al.

    A comparison of polynomial approximations and artificial neural nets as response surfaces

    Struct. Optim.

    (1993)
  • V. Cherkassky et al.

    Learning From Data: Concepts, Theory, and Methods

    (1998)
  • Cramer, N.L. 1985. A representation for the adaptive generation of simple sequential programs. In: J. J. Grefenstette,...
  • J. Dennis et al.

    Managing approximate models in optimization

  • Domingos, P., Hulten, G., 2000. Mining high-speed data streams, in: Proceedings of the Sixth ACM SIGKDD International...
  • Eggermont, J., and Hemert, J.I.V., 2001. Adaptive genetic programming applied to new and existing simple regression...
  • C. Ferreira.

    Gene expression programming: a new adaptive algorithm for solving problems

    Complex Syst.

    (2001)
  • C. Ferreira.

    Gene Expression Programming: Mathematical Modeling by an Artificial Intelligence

    (2002)
  • W.E. Hart et al.

    Optimization with genetic algorithm hybrids that use local search

  • S. Haykin

    Neural Networks: A Comprehensive Foundation

    (1994)
  • Hoai, N.X., McKay, R.I., Essam, D., Chau, R., 2002. Solving the symbolic regression problem with tree-adjunct grammar...
  • W. Hoeffding.

    Probability inequalities for sums of bounded random variables

    J. Am. Stat. Assoc.

    (1963)
  • Hulten, G., Spencer, L., Domingos, P., 2001. Mining time-changing data streams, in: Proceedings of the Seventh ACM...
  • Y. Jin

    A comprehensive survey of fitness approximation in evolutionary computation

    Soft Comput. J.

    (2005)
  • Jin, Y., Olhofer, M., and Sendhoff, B., 2000. On evolutionary optimization with approximate fitness functions, in:...
  • Y. Jin et al.

    A framework for evolutionary optimization with approximate fitness functions

    IEEE Trans. Evol. Comput.

    (2002)
  • Johanson, B., and Poli, R., 1998. GP-music: an interactice genetic programming system for music generation with...
  • Kim, H.S., and Cho., S.B., 2001. An efficient genetic algorithms with less fitness evaluation by clustering. In:...
  • Cited by (10)

    • Root cause analysis for inverters in solar photo-voltaic plants

      2020, Engineering Failure Analysis
      Citation Excerpt :

      Several sets are accumulated in the Base-line training. Besides, we have considered the innovative approach as the “classification of multi-carrier digital modulation signals using NCM clustering based feature-weighting method with good results” [17], and several test with automatic feedback with regression algorithms [18] and genetic algorithms [19]. N: It is the number of records of the set.

    • New approach for developing soft computational prediction models for moment and rotation of boltless steel connections

      2018, Thin-Walled Structures
      Citation Excerpt :

      The designation “linear” denotes to the architecture of the (domineering) programming depiction, and the genetic programming function unduplicated that are self-conscious to only nodes with linear list. In contrast, the highly non-linear solution could be represented by the genetic programs [35]. Major variances to common tree-based GP are the graph-based data stream that consequences from a numerous procedure of indexed parameters (register) insides and the presence of structurally unsuccessful code (introns).

    • Risk-Controlled Selective Prediction for Regression Deep Neural Network Models

      2020, Proceedings of the International Joint Conference on Neural Networks
    View all citing articles on Scopus
    View full text