Hoeffding bound based evolutionary algorithm for symbolic regression
Introduction
Evolutionary algorithms have been applied successfully to a variety of complicated real-world applications. One main difficulty in these applications is that EAs usually need a large number of fitness evaluations before obtaining a satisfying result. However, when the evaluation of the fitness is computationally very expensive or an explicit fitness function does not exist, we have to construct an approximate model to estimate the fitness.
Fitness approximation has become an active area in evolutionary computation with many varying approaches and results (Jin, 2005). There are several reasons for utilizing fitness approximation through modeling. The first and most common reason is to reduce the computational complexity of expensive fitness evaluations. Many applications of evolutionary algorithms are in high-complexity or intractable domains, where the fitness calculation can be prohibitively time consuming (Ong et al., 2003, Jin et al., 2002, Regis and Shoemaker, 2004, Regis and Shoemaker, 2005) and the exact fitness objective is unnecessary for evolutionary progress. At the same time, approximation can be used advantageously in other problems, such as handling noisy fitness functions, smooth multimodal landscapes, and defining a continuous fitness in domains that lack an explicit fitness (Regis and Shoemaker, 2004, Regis and Shoemaker, 2005, Arnold,, Sano and Kita,, Audet et al.,).
Traditionally, there are several approaches in fitness approximation. Problem approximation tries to replace the original problem by another problem which is approximately the same to the original problem but easier to solve (Anderson., 1995, Bradshaw et al., 1976). In functional approximation, an alternate and explicit expression is constructed for the objective function. Several methods have been proposed for constructing approximate models. In (Johanson and Poli, 1998), a neural network has been used to model the behavior of the human to reduce human fatigue and evaluate the fitness. In Hart and Belew (1996), Liang et al. (2000), approximate models were used to smooth the rough fitness functions and prevent the evolutionary algorithm from stagnating in a local optimum. In Myers and Montgomery (1995), Sacks et al. (1989), a global polynomial approximation and a local Gaussian process were combined together to estimate the parameters used. Artificial neural networks, including multilayer perceptions and radial basis function networks have also been used to build approximate models for evolutionary optimization (Bartelemy and Haftka, 1993, Carpenter and Barthelemy, 1993, Shyy et al., 1999, Simpson et al., 1998). Evolutionary approximation is a type of approximation for evolutionary algorithms. Fitness inheritance is a popular evolutionary approximation method, in which fitness evaluations can be spared by estimating the fitness value of the offspring individuals from the value of their parents. In Kim and Cho (2001), the individuals are clustered into several groups and only the individual representing its cluster will be evaluated using the fitness function. The fitness value of other individuals in the same cluster will be estimated from the representative individual based a distance measure.
However, to get an accurate approximate model is a difficult task, the models achieved often have large approximation errors and always introduces a false optima. So, the evolutionary algorithm with approximate models for fitness evaluation always converges to an incorrectly solution (Kim and Cho, 2001).
How to guarantee that the solution discovered is optimal or near-optimal is a key problem we have to face. Most methods with approximate model for fitness evaluation assume that the approximate model is correct and the evolutionary algorithm will converge to a global or near-optimal solution (Redmond and Parker, 1996, Pierret, 1999). These methods can only succeed when the approximate model is globally correct. To guarantee to converge to the global solution, many frameworks for managing approximate models were proposed (More, 1983, Schramm and Zowe, 1992, Brooker et al., 1998). In Dennis and Torczon (1997), a framework was represented based on the classical trust-region methods, which ensures that the search process converges to a reasonable solution of original problem. In Ratle (1998), a heuristic convergence criterion is used to determine the updating time of the approximate model. In Ratle (1999), the original fitness function is used in every k generations to update the approximate model. However, if there is a large discrepancy between the approximate model and the original fitness function, the evolution process may become very unstable (Jin et al., 2000). In Bull (1999), an evolutionary algorithm with a neural network trained with some initial samples to approximate the model is proposed. During the evolutionary process, the fittest individual in the population is evaluated on the original fitness function once in every 50 generations. The individual then replaces the one with the lowest fitness in the training set and the neural network is retrained. However, when the complexity of the original fitness landscape is high, the neural network model will mislead the evolutionary algorithm to a default way. To improve the convergence of the evolutionary algorithm, a concept of evolution control, in which the evolution proceeds based on fitness evaluations using not only the approximate fitness model, but also the original fitness function, was proposed in Jin et al. (2002).
In this paper, we propose a new framework for evolutionary algorithm named Hoeffding evolutionary algorithm and it can be used with other traditional evolutionary algorithm together to deal with a kind of problems in which the evaluation of the fitness is computationally very expensive. The algorithm can exactly decide how many samples are necessary for choosing i best individuals from a population in evolutionary algorithms without calculating the fitness completely. The major advantage of the proposed algorithm is that it can guarantee that the solution discovered has performance matching what would be discovered with a traditional GP selection operator, with a determinate probability; and with the similar accuracy, the HEA algorithm can find the solution more efficiently than tradition EA.
The rest of this paper is organized as follows. We first review the related work on the Hoeffding bound, symbolic regression and genetic programming. Next, different kinds of fitness functions were introduced and two iterative fitness function models were built. Third, we represent the Hoeffding-EA algorithm (Hoeffding evolutionary framework with evolutionary algorithm) for symbolic regression and give some theorem analysis. Then we describe several experiments in which the parameters of the algorithm are examined and the algorithm is effectively applied to find a regression model. Finally, we conclude this paper by highlighting the key contributions of this work.
Section snippets
Hoeffding bound and relevant algorithms
The distribution function for the sum of independent random variables, x1+x2++xi++xn, when some information about the distribution of the xi is available, is very important for the modern theory of probability. Much work has been carried out on the asymptotic form of the distribution of such sums when the number of component random variables is large or when the component variables have identical distributions. If we test a given model on N samples, the average error of them can be expressed
Iterative fitness function
In general, an optimization problem requires a fitness function which is given by systems of arbitrary complexity. Different fitness functions in evolutionary optimization were mainly categorized into four classes: noise, time-varying, robustness and iteration.
The evaluation of noise fitness function is subject to noise which may come from many different sources such as sensory measurement errors or randomized simulations. The two common noises are Gaussian and Cauhy (Stroud, 2001). The
Hoeffding selection operation
Consider a real-valued random variable r whose range is B. Suppose we have made n independent observations of this variable, and computed their mean rmean. The Hoeffding bound states that, with probability 1−δ, the true mean of the variable is at least rmean−ε, where
Hoeffding selection operation is operations that can be used to select individuals without entirely calculate the fitness value of them. We can solve the difficult problem of deciding exactly how many samples are
Performance evaluation
The experiments were conducted on a 3.0 GHz Pentium 4 with 1.0 GB of memory running Microsoft Windows XP. All code was compiled using Microsoft Visual C++6.0.
In fact, the proposed algorithm HEA/HGP is not a simply algorithm, but an evolution framework. Every kind of evolutionary algorithm can be inserted in it. In Sections 5.1–5.5, we inserted a traditional EA algorithm to this framework. We use the function “f(x)=min{2/x, sin(4x)+1}” to produce the set of sample points. When we searched for the
Discussions and conclusion
In fact, different evolutionary algorithms have been proposed to dealing with the problem of symbolic regression. For example, linear genetic programming (LGP), Cartesian genetic programming (CGP), gene expression programming (GEP), grammar guided genetic programming (GGGP), Tree adjunct grammar guided genetic programming (TAG3P), etc. But most of them pay more attention to the manner of representation, selection, crossover, and mutation. When the number of samples is very large, it is very
Acknowledgment
We are grateful to the anonymous referees for their invaluable suggestions to improve the paper. This work was supported by the National Natural Science Foundation of China (Grant nos. 60873035, 61073091, 61100009) and by Foundation of Excellent Doctoral Dissertation of Xi'an University of Technology (116-211102).
References (58)
- et al.
Wrappers for feature subset selection
Artif. Intell.
(1997) Computational Fluid Dynamics: The Basics with Applications
(1995)- Arnold, D.V., 2001. Evolution strategies in noisy environments-a survey of existing work, in: Theoretical Aspects of...
- Audet, C., Dennis, J.J.E., Moore, D.W., Booker, A., and Frank, P.D., 2000. Surrogate-model-based method for constrained...
- et al.
Approximation concepts for optimum structural design — a review
Struct. Optim.
(1993) - Birattari, M., Stutzle, T., Paquete, L., and Varrentrapp, K., 2002. A racing algorithm for configuring metaheuristics,...
- et al.
The degree sequence of a scale-free random graph process
Random Struct. Algorithms
(2001) - et al.
Calculation of compressible turbulent boundary layers on straight-tapered swept wings
AIAA J.
(1976) - et al.
A rigorous framework for optimization of expensive functions by surrogates
Struct. Optim.
(1998) On model-based evolutionary computation
Soft Comput.
(1999)
A comparison of polynomial approximations and artificial neural nets as response surfaces
Struct. Optim.
Learning From Data: Concepts, Theory, and Methods
Managing approximate models in optimization
Gene expression programming: a new adaptive algorithm for solving problems
Complex Syst.
Gene Expression Programming: Mathematical Modeling by an Artificial Intelligence
Optimization with genetic algorithm hybrids that use local search
Neural Networks: A Comprehensive Foundation
Probability inequalities for sums of bounded random variables
J. Am. Stat. Assoc.
A comprehensive survey of fitness approximation in evolutionary computation
Soft Comput. J.
A framework for evolutionary optimization with approximate fitness functions
IEEE Trans. Evol. Comput.
Cited by (10)
LBE corrosion fatigue life prediction of T91 steel and 316 SS using machine learning method assisted by symbol regression
2023, International Journal of FatigueRoot cause analysis for inverters in solar photo-voltaic plants
2020, Engineering Failure AnalysisCitation Excerpt :Several sets are accumulated in the Base-line training. Besides, we have considered the innovative approach as the “classification of multi-carrier digital modulation signals using NCM clustering based feature-weighting method with good results” [17], and several test with automatic feedback with regression algorithms [18] and genetic algorithms [19]. N: It is the number of records of the set.
New approach for developing soft computational prediction models for moment and rotation of boltless steel connections
2018, Thin-Walled StructuresCitation Excerpt :The designation “linear” denotes to the architecture of the (domineering) programming depiction, and the genetic programming function unduplicated that are self-conscious to only nodes with linear list. In contrast, the highly non-linear solution could be represented by the genetic programs [35]. Major variances to common tree-based GP are the graph-based data stream that consequences from a numerous procedure of indexed parameters (register) insides and the presence of structurally unsuccessful code (introns).
Hierarchical clustering for multiple nominal data streams with evolving behaviour
2022, Complex and Intelligent SystemsRisk-Controlled Selective Prediction for Regression Deep Neural Network Models
2020, Proceedings of the International Joint Conference on Neural NetworksA Discrete Cosine Transform Based Evolutionary Algorithm and Its Application for Symbolic Regression
2019, Advances in Intelligent Systems and Computing