Skip to main content
SpringerLink
Log in

Bayesian Methods for Efficient Genetic Programming

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

A Bayesian framework for genetic programming (GP) is presented. This is motivated by the observation that genetic programming iteratively searches populations of fitter programs and thus the information gained in the previous generation can be used in the next generation. The Bayesian GP makes use of Bayes theorem to estimate the posterior distribution of programs from their prior distribution and likelihood for the fitness data observed. Offspring programs are then generated by sampling from the posterior distribution by genetic variation operators. We present two GP algorithms derived from the Bayesian GP framework. One is the genetic programming with the adaptive Occam's razor (AOR) designed to evolve parsimonious programs. The other is the genetic programming with incremental data inheritance (IDI) designed to accelerate evolution by active selection of fitness cases. A multiagent learning task is used to demonstrate the effectiveness of the presented methods. In a series of experiments, AOR reduced solution complexity by 20% and IDI doubled evolution speed, both without loss of solution accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. D. Andre, “Automatically defined features: The simultaneous evolution of 2-dimensional feature detectors and an algorithm for using them,” In K. E. Kinnear, Jr. ed, Advances in Genetic Programming, MIT Press: Cambridge, MA, 1994, Chapter 23, 0pp. 477

    Google Scholar 

  2. P. J. Angeline and J. B. Pollack, “Coevolving high-level representations, in Artificial Life III, C. G. Langton ed., Addison-Wesley: Reading, MA, 1993.

    Google Scholar 

  3. P. J. Angeline, “Genetic programming and emergent intelligence,” in K. E. Kinnear, Jr. ed., Advances in Genetic Programming, MIT Press: Cambridge, MA, 1994. Chapter 4, pp. 75–98.

    Google Scholar 

  4. P. J. Angeline, “Subtree crossover: Building block engine or macromutation?” in J. R. Koza, et al. eds., The Second Genetic Programming Conf. GP-97, Morgan Kaufmann, San Francisco, CA, 1997, pp. 9–17.

    Google Scholar 

  5. T. Back, Evolutionary Algorithms in Theory and Practice, Oxford, UK: Oxford University Press, 1996.

    Google Scholar 

  6. W. Banzhaf, F. Francone, and P. Nordin, “The effect of extensive use of the mutation operator on generalization in genetic programming using sparse data sets,” In Proc. 4th Int. Conf. on Parallel Problem Solving from Nature PPSN-96, W. Ebeling, I. Rechenberg, H.-P. Schwefel, H. M. Voigt eds., Springer: Berlin, 1996, pp. 300–309.

    Google Scholar 

  7. W. Banzhaf, P. Nordin, and F. Francone, “On some emergent properties of variable size evolutionary algorithms,” in ICGA-97 Workshop on Evolutionary Computation with Variable-Size Representation, 1997, http:rrwww.ai.mit.edurpeoplerunamayricga-ws.html.

  8. W. Banzhaf, P. Nordin, R. Keller, and F. Francone, Genetic Programming: An Introduction, Morgan Kaufmann: San Francisco, CA.

  9. C. M. Bishop, Neural Networks for Pattern Recognition. Oxford University Press: Oxford, UK, 1995.

    Google Scholar 

  10. T. Blickle, “Evolving compact solutions in genetic programming: A case study,” in H.-M. Voigt et al. eds., Parallel Problem Solving from Nature IV, Springer-Verlag: Berlin, 1996, pp. 564–573.

    Google Scholar 

  11. K. Chellapilla, “Evolutionary programming with tree mutations: Evolving computer programs without crossover,” in J. R. Koza, et al. eds., The Second Genetic Programming Conf. GP-97, Morgan Kaufmann: San Francisco, CA, 1997, pp. 431–438.

    Google Scholar 

  12. T. M. Cover and J. A. Thomas, Elements of Information Theory, Wiley: New York, 1991.

    Google Scholar 

  13. C. Gathercole and P. Ross, “Dynamic training subset selection for supervised learning in genetic programming,” in Parallel Problem Solving from Nature III, Y. Davidor, et al. Eds., Springer-Verlag: Berlin, 1994, pp. 312–321.

    Google Scholar 

  14. C. Gathercole and P. Ross, “Small populations over many generations can beat large populations over few generations in genetic programming,” in J. R. Koza, et al. eds., The Second Genetic Programming Conf. GP-97, Morgan Kaufmann: San Francisco, CA, 1997, pp. 111–118.

    Google Scholar 

  15. A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin, Bayesian Data Analysis, Chapman & Hall: London, 1995.

    Google Scholar 

  16. T. Haynes, “Perturbing the representation, decoding, and evaluation of chromosomes,” in J. R. Koza, et al. eds., The Third Genetic Programming Conf. GP-98, Morgan Kaufmann: San Francisco, CA, 1998, pp. 122–127.

    Google Scholar 

  17. D. Hillis, “Co-evolving parasites improves simulated evolution as an optimization procedure,” in Artificial Life II, C. Langton, et al. Eds., Addison-Wesley: Reading, MA, 1992, pp. 313–324.

    Google Scholar 

  18. H. Iba, H. de Garis, and T. Sato, “Genetic programming using a minimum description length principle,” in K. E. Kinnear, Jr. ed., Advances in Genetic Programming, MIT Press: Cambridge, MA, 1994, Chapter 12, pp. 265–284.

    Google Scholar 

  19. K. E. Kinnear, Jr. “Generality and difficulty in genetic programming: Evolving a sort,” in Proc. of 5th Int. Conf. on Genetic Algorithms ICGA-93, S. Forrest ed., Morgan Kaufmann: San Francisco, CA, 1993, pp. 287–294.

    Google Scholar 

  20. J. R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, MIT Press: Cambridge, MA, 1992.

    Google Scholar 

  21. J. R. Koza, “Scalable learning in genetic programming using automatic function definition,” in K. E. Kinnear, Jr. ed., Advances in Genetic Programming, MIT Press: Cambridge, MA, 1994, Chapter 5, pp. 99–117.

    Google Scholar 

  22. W. B. Langdon and R. Poli, “Fitness causes bloat: Mutation,” in W. Banzhaf, R. Poli, M. Schoenauer, and T. Fogarty eds., The First European Workshop on Genetic Programming EuroGP'98, Paris, LNCS 1391, Springer-Verlag: Berlin, 1998, pp. 37–48.

    Google Scholar 

  23. W. B. Langdon, “Size fair and homologous tree crossovers,” Genetic Programming and Evolvable Machines, vol. 1(1), pp. 95–119, 2000.

    Google Scholar 

  24. S. Luke and L. Spector, “A comparison of crossover and mutation in genetic programming,” in J. R. Koza, et al. eds., The Second Genetic Programming Conf. GP-97, Morgan Kaufmann: San Francisco, CA, 1997, pp. 240–248.

    Google Scholar 

  25. H. Muhlenbein and D. Schlierkamp-Voosen, “The science of breeding and its application to the breeder genetic algorithm,” Evolutionary Computation, vol. 1(4) pp. 335–360, 1994.

    Google Scholar 

  26. R. M. Neal, “Probabilistic inference using Markov chain Monte Carlo methods,” Technical Report CRG-TR–93-1, Dept. of Computer Science, University of Toronto, 1993.

  27. P. Nordin, F. Francone, and W. Banzhaf, “Explicitly defined introns and destructive crossover in genetic programming,” in P. J. Angeline and K. E. Kinnear, Jr. eds., Advances in Genetic Programming 2, MIT Press: Cambridge, MA, 1996, pp. 111–134.

    Google Scholar 

  28. U.-M. O'Reilly, “Investigating the generality of automatically defined functions,” in The First Genetic Programming Conf. GP-96, J. R. Koza eds., Morgan Kaufmann: San Francisco, CA, 1996, pp. 351–356.

    Google Scholar 

  29. R. Poli, and W. B. Langdon, “On the search properties of different crossover operators in genetic programming,” The Third Genetic Programming Conf. GP-98, Morgan Kaufmann: San Francisco, CA, 1998, pp. 293–301.

    Google Scholar 

  30. J. Rissanen, “Stochastic complexity and modeling,” Ann. Statist. vol. 14, pp. 1080–1100, 1986.

    Google Scholar 

  31. J. P. Rosca, “Analysis of complexity drift in genetic programming,” in The Second Genetic Programming Conf. GP-97, J. R. Koza eds., Morgan Kaufmann: San Francisco, CA, 1997, pp. 286–294.

    Google Scholar 

  32. J. P. Rosca and D. H. Ballard, “Discovery of subroutines in genetic programming,” in P. J. Angeline and K. E. Kinnear, Jr. ed., Advances in Genetic Programming 2, MIT Press: Cambridge, MA, 1996, pp. 177–201.

    Google Scholar 

  33. M. Schoenauer, M. Sebag, F. Jouve, B. Lamy, and H. Maitournam, “Evolutionary identification of macro-mechanical models,” in Advances in Genetic Programming 2, P. J. Angeline and K. E. Kinnear, Jr. eds., MIT Press: Cambridge, MA, 1996, pp. 467–488.

    Google Scholar 

  34. E. V. Siegel, “Competitively evolving decision trees against fixed training cases for natural language processing,” in Advances in Genetic Programming, K. E. Kinnear, Jr. ed., MIT Press: Cambridge, MA, 1994, Chapter 19, pp. 409–423.

    Google Scholar 

  35. T. Soule, J. A. Foster, and J. Dickinson, “Code growth in genetic programming,” in The First Genetic Programming Conf. GP-96, J. P. Koza, et al. eds., Morgan Kaufmann: San Francisco, CA, 1996, pp. 215–223.

    Google Scholar 

  36. T. Soule and J. A. Foster, “Effects of code growth and parsimony pressure on populations in genetic programming,” Evolutionary Computation vol. 6(4), pp. 293–309, 1998.

    Google Scholar 

  37. L. Spector, “Simultaneous evolution of programs and their control structures,” in Advances in Genetic Programming 2, P. J. Angeline and K. E. Kinnear, Jr. eds., MIT Press: Cambridge, MA, 1996, Chapter 7, pp. 137–154.

    Google Scholar 

  38. A. Teller and D. Andre, “Automatically choosing the number of fitness cases: The rational allocation of trials,” in The Second Genetic Programming Conf. GP-97, J. R. Koza, et al. eds., Morgan Kaufmann: San Francisco, CA, 1997, pp. 321–328.

    Google Scholar 

  39. A. S. Wu and R. K. Lindsay, “Empirical studies of the genetic algorithm with noncoding segments,” Evolutionary Computation, vol. 3(2), pp. 121–147, 1996.

    Google Scholar 

  40. B.-T. Zhang, “Accelerated learning by active example selection,” Int. J. Neural Syst. vol. 5(4) pp. 67–75, 1994.

    Google Scholar 

  41. B.-T. Zhang, “A Bayesian framework for evolutionary computation,” in The 1999 Congress on Evolutionary Computation CEC99, Special Session on Theory and Foundations of Evolutionary Computation, IEEE Press, 1999, pp. 722–727.

  42. B.-T. Zhang and J.-G. Joung, “Genetic programming with incremental data inheritance,” In The 1999 Genetic and Evolutionary Computation Conf. GECCO-99, W. Banzhaf et al. eds., Morgan Kaufmann: San Francisco, CA, 1999, pp. 1217–1224.

    Google Scholar 

  43. B.-T. Zhang and H. Muhlenbein, “Genetic programming of minimal neural nets using Occam's razor,” in Proc. of 5th Int. Conf. on Genetic Algorithms ICGA-93, S. Forrest ed., Morgan Kaufmann: San Francisco, CA, 1993, pp. 342–349.

    Google Scholar 

  44. B.-T. Zhang and H. Muhlenbein, “Balancing accuracy and parsimony in genetic programming,” Evolutionary Computation, vol. 3(1) pp. 17–38, 1995.

    Google Scholar 

  45. B.-T. Zhang, P. Ohm, and H. Muhlenbein, “Evolutionary induction of sparse neural trees,” Evolutionary Computation, Vol. 5(1) pp. 213–236, 1997.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Artificial Intelligence Lab (SCAI), School of Computer Science and Engineering, Seoul National University, Seoul, 151-742, Korea

    Byong-Tak Zhang

Authors
  1. Byong-Tak Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, BT. Bayesian Methods for Efficient Genetic Programming. Genetic Programming and Evolvable Machines 1, 217–242 (2000). https://doi.org/10.1023/A:1010010230007

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1010010230007

Search

Navigation