Skip to main content
SpringerLink
Log in
Book cover

International Conference on Parallel Problem Solving from Nature

PPSN 2020: Parallel Problem Solving from Nature – PPSN XVI pp 63–78Cite as

Image Feature Learning with Genetic Programming

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12270))

Abstract

Learning features from raw data is an important topic in machine learning. This paper presents Genetic Program Feature Learner (GPFL), a novel generative GP feature learner for 2D images. GPFL executes multiple GP runs, each run generates a model that focuses on a particular high-level feature of the training images. Then, it combines the models generated by each run into a function that reconstructs the observed images. As a sanity check, we evaluated GPFL on the popular MNIST dataset of handwritten digits, and compared it with the convolutional neural network LeNet5. Our evaluation results show that when considering smaller training sets, GPFL achieves comparable/slightly-better classification accuracy than LeNet5. However, GPFL drastically outperforms LeNet5 when considering noisy images as test sets.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    We presented a preliminary version of this work in a poster paper  [40].

  2. 2.

    The cost of computing the linear scaling coefficients is \(\mathcal {O}(\mid \mathbb {\hat{Y}}\mid \cdot \mid \mathcal {P} \mid )\).

  3. 3.

    When comparing the classification accuracy of GPFL and LeNet5, we computed the p-values with the non-parametric pairwise Wilcoxon rank-sum test  [15].

References

  1. Albukhanajer, W.A., Briffa, J.A., Jin, Y.: Evolutionary multiobjective image feature extraction in the presence of noise. IEEE Trans. Cybern. 45(9), 1757–1768 (2015). https://doi.org/10.1109/TCYB.2014.2360074

    Article  Google Scholar 

  2. Alvear-Sandoval, R.F., Sancho-Gómez, J.L., Figueiras-Vidal, A.R.: On improving CNNs performance: the case of MNIST. Inf. Fusion 52, 106–109 (2019)

    Article  Google Scholar 

  3. Baldominos, A., Saez, Y., Isasi, P.: Evolutionary convolutional neural networks: an application to handwriting recognition. Neurocomputing 283, 38–52 (2018). https://doi.org/10.1016/j.neucom.2017.12.049

    Article  Google Scholar 

  4. Baldominos, A., Saez, Y., Isasi, P.: Model selection in committees of evolved convolutional neural networks using genetic algorithms. In: Intelligent Data Engineering and Automated Learning, IDEAL 2018, pp. 364–373 (2018). https://doi.org/10.1007/978-3-030-03493-1_39

  5. Baldominos, A., Saez, Y., Isasi, P.: Hybridizing evolutionary computation and deep neural networks: an approach to handwriting recognition using committees and transfer learning. Complexity (2019). https://doi.org/10.1155/2019/2952304

    Article  Google Scholar 

  6. Baldominos, A., Saez, Y., Isasi, P.: A survey of handwritten character recognition with MNIST and EMNIST. Appl. Sci. 9(15), 3169 (2019)

    Article  Google Scholar 

  7. Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_32

    Chapter  Google Scholar 

  8. Bochinski, E., Senst, T., Sikora, T.: Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. In: Proceedings International Conference on Image Processing, ICIP 2017, pp. 3924–3928 (2017). https://doi.org/10.1109/ICIP.2017.8297018

  9. Brameier, M.F., Banzhaf, W.: Linear Genetic Programming. Springer, Boston (2007). https://doi.org/10.1007/978-0-387-31030-5_1

    Book  MATH  Google Scholar 

  10. Butterworth, J., Savani, R., Tuyls, K.: Evolving indoor navigational strategies using gated recurrent units in NEAT. In: Proceedings of the Companion of Genetic and Evolutionary Computation Conference, GECCO 2019, pp. 111–112 (2019). https://doi.org/10.1145/3319619.3321995

  11. Davison, J.: DEvol: automated deep neural network design via genetic programming (2020). https://github.com/joeddav/devol

  12. Fei-Fei, L., Fergus, R., Perona, P.: One-shot learning of object categories. IEEE Trans. Pattern Anal. Mach. Intell. 28(4), 594–611 (2006)

    Article  Google Scholar 

  13. George, D., et al.: A generative vision model that trains with high data efficiency and breaks text-based captchas. Science 358(6368), 2612 (2017)

    Article  Google Scholar 

  14. Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in Neural Information Processing Systems, pp. 1135–1143 (2015)

    Google Scholar 

  15. Haynes, W.: Wilcoxon rank sum test. In: Encyclopedia of Systems Biology, pp. 2354–2355 (2013)

    Google Scholar 

  16. Impedovo, S., Mangini, F.: A novel technique for handwritten digit classification using genetic clustering. In: Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2012, pp. 236–240 (2012). https://doi.org/10.1109/ICFHR.2012.167

  17. Keijzer, M.: Improving symbolic regression with interval arithmetic and linear scaling. In: Ryan, C., Soule, T., Keijzer, M., Tsang, E., Poli, R., Costa, E. (eds.) EuroGP 2003. LNCS, vol. 2610, pp. 70–82. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36599-0_7

    Chapter  Google Scholar 

  18. Kramer, M.A.: Nonlinear principal component analysis using autoassociative neural networks. AIChE J. 37(2), 233–243 (1991)

    Article  Google Scholar 

  19. LeCun, Y.: Lenet-5, convolutional neural networks (2020). http://yann.lecun.com/exdb/lenet

  20. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  21. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition, vol. 86, pp. 2278–2324. IEEE (1998)

    Google Scholar 

  22. Legge, G.E., Foley, J.M.: Contrast masking in human vision. Josa 70(12), 1458–1471 (1980)

    Article  Google Scholar 

  23. Lensen, A., Al-Sahaf, H., Zhang, M., Xue, B.: Genetic programming for region detection, feature extraction, feature construction and classification in image data. In: Heywood, M.I., McDermott, J., Castelli, M., Costa, E., Sim, K. (eds.) EuroGP 2016. LNCS, vol. 9594, pp. 51–67. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30668-1_4

    Chapter  Google Scholar 

  24. Lensen, A., Xue, B., Zhang, M.: Can genetic programming do manifold learning too? In: Sekanina, L., Hu, T., Lourenço, N., Richter, H., García-Sánchez, P. (eds.) EuroGP 2019. LNCS, vol. 11451, pp. 114–130. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16670-0_8

    Chapter  Google Scholar 

  25. Lensen, A., Zhang, M., Xue, B.: Multi-objective genetic programming for manifold learning: balancing quality and dimensionality. Genet. Program Evolvable Mach. 21(3), 399–431 (2020). https://doi.org/10.1007/s10710-020-09375-4

    Article  Google Scholar 

  26. Liu, L., Shao, L., Li, X.: Evolutionary compact embedding for large-scale image classification. Inf. Sci. 316, 567–581 (2015). https://doi.org/10.1016/j.ins.2014.06.030

    Article  Google Scholar 

  27. López, U., Trujillo, L., Martinez, Y., Legrand, P., Naredo, E., Silva, S.: RANSAC-GP: dealing with outliers in symbolic regression with genetic programming. In: Proceedings of the European Conference on Genetic Programming, EuroGP 2017, pp. 114–130 (2017)

    Google Scholar 

  28. Makhzani, A., Frey, B.J.: Winner-take-all autoencoders. In: Advances in Neural Information Processing Systems, pp. 2791–2799 (2015)

    Google Scholar 

  29. McDermott, J.: Why is auto-encoding difficult for genetic programming? In: Sekanina, L., Hu, T., Lourenço, N., Richter, H., García-Sánchez, P. (eds.) EuroGP 2019. LNCS, vol. 11451, pp. 131–145. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16670-0_9

    Chapter  Google Scholar 

  30. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.: Reading digits in natural images with unsupervised feature learning. Google technical report (2011)

    Google Scholar 

  31. Neumann, L., Matas, J.: Real-time scene text localization and recognition. In: Proceedings of Conference on Computer Vision and Pattern Recognition, CVPR 2012, pp. 3538–3545 (2012)

    Google Scholar 

  32. Oliva, A., Torralba, A.: Building the gist of a scene: the role of global image features in recognition. Prog. Brain Res. 155, 23–36 (2006)

    Article  Google Scholar 

  33. Orzechowski, P., La Cava, W., Moore, J.H.: Where are we now?: a large benchmark study of recent symbolic regression methods. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, pp. 1183–1190 (2018). https://doi.org/10.1145/3205455.3205539

  34. Papavasileiou, E., Jansen, B.: An investigation of topological choices in FS-NEAT and FD-NEAT on XOR-based problems of increased complexity. In: Proceedings of the Companion of Genetic and Evolutionary Computation Conference, GECCO 2017, pp. 1431–1434 (2017). https://doi.org/10.1145/3067695.3082497

  35. Peng, Y., Chen, G., Singh, H., Zhang, M.: NEAT for large-scale reinforcement learning through evolutionary feature learning and policy gradient search. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, pp. 490–497 (2018). https://doi.org/10.1145/3205455.3205536

  36. Perez, C.B., Olague, G.: Genetic programming as strategy for learning image descriptor operators. Intell. Data Anal. 17(4), 561–583 (2013). https://doi.org/10.3233/IDA-130594

    Article  Google Scholar 

  37. Rodriguez-Coayahuitl, L., Morales-Reyes, A., Escalante, H.J.: Structurally layered representation learning: towards deep learning through genetic programming. In: Castelli, M., Sekanina, L., Zhang, M., Cagnoni, S., García-Sánchez, P. (eds.) EuroGP 2018. LNCS, vol. 10781, pp. 271–288. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77553-1_17

    Chapter  Google Scholar 

  38. Rodriguez-Coayahuitl, L., Morales-Reyes, A., Escalante, H.J.: Evolving autoencoding structures through genetic programming. Genet. Program Evolvable Mach. 20(3), 413–440 (2019). https://doi.org/10.1007/s10710-019-09354-4

    Article  Google Scholar 

  39. Ruberto, S., Terragni, V., Moore, J.H.: GPFL replication package. experimental data of GPFL and source code of Lenet5, April 2020. https://doi.org/10.5281/zenodo.3899891

  40. Ruberto, S., Terragni, V., Moore, J.H.: Image feature learning with a genetic programming autoencoder. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2020 (2020)

    Google Scholar 

  41. Ruberto, S., Terragni, V., Moore, J.H.: SGP-DT: semantic genetic programming based on dynamic targets. In: Hu, T., Lourenço, N., Medvet, E., Divina, F. (eds.) EuroGP 2020. LNCS, vol. 12101, pp. 167–183. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-44094-7_11

    Chapter  Google Scholar 

  42. Ruberto, S., Terragni, V., Moore, J.H.: SGP-DT: towards effective symbolic regression with a semantic GP approach based on dynamic targets. In: Proceedings of the Genetic and Evolutionary Computation Conference (Hot Off the Press track), GECCO 2020 (2020)

    Google Scholar 

  43. Ruberto, S., Vanneschi, L., Castelli, M.: Genetic programming with semantic equivalence classes. Swarm Evol. Comput. 44, 453–469 (2019). https://doi.org/10.1016/j.swevo.2018.06.001

    Article  Google Scholar 

  44. Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., Madry, A.: Robustness may be at odds with accuracy (2018)

    Google Scholar 

  45. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the International Conference on Machine Learning, ICML 2008, pp. 1096–1103 (2008)

    Google Scholar 

  46. Wang, J., Zhang, Z., Zha, H.: Adaptive manifold learning. In: Advances in Neural Information Processing Systems, NIPS 2005 (2005)

    Google Scholar 

  47. Yadav, C., Bottou, L.: Cold case: the lost MNIST digits. In: Advances in Neural Information Processing Systems, NIPS 2019, pp. 13443–13452 (2019)

    Google Scholar 

  48. Yann LeCun, C.C., Burges, C.: MNIST handwritten digit database (2020)

    Google Scholar 

  49. Zheng, A., Casari, A.: Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists. O’Reilly Media Inc., Sebastopol (2018)

    Google Scholar 

  50. Zhou, H., Yuan, Y., Shi, C.: Object tracking using sift features and mean shift. Comput. Vis. Image Underst. 113(3), 345–352 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, USA

    Stefano Ruberto & Jason H. Moore

  2. Faculty of Informatics, Università della Svizzera Italiana USI, Lugano, Switzerland

    Valerio Terragni

Authors
  1. Stefano Ruberto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Valerio Terragni
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jason H. Moore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefano Ruberto .

Editor information

Editors and Affiliations

  1. Leiden University, Leiden, The Netherlands

    Thomas Bäck

  2. Leiden University, Leiden, The Netherlands

    Mike Preuss

  3. Leiden University, Leiden, The Netherlands

    André Deutz

  4. Sorbonne University, Paris, France

    Hao Wang

  5. Sorbonne University, Paris, France

    Carola Doerr

  6. Leiden University, Leiden, The Netherlands

    Michael Emmerich

  7. University of Münster, Münster, Germany

    Heike Trautmann

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ruberto, S., Terragni, V., Moore, J.H. (2020). Image Feature Learning with Genetic Programming. In: Bäck, T., et al. Parallel Problem Solving from Nature – PPSN XVI. PPSN 2020. Lecture Notes in Computer Science(), vol 12270. Springer, Cham. https://doi.org/10.1007/978-3-030-58115-2_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58115-2_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58114-5

  • Online ISBN: 978-3-030-58115-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation