Skip to main content

Advertisement

Log in

A grammar-based GP approach applied to the design of deep neural networks

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

Deep Learning has been very successful in automating the feature engineering process, widely applied for various tasks, such as speech recognition, classification, segmentation of images, time-series forecasting, among others. Deep neural networks (DNNs) incorporate the power to learn patterns through data, following an end-to-end fashion and expand the applicability in real world problems, since less pre-processing is necessary. With the fast growth in both scale and complexity, a new challenge has emerged regarding the design and configuration of DNNs. In this work, we present a study on applying an evolutionary grammar-based genetic programming algorithm (GP) as a unified approach to the design of DNNs. Evolutionary approaches have been growing in popularity for this subject as Neuroevolution is studied more. We validate our approach in three different applications: the design of Convolutional Neural Networks for image classification, Graph Neural Networks for text classification, and U-Nets for image segmentation. The results show that evolutionary grammar-based GP can efficiently generate different DNN architectures, adapted to each problem, employing choices that differ from what is usually seen in networks designed by hand. This approach has shown a lot of promise regarding the design of architectures, reaching competitive results with their counterparts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. https://www.cs.umb.edu/smimarog/textmining/datasets/

  2. ftp://medir.ohsu.edu/pub/ohsumed

  3. http://www.cs.cornell.edu/people/pabo/movie-review-data/

  4. For this task, we use the pre-trained model paraphrase-MiniLM-L12-v2 available at: https://www.sbert.net/docs/pretrained_models.html

  5. https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/

  6. http://brainiac2.mit.edu/isbi_challenge/home

References

  1. K.O. Stanley, J. Clune, J. Lehman, R. Miikkulainen, Designing neural networks through neuroevolution. Nat. Mach. Intell. 1, 24–35 (2019)

    Article  Google Scholar 

  2. T. Elsken, J.-H. Metzen, F. Hutter, Neural architecture search: a survey. J. Mach. Learn. Res. 20, 1–21 (2019)

    MathSciNet  MATH  Google Scholar 

  3. R. Miikkulainen, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, H. Shahrzad, A. Navruzyan, N. Duffy, Evolving deep neural networks, in Artificial Intelligence in the Age of Neural Networks and Brain Computing, ed. by R. Kozma, C. Alippi, Y. Choe, F. Morabito (Elsevier, Netherlands, 2019), pp. 293–312

  4. E. Galván, P. Mooney, Neuroevolution in deep neural networks: current trends and future challenges (IEEE Transactions on Artificial Intelligence, Manhattan, 2021)

    Book  Google Scholar 

  5. G. Sainath, S. Vignesh, S. Siddarth, G. Suganya, Application of neuroevolution in autonomous cars. in International Virtual Conference on Industry 4.0. (Springer, Singapore, 2021), pp. 301–311

  6. F. Assunçao, N. Lourenço, P. Machado, B. Ribeiro, Towards the evolution of multi-layered neural networks: a dynamic structured grammatical evolution approach, in Proceedings of the Genetic and Evolutionary Computation Conference. (ACM, 2017), pp. 393–400

  7. R.H.R. Lima, A.T.R. Pozo, A study on auto-configuration of multi-objective particle swarm optimization algorithm, in Proceedings of the 2017 IEEE Congress on Evolutionary Computation (CEC’17), ed. by J.A. Lozano, C. Coello (IEEE, 2017), pp. 718–725

  8. R.H.R. Lima, A.T.R. Pozo, Evolving convolutional neural networks through grammatical evolution. in Proceedings of the 2019 Genetic and Evolutionary Computation Conference (GECCO’19). (ACM, 2019), pp. 179–180

  9. C. Ryan, J.J. Collins, M. O’Neill, Grammatical evolution: evolving programs for an arbitrary language. in Proceedings of the European Conference on Genetic Programming (EuroGP’98). (Springer, 1998), pp. 83–96

  10. R.H.R. Lima, A.T.R. Pozo, A. Mendiburu, R. Santana, Automatic design of deep neural networks applied to image segmentation problems, in Genetic Programming-24th European Conference, EuroGP 2021, Held as Part of EvoStar 2021, Virtual Event, April 7–9, 2021, Proceedings, vol. 12691. (Springer, United States, 2021), pp. 98–113

  11. R.H.R. Lima, A.T.R. Pozo, A. Mendiburu, R. Santana, A Symmetric grammar approach for designing segmentation models, in Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC’20), ed. by T. Hu, N. Lourenço, E. Medvet (IEEE, United States, 2020), pp. 1–8

  12. F. Assunçao, N. Lourenço, P. Machado, B. Ribeiro, DENSER: deep evolutionary network structured representation. Genet. Program. Evolvable Mach. 20(1), 5–35 (2019)

    Article  Google Scholar 

  13. M. Shinozuka, B. Mansouri, Synthetic aperture radar and remote sensing technologies for structural health monitoring of civil infrastructure systems, in Structural Health Monitoring of Civil Infrastructure Systems, ed. by V.M. Karbhari, F. Ansari (Elsevier, Netherlands, 2009), pp. 113–151

  14. K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. in Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 2016, pp. 770–778

  15. A. Krizhevsky, G. Hinton, et al., Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario (2009)

  16. T.N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks. CoRR abs/1609.02907 (2016) arXiv:1609.02907

  17. W.L. Hamilton, R. Ying, J. Leskovec, Inductive representation learning on large graphs. CoRR abs/1706.02216 (2017) arXiv:1706.02216

  18. F.M. Bianchi, D. Grattarola, L. Livi, C. Alippi, Graph neural networks with convolutional ARMA filters. CoRR abs/1901.01343 (2019) arXiv:1901.01343

  19. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, Y. Bengio, Graph attention networks. arXiv preprint arXiv:1710.10903 (2017) arXiv:1710.10903

  20. J. Klicpera, A. Bojchevski, S. Günnemann, Personalized embedding propagation: Combining neural networks on graphs with personalized pagerank. CoRR abs/1810.05997 (2018) arXiv:1810.05997

  21. N. Reimers, I. Gurevych, Sentence-bert: sentence embeddings using siamese BERT-networks. in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. (Association for Computational Linguistics, Unites States, 2019)

  22. L.G. Shapiro, G.C. Stockman, Computer Vision (Prentice-Hall, New Jersey, 2001), p. 580

    Google Scholar 

  23. P. Arbelaez, M. Maire, C. Fowlkes, J. Malik, Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)

    Article  Google Scholar 

  24. A. Cardona, S. Saalfeld, S. Preibisch, B. Schmid, A. Cheng, J. Pulokas, P. Tomancak, V. Hartenstein, An integrated micro-and macroarchitectural analysis of the drosophila brain by computer-assisted serial section electron microscopy. PLoS Biol. 8(10), 1000502 (2010)

    Article  Google Scholar 

  25. G.F. Miller, P.M. Todd, S.U. Hegde, Designing Neural Networks using Genetic Algorithms. in Proceedings of the International Conference on Genetic Algorithms (ICGA’89), vol. 89. (1989), pp. 379–384

  26. D. Whitley, T. Starkweather, C. Bogart, Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput 14(3), 347–361 (1990)

    Article  Google Scholar 

  27. E.-G. Talbi, Automated design of deep neural networks: a survey and unified taxonomy. ACM Comput. Surv. (CSUR) 54(2), 1–37 (2021)

    Article  Google Scholar 

  28. D.B. Fogel, L.J. Fogel, V. Porto, Evolving neural networks. Biol. Cybern. 63(6), 487–493 (1990)

    Article  Google Scholar 

  29. S. Ding, C. Su, J. Yu, An optimizing BP neural network algorithm based on genetic algorithm. Artif. Intell. Rev. 36(2), 153–162 (2011)

    Article  Google Scholar 

  30. F. Petroski Such, V. Madhavan, E. Conti, J. Lehman, K.O. Stanley, J. Clune, Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. CoRR (2017) arXiv:1712.06567

  31. K.O. Stanley, R. Miikkulainen, Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  32. P.R. Lorenzo, J. Nalepa, Memetic evolution of deep neural networks. in Proceedings of the Genetic and Evolutionary Computation Conference. (2018), pp. 505–512

  33. R. Jie, J. Gao, Differentiable neural architecture search for high-dimensional time series forecasting. IEEE Access 9, 20922–20932 (2021)

    Article  Google Scholar 

  34. T. Londt, X. Gao, P. Andreae, Evolving character-level densenet architectures using genetic programming. in International Conference on the Applications of Evolutionary Computation (Part of EvoStar). (Springer, United States, 2021), pp. 665–680

  35. M. Suganuma, S. Shirakawa, T. Nagao, A genetic programming approach to designing convolutional neural network architectures. in Proceedings of the 2017 Genetic and Evolutionary Computation Conference (GECCO’17). (ACM, 2017), pp. 497–504

  36. N. Lourenço, F.B. Pereira, E. Costa, Unveiling the properties of structured grammatical evolution. Genet. Program. Evolvable Mach. 17(3), 251–289 (2016)

    Article  Google Scholar 

  37. N. Lourenço, F. Assunção, F.B. Pereira, E. Costa, P. Machado, Structured grammatical evolution: a dynamic approach (Springer, United States, 2018), pp. 137–161

    Google Scholar 

  38. M. Sundermeyer, R. Schlüter, H. Ney, LSTM neural networks for language modeling. in Thirteenth Annual Conference of the International Speech Communication Association, (2012)

  39. A. Graves, A.-r. Mohamed, G. Hinton, Speech recognition with deep recurrent neural networks. in Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’13). (IEEE, 2013), pp. 6645–6649

  40. M. Schuster, K.K. Paliwal, Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)

    Article  Google Scholar 

  41. N. Lourenço, F. Pereira, E. Costa, Evolving evolutionary algorithms. in Proceedings of the 14th Annual Conference Companion on Genetic and Evolutionary Computation (GECCO’12). (ACM, 2012), pp. 51–58

  42. S. Luke, Essentials of Metaheuristics, 2nd edn. (Lulu, United States, 2013)

    Google Scholar 

  43. Y. Sun, B. Xue, M. Zhang, G.G. Yen, Evolving deep convolutional neural networks for image classification. IEEE Trans. Evol. Comput. 24(2), 394–407 (2019)

    Article  Google Scholar 

  44. G. Huang, Z. Liu, L. Van Der Maaten, K.Q. Weinberger, Densely connected convolutional networks. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2017), pp. 4700–4708

  45. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions. in Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15), (2015)

  46. D.P. Kingma, J. Ba, Adam: a method for stochastic optimization. arXiv (2014) arXiv:1412.6980

  47. S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, J. Gao, Deep learning-based text classification: a comprehensive review. ACM Comput. Surv. (CSUR) 54(3), 1–40 (2021)

    Article  Google Scholar 

  48. A. Baliyan, A. Batra, S.P. Singh, Multilingual sentiment analysis using RNN-LSTM and neural machine translation. in 2021 8th International Conference on Computing for Sustainable Global Development (INDIACom). (IEEE, 2021), pp. 710–713

  49. M.E. Basiri, S. Nemati, M. Abdar, E. Cambria, U.R. Acharya, Abcdm: an attention-based bidirectional CNN-RNN deep model for sentiment analysis. Futur. Gener. Comput. Syst. 115, 279–294 (2021)

    Article  Google Scholar 

  50. L. Yao, C. Mao, Y. Luo, Graph convolutional networks for text classification. in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33. (2019), pp. 7370–7377

  51. J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018) arXiv:1810.04805

  52. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach (2019) arXiv:1907.11692

  53. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale. CoRR (2019) arXiv:1911.02116

  54. O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation. in International Conference on Medical Image Computing and Computer-assisted Intervention. (Springer, United States, 2015), pp. 234–241

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ricardo H. R. Lima.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lima, R.H.R., Magalhães, D., Pozo, A. et al. A grammar-based GP approach applied to the design of deep neural networks. Genet Program Evolvable Mach 23, 427–452 (2022). https://doi.org/10.1007/s10710-022-09432-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-022-09432-0

Keywords

Navigation