Skip to main content
SpringerLink
Log in

Cooperative Coevolutionary Genetic Programming Hyper-Heuristic for Budget Constrained Dynamic Multi-workflow Scheduling in Cloud Computing

  • Conference paper
  • First Online:
Book cover Evolutionary Computation in Combinatorial Optimization (EvoCOP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13987))

  • 348 Accesses

Abstract

Dynamic Multi-workflow Scheduling (DMWS) in cloud computing is a well-known combinatorial optimisation problem. It is a great challenge to tackle this problem by scheduling multiple workflows submitted at different times and meet user-defined quality of service objectives. Scheduling with user-defined budget constraints is becoming increasingly important due to cloud dynamics associated with on-demand provisioning, instance types, and pricing. To address the Budget-Constrained Dynamic Multi-workflow Scheduling (BC-DMWS) problem, a novel Cooperative Coevolution Genetic Programming (CCGP) approach is proposed. Two heuristic rules, namely VM Selection/Creation Rule (VMR) and Budget Alert Rule (BAR), are learned automatically by CCGP. VMR is used to allocate ready tasks to either existing or newly rented VM instances, while BAR makes decisions to downgrade VM instances so as to meet the budget constraint. Experiments show significant performance and success rate improvement compared to state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arabnejad, V., Bubendorfer, K., Ng, B.: Dynamic multi-workflow scheduling: a deadline and cost-aware approach for commercial clouds. Future Gener. Comput. Syst. 100, 98–108 (2019)

    Article  Google Scholar 

  2. AWS: Amazon EC2 on demand pricing (2022). https://aws.amazon.com/ec2/pricing/on-demand/

  3. Blythe, J., Jain, S., et al.: Task scheduling strategies for workflow-based applications in grids. In: CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005, vol. 2, pp. 759–767. IEEE (2005)

    Google Scholar 

  4. Branke, J., Nguyen, S., Pickardt, C.W., Zhang, M.: Automated design of production scheduling heuristics: a review. IEEE Trans. Evol. Comput. 20(1), 110–124 (2015)

    Article  Google Scholar 

  5. Braun, T.D., Siegel, H.J., et al.: A comparison of eleven static heuristics for mapping a class of independent tasks onto heterogeneous distributed computing systems. J. Parallel Distrib. Comput. 61(6), 810–837 (2001)

    Article  Google Scholar 

  6. Burke, E.K., et al.: A classification of hyper-heuristic approaches. In: Gendreau, M., Potvin, J.Y. (eds.) Handbook of Metaheuristics. International Series in Operations Research & Management Science, vol. 146, pp. 449–468. Springer, Boston, MA (2010). https://doi.org/10.1007/978-1-4419-1665-5_15

  7. Chakravarthi, K.K., Neelakantan, P., Shyamala, L., Vaidehi, V.: Reliable budget aware workflow scheduling strategy on multi-cloud environment. Cluster Comput. 25(2), 1189–1205 (2022)

    Article  Google Scholar 

  8. Chen, W., Deelman, E.: Workflowsim: a toolkit for simulating scientific workflows in distributed environments. In: 2012 IEEE 8th International Conference on E-Science, pp. 1–8. IEEE (2012)

    Google Scholar 

  9. Escott, K.-R., Ma, H., Chen, G.: Genetic programming based hyper heuristic approach for dynamic workflow scheduling in the cloud. In: Hartmann, S., Küng, J., Kotsis, G., Tjoa, A.M., Khalil, I. (eds.) DEXA 2020. LNCS, vol. 12392, pp. 76–90. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59051-2_6

    Chapter  Google Scholar 

  10. Escott, K.R., Ma, H., Chen, G.: A genetic programming hyper-heuristic approach to design high-level heuristics for dynamic workflow scheduling in cloud. In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 3141–3148. IEEE (2020)

    Google Scholar 

  11. Escott, K.-R., Ma, H., Chen, G.: Transfer learning assisted GPHH for dynamic multi-workflow scheduling in cloud computing. In: Long, G., Yu, X., Wang, S. (eds.) AI 2022. LNCS (LNAI), vol. 13151, pp. 440–451. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-97546-3_36

    Chapter  Google Scholar 

  12. Faragardi, H.R., Sedghpour, M.R.S., Fazliahmadi, S., Fahringer, T., Rasouli, N.: GRP-HEFT: a budget-constrained resource provisioning scheme for workflow scheduling in IaaS clouds. IEEE Trans. Parallel Distrib. Syst. 31(6), 1239–1254 (2019)

    Article  Google Scholar 

  13. Gao, T., Wu, C.Q., Hou, A., Wang, Y., Li, R., Xu, M.: Minimizing financial cost of scientific workflows under deadline constraints in multi-cloud environments. In: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, pp. 114–121 (2019)

    Google Scholar 

  14. Jakobović, D., Jelenković, L., Budin, L.: Genetic programming heuristics for multiple machine scheduling. In: Ebner, M., O’Neill, M., Ekárt, A., Vanneschi, L., Esparcia-Alcázar, A.I. (eds.) EuroGP 2007. LNCS, vol. 4445, pp. 321–330. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-71605-1_30

    Chapter  Google Scholar 

  15. Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K.: Characterizing and profiling scientific workflows. Future Gener. Comput. Syst. 29(3), 682–692 (2013)

    Article  Google Scholar 

  16. Koza, J.R.: Genetic programming as a means for programming computers by natural selection. Statist. Comput. 4(2), 87–112 (1994)

    Article  Google Scholar 

  17. Koza, J.R., Koza, J.R.: Genetic Programming: on the Programming of Computers by Means of Natural Selection, vol. 1. MIT Press (1992)

    Google Scholar 

  18. Li, H., Wang, D., Xu, G., Yuan, Y., Xia, Y.: Improved swarm search algorithm for scheduling budget-constrained workflows in the cloud. Soft Comput. 26(8), 3809–3824 (2022). https://doi.org/10.1007/s00500-022-06782-w

    Article  Google Scholar 

  19. Lin, J., Zhu, L., Gao, K.: A genetic programming hyper-heuristic approach for the multi-skill resource constrained project scheduling problem. Expert Syst. Appl. 140, 112915 (2020)

    Article  Google Scholar 

  20. Liu, L., Zhang, M., Buyya, R., Fan, Q.: Deadline-constrained coevolutionary genetic algorithm for scientific workflow scheduling in cloud computing. Concurr. Comput. Pract. Exp. 29(5), e3942 (2017)

    Article  Google Scholar 

  21. MacLachlan, J., Mei, Y.: Look-ahead genetic programming for uncertain capacitated arc routing problem. In: 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 1872–1879. IEEE (2021)

    Google Scholar 

  22. Nguyen, S., Zhang, M., Johnston, M., Tan, K.C.: Automatic design of scheduling policies for dynamic multi-objective job shop scheduling via cooperative coevolution genetic programming. IEEE Trans. Evol. Comput. 18(2), 193–208 (2013)

    Article  Google Scholar 

  23. Rizvi, N., Ramesh, D.: Fair budget constrained workflow scheduling approach for heterogeneous clouds. Cluster Comput. 23(4), 3185–3201 (2020). https://doi.org/10.1007/s10586-020-03079-1

    Article  Google Scholar 

  24. Sahni, J., Vidyarthi, D.P.: A cost-effective deadline-constrained dynamic scheduling algorithm for scientific workflows in a cloud environment. IEEE Trans. Cloud Comput. 6(1), 2–18 (2015)

    Article  Google Scholar 

  25. Shi, T., Ma, H., Chen, G., Hartmann, S.: Location-aware and budget-constrained service deployment for composite applications in multi-cloud environment. IEEE Trans. Parallel Distrib. Syst. 31(8), 1954–1969 (2020)

    Article  Google Scholar 

  26. Tan, B., Ma, H., Mei, Y., Zhang, M.: A cooperative coevolution genetic programming hyper-heuristic approach for on-line resource allocation in container-based clouds. IEEE Trans. Cloud Comput. (2020)

    Google Scholar 

  27. Wu, C.Q., Cao, H.: Optimizing the performance of big data workflows in multi-cloud environments under budget constraint. In: 2016 IEEE International Conference on Services Computing (SCC), pp. 138–145. IEEE (2016)

    Google Scholar 

  28. Xiao, Q.Z., Zhong, J., Feng, L., Luo, L., Lv, J.: A cooperative coevolution hyper-heuristic framework for workflow scheduling problem. IEEE Trans. Serv. Comput. (2019)

    Google Scholar 

  29. Yang, Y., Chen, G., Ma, H., Zhang, M., Huang, V.: Budget and SLA aware dynamic workflow scheduling in cloud computing with heterogeneous resources. In: 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 2141–2148. IEEE (2021)

    Google Scholar 

  30. Yu, Y., Ma, H., Chen, G.: Achieving multi-objective scheduling of heterogeneous workflows in cloud through a genetic programming based approach. In: 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 1880–1887. IEEE (2021)

    Google Scholar 

  31. Zhang, F., Mei, Y., Nguyen, S., Zhang, M.: Collaborative multifidelity-based surrogate models for genetic programming in dynamic flexible job shop scheduling. IEEE Trans. Cybern. (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Victoria University of Wellington, Wellington, 6012, New Zealand

    Kirita-Rose Escott, Hui Ma & Gang Chen

Authors
  1. Kirita-Rose Escott
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kirita-Rose Escott .

Editor information

Editors and Affiliations

  1. Pontificia Universidad Católica de Valparaíso, Valparaíso, Chile

    Leslie Pérez Cáceres

  2. Université libre de Bruxelles, Bruxelles, Belgium

    Thomas Stützle

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Escott, KR., Ma, H., Chen, G. (2023). Cooperative Coevolutionary Genetic Programming Hyper-Heuristic for Budget Constrained Dynamic Multi-workflow Scheduling in Cloud Computing. In: Pérez Cáceres, L., Stützle, T. (eds) Evolutionary Computation in Combinatorial Optimization. EvoCOP 2023. Lecture Notes in Computer Science, vol 13987. Springer, Cham. https://doi.org/10.1007/978-3-031-30035-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-30035-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-30034-9

  • Online ISBN: 978-3-031-30035-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation