Edward K. Smith
Yuriy Brun
ABSTRACT
Automated program repair has shown promise for reducing the significant manual effort debugging requires. This paper addresses a deficit of earlier evaluations of automated repair techniques caused by repairing programs and evaluating generated patches' correctness using the same set of tests. Since tests are an imperfect metric of program correctness, evaluations of this type do not discriminate between correct patches and patches that overfit the available tests and break untested but desired functionality. This paper evaluates two well-studied repair tools, GenProg and TrpAutoRepair, on a publicly available benchmark of bugs, each with a human-written patch. By evaluating patches using tests independent from those used during repair, we find that the tools are unlikely to improve the proportion of independent tests passed, and that the quality of the patches is proportional to the coverage of the test suite used during repair. For programs that pass most tests, the tools are as likely to break tests as to fix them. However, novice developers also overfit, and automated repair performs no worse than these developers. In addition to overfitting, we measure the effects of test suite coverage, test suite provenance, and starting program quality, as well as the difference in quality between novice-developer-written and tool-generated patches when quality is assessed with a test suite independent from the one used for patch generation.
- M. Abd-El-Malek, G. R. Ganger, G. R. Goodson, M. K. Reiter, and J. J. Wylie. Fault-scalable Byzantine fault-tolerant services. In ACM Symposium on Operating Systems Principles (SOSP), pages 59–74, Brighton, UK, 2005. Google Scholar
Digital Library
- R. E. Adamson. Functional fixedness as related to problem solving: A repetition of three experiments. Journal of Experimental Psychology, 44(4):288–291, 1952.Google ScholarCross Ref
- E. Alba and F. Chicano. Finding safety errors with ACO. In Conference on Genetic and Evolutionary Computation (GECCO), pages 1066–1073, London, England, UK, July 2007. Google ScholarDigital Library
- M. Alkhalaf, A. Aydin, and T. Bultan. Semantic differential repair for input validation and sanitization. In International Symposium on Software Testing and Analysis (ISSTA), pages 225–236, San Jose, CA, USA, July 2014. Google ScholarDigital Library
- A. Arcuri and L. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 1–10, Honolulu, HI, USA, 2011. Google ScholarDigital Library
- A. Arcuri and X. Yao. A novel co-evolutionary approach to automatic software bug fixing. In Congress on Evolutionary Computation, pages 162–168, 2008.Google ScholarCross Ref
- E. T. Barr, Y. Brun, P. Devanbu, M. Harman, and F. Sarro. The plastic surgery hypothesis. In Symposium on the Foundations of Software Engineering (FSE), pages 306–317, Hong Kong, China, November 2014. Google ScholarDigital Library
- A. Barreto, M. Barros, and C. Werner. Staffing a software project: A constraint satisfaction approach. Computers and Operations Research, 35(10):3073–3089, 2008. Google ScholarDigital Library
- A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. Henri-Gros, A. Kamsky, S. McPeak, and D. Engler. A few billion lines of code later: Using static analysis to find bugs in the real world. Communications of the ACM, 53(2):66–75, Feb. 2010. Google ScholarDigital Library
- C. Bird, A. Bachmann, E. Aune, J. Duffy, A. Bernstein, V. Filkov, and P. Devanbu. Fair and balanced?: Bias in bugfix datasets. In European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE), pages 121–130, Amsterdam, The Netherlands, August 2009. Google ScholarDigital Library
- Y. Brun, E. Barr, M. Xiao, C. Le Goues, and P. Devanbu. Evolution vs. intelligent design in program patching. Technical Report https://escholarship.org/uc/item/3z8926ks, UC Davis: College of Engineering, 2013.Google Scholar
- C. Cadar, D. Dunbar, and D. Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In USENIX Conference on Operating Systems Design and Implementation (OSDI), pages 209–224, San Diego, CA, USA, 2008. Google ScholarDigital Library
- A. Carzaniga, A. Gorla, A. Mattavelli, N. Perino, and M. Pezzè. Automatic recovery from runtime failures. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 782–791, San Francisco, CA, USA, 2013. Google ScholarDigital Library
- A. Carzaniga, A. Gorla, N. Perino, and M. Pezzè. Automatic workarounds for web applications. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), pages 237–246, Santa Fe, New Mexico, USA, 2010. Google ScholarDigital Library
- L. Chen and A. Avižienis. N-version programming: A faulttolerance approach to reliability of software operation. In IEEE International Symposium on Fault-Tolerant Computing (FTCS), pages 3–9, 1978.Google Scholar
- R. Cochran, L. D’Antoni, B. Livshits, D. Molnar, and M. Veanes. Program boosting: Program synthesis via crowdsourcing. In Symposium on Principles of Programming Languages (POPL), pages 677–688, Mumbai, India, January 2015. Google ScholarDigital Library
- Z. Coker and M. Hafiz. Program transformations to fix C integers. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 792–801, San Francisco, CA, USA, 2013. Google ScholarDigital Library
- V. Debroy and W. Wong. Using mutation to automatically suggest fixes for faulty programs. In International Conference on Software Testing, Verification, and Validation, pages 65–74, Paris, France, 2010. Google ScholarDigital Library
- T. Durieux, M. Martinez, M. Monperrus, R. Sommerard, and J. Xuan. Automatic repair of real bugs: An experience report on the Defects4J dataset. CoRR, abs/1505.07002, 2015.Google Scholar
- H.-C. Estler, C. A. Furia, M. Nordio, M. Piccioni, and B. Meyer. Contracts in practice. In International Symposium on Formal Methods (FM), pages 230–246, Singapore, May 2014.Google ScholarDigital Library
- Z. P. Fry, B. Landau, and W. Weimer. A human study of patch maintainability. In International Symposium on Software Testing and Analysis (ISSTA), pages 177–187, Minneapolis, MN, USA, July 2012. Google ScholarDigital Library
- M. Gabel and Z. Su. Testing mined specifications. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), Cary, NC, USA, 2012. Google ScholarDigital Library
- S. Gulwani. Automating string processing in spreadsheets using input-output examples. In Symposium on Principles of Programming Languages (POPL), pages 317–330, Austin, TX, USA, 2011. Google ScholarDigital Library
- S. Gustafson, A. Ekárt, E. Burke, and G. Kendall. Problem difficulty and code growth in genetic programming. Genetic Programming and Evolvable Machines, pages 271–290, Sept. 2004. Google ScholarDigital Library
- M. Harman. The current state and future of search based software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 342–357, 2007. Google ScholarDigital Library
- G. Jin, L. Song, W. Zhang, S. Lu, and B. Liblit. Automated atomicity-violation fixing. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 389–400, San Jose, CA, USA, 2011. Google ScholarDigital Library
- R. Just, D. Jalali, and M. D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pages 437–440, San Jose, CA, USA, July 2014. Google ScholarDigital Library
- Y. Ke, K. T. Stolee, C. Le Goues, and Y. Brun. Repairing programs with semantic code search. In Proceedings of the 30th IEEE/ACM International Conference On Automated Software Engineering (ASE), Lincoln, NE, USA, November 2015.Google ScholarDigital Library
- D. Kim, J. Nam, J. Song, and S. Kim. Automatic patch generation learned from human-written patches. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 802–811, San Francisco, CA, USA, 2013. Google ScholarDigital Library
- J. C. Knight and N. G. Leveson. An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering (TSE), 12(1):96– 109, 1986. Google ScholarDigital Library
- J. R. Koza. Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, 1992. Google ScholarDigital Library
- C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In AMC/IEEE International Conference on Software Engineering (ICSE), pages 3–13, Zurich, Switzerland, 2012. Google ScholarDigital Library
- C. Le Goues, S. Forrest, and W. Weimer. Representations and operators for improving evolutionary software repair. In Conference on Genetic and Evolutionary Computation (GECCO), pages 959–966, Philadelphia, PA, USA, July 2012. Google ScholarDigital Library
- C. Le Goues, N. Holtschulte, E. K. Smith, Y. Brun, P. Devanbu, S. Forrest, and W. Weimer. The ManyBugs and Intro-Class benchmarks for automated repair of C programs. IEEE Transactions on Software Engineering (TSE), in press, 2015.Google ScholarCross Ref
- C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer. Gen-Prog: A generic method for automatic software repair. IEEE Transactions on Software Engineering (TSE), 38:54–72, 2012. Google ScholarDigital Library
- Y. Lin and S. S. Kulkarni. Automatic repair for multi-threaded programs with deadlock/livelock using maximum satisfiability. In International Symposium on Software Testing and Analysis (ISSTA), pages 237–247, San Jose, CA, USA, July 2014. Google ScholarDigital Library
- P. Liu, O. Tripp, and C. Zhang. Grail: Context-aware fixing of concurrency bugs. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), pages 318– 329, Hong Kong, China, Nov. 2014. Google ScholarDigital Library
- A. S. Luchins. Mechanization in problem solving: The effect of Einstellung. Psychological Monographs, 54(6):i–95, 1942.Google Scholar
- S. Mechtaev, J. Yi, and A. Roychoudhury. DirectFix: Looking for simple program repairs. In International Conference on Software Engineering (ICSE), Florence, Italy, May 2015.Google ScholarDigital Library
- C. C. Michael, G. McGraw, and M. A. Schatz. Generating software test data by evolution. IEEE Transactions on Software Engineering (TSE), 27(12):1085–1110, Dec. 2001. Google ScholarDigital Library
- T. M. Mitchell. Machine Learning. McGraw-Hill, New York, 1997. Google ScholarDigital Library
- M. Monperrus. A critical review of “Automatic patch generation learned from human-written patches”: Essay on the problem statement and the evaluation of automatic software repair. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 234–242, Hyderabad, India, June 2014. Google ScholarDigital Library
- H. D. T. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra. SemFix: Program repair via semantic analysis. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 772–781, San Francisco, CA, USA, 2013. Google ScholarDigital Library
- M. Orlov and M. Sipper. Flight of the FINCH through the Java wilderness. IEEE Transactions on Evolutionary Computation, 15(2):166–182, Apr. 2011. Google ScholarDigital Library
- C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In International Symposium on Software Testing and Analysis (ISSTA), pages 199–209, Toronto, ON, Canada, 2011. Google ScholarDigital Library
- Y. Pei, C. A. Furia, M. Nordio, Y. Wei, B. Meyer, and A. Zeller. Automated fixing of programs with contracts. IEEE Transactions on Software Engineering (TSE), 40(5):427–449, 2014. Google ScholarDigital Library
- J. H. Perkins, S. Kim, S. Larsen, S. Amarasinghe, J. Bachrach, M. Carbin, C. Pacheco, F. Sherwood, S. Sidiroglou, G. Sullivan, W.-F. Wong, Y. Zibin, M. D. Ernst, and M. Rinard. Automatically patching errors in deployed software. In ACM Symposium on Operating Systems Principles (SOSP), pages 87–102, Big Sky, MT, USA, October 12–14, 2009. Google ScholarDigital Library
- D. Posnett, V. Filkov, and P. Devanbu. Ecological inference in empirical software engineering. In International Conference on Automated Software Engineering (ASE), pages 362–371, Lawrence, KS, USA, November 2011. Google ScholarDigital Library
- Y. Qi, X. Mao, and Y. Lei. Efficient automated program repair through fault-recorded testing prioritization. In International Conference on Software Maintenance (ICSM), pages 180–189, Eindhoven, The Netherlands, Sept. 2013. Google ScholarDigital Library
- Z. Qi, F. Long, S. Achour, and M. Rinard. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In International Symposium on Software Testing and Analysis (ISSTA), pages 24–36, Baltimore, MD, USA, 2015. Google ScholarDigital Library
- Research Triangle Institute. The economic impacts of inadequate infrastructure for software testing. NIST Planning Report 02-3, May 2002.Google Scholar
- J. Rissanen. Modelling by the shortest data description. Automatica, 14:465–471, 1978. Google ScholarDigital Library
- O. Seng, J. Stammel, and D. Burkhart. Search-based determination of refactorings for improving the class structure of object-oriented systems. In Conference on Genetic and Evolutionary Computation (GECCO), pages 1909–1916, Seattle, WA, USA, July 2006. Google ScholarDigital Library
- S. Sidiroglou and A. D. Keromytis. Countering network worms through automatic patch generation. IEEE Security and Privacy, 3(6):41–49, Nov. 2005. Google ScholarDigital Library
- S. Sidiroglou-Douskos, E. Lahtinen, F. Long, and M. Rinard. Automatic error elimination by horizontal code transfer across multiple applications. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), pages 43–54, Portland, OR, USA, 2015. Google ScholarDigital Library
- S. Silva and E. Costa. Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories. Genetic Programming and Evolvable Machines, 10(2):141–179, June 2009. Google ScholarDigital Library
- A. Smirnov and T. cker Chiueh. Dira: Automatic detection, identification and repair of control-hijacking attacks. In Network and Distributed System Security Symposium (NDSS), San Diego, CA, USA, Feb. 2005.Google Scholar
- S. H. Tan and A. Roychoudhury. relifix: Automated repair of software regressions. In International Conference on Software Engineering (ICSE), Florence, Italy, May 2015.Google ScholarCross Ref
- K. R. Walcott, M. Soffa, G. M. Kapfhammer, and R. S. Roos. Time-aware test suite prioritization. In International Symposium on Software Testing and Analysis (ISSTA), pages 1–12, Portland, ME, USA, July 2006. Google ScholarDigital Library
- Y. Wei, Y. Pei, C. A. Furia, L. S. Silva, S. Buchholz, B. Meyer, and A. Zeller. Automated fixing of programs with contracts. In International Symposium on Software Testing and Analysis (ISSTA), pages 61–72, Trento, Italy, 2010. Google ScholarDigital Library
- W. Weimer, Z. P. Fry, and S. Forrest. Leveraging program equivalence for adaptive program repair: Models and first results. In IEEE/ACM International Conference on Automated Software Engineering (ASE), Palo Alto, CA, USA, 2013.Google ScholarDigital Library
- W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest. Automatically finding patches using genetic programming. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 364–374, Vancouver, BC, Canada, 2009. Google ScholarDigital Library
- A. Zeller and R. Hildebrandt. Simplifying and isolating failureinducing input. IEEE Transactions on Software Engineering, 28(2):183–200, February 2002. Google ScholarDigital Library
- H. Zhong and Z. Su. An empirical study on real bug fixes. In ACM/IEEE International Conference on Software Engineering (ICSE), Florence, Italy, May 2015.Google ScholarCross Ref
Index Terms
- Is the cure worse than the disease? overfitting in automated program repair
Recommendations
A correlation study between automated program repair and test-suite metrics
Automated program repair is increasingly gaining traction, due to its potential to reduce debugging cost greatly. The feasibility of automated program repair has been shown in a number of works, and the research focus is gradually shifting toward the ...
Efficient Automated Program Repair through Fault-Recorded Testing Prioritization
ICSM '13: Proceedings of the 2013 IEEE International Conference on Software MaintenanceMost techniques for automated program repair use test cases to validate the effectiveness of the produced patches. The validation process can be time-consuming especially when the object programs ship with either lots of test cases or some long-running ...
Do automated program repair techniques repair hard and important bugs?
Existing evaluations of automated repair techniques focus on the fraction of the defects for which the technique can produce a patch, the time needed to produce patches, and how well patches generalize to the intended specification. However, these ...
Comments