Created by W.Langdon from gp-bibliography.bib Revision:1.7954
See also Li et al.,Science 378, 1092--1097 (2022) 9 December 2022 DOI: 10.1126/science.abq115
pre-training dataset 715.1 GB of code
p7 'Fine-tuning the model on a dedicated competitive programming dataset is critical for performance'
'high-quality test cases are not readily available'
'slow positives where correct but algorithmically inefficient'
'We reduced the false positive rates of our dataset by generating additional test cases, created by mutating existing test inputs'
sequence-to-sequence, eg NLP to Java, encoder(1536 tokens)-decoder(768 tokens) transformer architecture. AlphaCode 41B 41.1B parameters.
CodeContests: train on both ok and bad answers.
'perhaps because there are many ways solutions can be incorrect while correct solutions tend to behave the same and so are grouped into larger clusters' Anna Karenina (by Leo Tolstoy).
p16 'Solve rates scale log-linearly with more samples.' 'Solve rates scale log-linearly with more compute.'
p21 'boilerplate code for reading and parsing the input data format, rather than key logic for solving problems'.
p22 'AlphaCode generates approximately the same amount of dead code as humans.'
p29 'Improving human readable code generation'
p30 'no knowledge of coding is required to create software.' 'Interpretability makes code generation safer for real-world environments and for fairer machine learning.' 'Generalization.' 'outdated APIs'
'required hundreds of petaFLOPS days'
p31 'Coding capabilities could lead to [Artificial Intelligence] systems that can recursively write and improve themselves, rapidly leading to more and more advanced [AI] systems.'
",
Genetic Programming entries for Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser Remi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d'Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu Oriol Vinyals