AlphaGo Zero

AlphaGo Zero is a version of DeepMind's Go software AlphaGo. AlphaGo's team published an article in Nature in October 2017 introducing AlphaGo Zero, a version created without using data from human games, and stronger than any previous version.^[1] By playing games against itself, AlphaGo Zero: surpassed the strength of AlphaGo Lee in three days by winning 100 games to 0; reached the level of AlphaGo Master in 21 days; and exceeded all previous versions in 40 days.^[2]

Training artificial intelligence (AI) without datasets derived from human experts has significant implications for the development of AI with superhuman skills, as expert data is "often expensive, unreliable, or simply unavailable."^[3] Demis Hassabis, the co-founder and CEO of DeepMind, said that AlphaGo Zero was so powerful because it was "no longer constrained by the limits of human knowledge".^[4] Furthermore, AlphaGo Zero performed better than standard deep reinforcement learning models (such as Deep Q-Network implementations^[5]) due to its integration of Monte Carlo tree search. David Silver, one of the first authors of DeepMind's papers published in Nature on AlphaGo, said that it is possible to have generalized AI algorithms by removing the need to learn from humans.^[6]

Google later developed AlphaZero, a generalized version of AlphaGo Zero that could play chess and Shōgi in addition to Go.^[7] In December 2017, AlphaZero beat the 3-day version of AlphaGo Zero by winning 60 games to 40, and with 8 hours of training it outperformed AlphaGo Lee on an Elo scale. AlphaZero also defeated a top chess program (Stockfish) and a top Shōgi program (Elmo).^[8][9]

Architecture

The network in AlphaGo Zero is a ResNet with two heads.^{[1]: Appendix: Methods}

• The stem of the network takes as input a 17x19x19 tensor representation of the Go board.
  • 8 channels are the positions of the current player's stones from the last eight time steps. (1 if there is a stone, 0 otherwise. If the time step go before the beginning of the game, then 0 in all positions.)
  • 8 channels are the positions of the other player's stones from the last eight time steps.
  • 1 channel is all 1 if black is to move, and 0 otherwise.
• The body is a ResNet with either 20 or 40 residual blocks and 256 channels.
• There are two heads, a policy head and a value head.
  • Policy head outputs a logit array of size ${\displaystyle 19\times 19+1}$, representing the logit of making a move in one of the points, plus the logit of passing.
  • Value head outputs a number in the range ${\displaystyle (-1,+1)}$, representing the expected score for the current player. -1 represents current player losing, and +1 winning.

Training

AlphaGo Zero's neural network was trained using TensorFlow, with 64 GPU workers and 19 CPU parameter servers. Only four TPUs were used for inference. The neural network initially knew nothing about Go beyond the rules. Unlike earlier versions of AlphaGo, Zero only perceived the board's stones, rather than having some rare human-programmed edge cases to help recognize unusual Go board positions. The AI engaged in reinforcement learning, playing against itself until it could anticipate its own moves and how those moves would affect the game's outcome.^[10] In the first three days AlphaGo Zero played 4.9 million games against itself in quick succession.^[11] It appeared to develop the skills required to beat top humans within just a few days, whereas the earlier AlphaGo took months of training to achieve the same level.^[12]

Training cost 3e23 FLOPs, ten times that of AlphaZero.^[13]

For comparison, the researchers also trained a version of AlphaGo Zero using human games, AlphaGo Master, and found that it learned more quickly, but actually performed more poorly in the long run.^[14] DeepMind submitted its initial findings in a paper to Nature in April 2017, which was then published in October 2017.^[1]

Hardware cost

The hardware cost for a single AlphaGo Zero system in 2017, including the four TPUs, has been quoted as around $25 million.^[15]

Applications

According to Hassabis, AlphaGo's algorithms are likely to be of the most benefit to domains that require an intelligent search through an enormous space of possibilities, such as protein folding (see AlphaFold) or accurately simulating chemical reactions.^[16] AlphaGo's techniques are probably less useful in domains that are difficult to simulate, such as learning how to drive a car.^[17] DeepMind stated in October 2017 that it had already started active work on attempting to use AlphaGo Zero technology for protein folding, and stated it would soon publish new findings.^[18][19]

Reception

AlphaGo Zero was widely regarded as a significant advance, even when compared with its groundbreaking predecessor, AlphaGo. Oren Etzioni of the Allen Institute for Artificial Intelligence called AlphaGo Zero "a very impressive technical result" in "both their ability to do it—and their ability to train the system in 40 days, on four TPUs".^[10] The Guardian called it a "major breakthrough for artificial intelligence", citing Eleni Vasilaki of Sheffield University and Tom Mitchell of Carnegie Mellon University, who called it an impressive feat and an “outstanding engineering accomplishment" respectively.^[17] Mark Pesce of the University of Sydney called AlphaGo Zero "a big technological advance" taking us into "undiscovered territory".^[20]

Gary Marcus, a psychologist at New York University, has cautioned that for all we know, AlphaGo may contain "implicit knowledge that the programmers have about how to construct machines to play problems like Go" and will need to be tested in other domains before being sure that its base architecture is effective at much more than playing Go. In contrast, DeepMind is "confident that this approach is generalisable to a large number of domains".^[11]

In response to the reports, South Korean Go professional Lee Sedol said, "The previous version of AlphaGo wasn’t perfect, and I believe that’s why AlphaGo Zero was made." On the potential for AlphaGo's development, Lee said he will have to wait and see but also said it will affect young Go players. Mok Jin-seok, who directs the South Korean national Go team, said the Go world has already been imitating the playing styles of previous versions of AlphaGo and creating new ideas from them, and he is hopeful that new ideas will come out from AlphaGo Zero. Mok also added that general trends in the Go world are now being influenced by AlphaGo's playing style. "At first, it was hard to understand and I almost felt like I was playing against an alien. However, having had a great amount of experience, I’ve become used to it," Mok said. "We are now past the point where we debate the gap between the capability of AlphaGo and humans. It’s now between computers." Mok has reportedly already begun analyzing the playing style of AlphaGo Zero along with players from the national team. "Though having watched only a few matches, we received the impression that AlphaGo Zero plays more like a human than its predecessors," Mok said.^[21] Chinese Go professional Ke Jie commented on the remarkable accomplishments of the new program: "A pure self-learning AlphaGo is the strongest. Humans seem redundant in front of its self-improvement."^[22]

Comparison with predecessors

Configuration and strength^[23]

Versions	Playing hardware[24]	Elo rating	Matches
AlphaGo Fan	176 GPUs,[2] distributed	3,144[1]	5:0 against Fan Hui
AlphaGo Lee	48 TPUs,[2] distributed	3,739[1]	4:1 against Lee Sedol
AlphaGo Master	4 TPUs,[2] single machine	4,858[1]	60:0 against professional players; Future of Go Summit
AlphaGo Zero (40 days)	4 TPUs,[2] single machine	5,185[1]	100:0 against AlphaGo Lee 89:11 against AlphaGo Master
AlphaZero (34 hours)	4 TPUs, single machine[8]	4,430 (est.)[8]	60:40 against a 3-day AlphaGo Zero

AlphaZero

Main article: AlphaZero

On 5 December 2017, DeepMind team released a preprint on arXiv, introducing AlphaZero, a program using generalized AlphaGo Zero's approach, which achieved within 24 hours a superhuman level of play in chess, shogi, and Go, defeating world-champion programs, Stockfish, Elmo, and 3-day version of AlphaGo Zero in each case.^[8]

AlphaZero (AZ) is a more generalized variant of the AlphaGo Zero (AGZ) algorithm, and is able to play shogi and chess as well as Go. Differences between AZ and AGZ include:^[8]

• AZ has hard-coded rules for setting search hyperparameters.
• The neural network is now updated continually.
• Chess (unlike Go) can end in a tie; therefore AZ can take into account the possibility of a tie game.

An open source program, Leela Zero, based on the ideas from the AlphaGo papers is available. It uses a GPU instead of the TPUs recent versions of AlphaGo rely on.

References

1. Silver, David; Schrittwieser, Julian; Simonyan, Karen; Antonoglou, Ioannis; Huang, Aja; Guez, Arthur; Hubert, Thomas; Baker, Lucas; Lai, Matthew; Bolton, Adrian; Chen, Yutian; Lillicrap, Timothy; Fan, Hui; Sifre, Laurent; Driessche, George van den; Graepel, Thore; Hassabis, Demis (19 October 2017). "Mastering the game of Go without human knowledge" (PDF). Nature. 550 (7676): 354–359. Bibcode:2017Natur.550..354S. doi:10.1038/nature24270. ISSN 0028-0836. PMID 29052630. S2CID 205261034. Archived (PDF) from the original on 18 July 2018. Retrieved 2 September 2019.
2. Hassabis, Demis; Siver, David (18 October 2017). "AlphaGo Zero: Learning from scratch". DeepMind official website. Archived from the original on 19 October 2017. Retrieved 19 October 2017.
3. "Google's New AlphaGo Breakthrough Could Take Algorithms Where No Humans Have Gone". Yahoo! Finance. 19 October 2017. Archived from the original on 19 October 2017. Retrieved 19 October 2017.
4. Knapton, Sarah (18 October 2017). "AlphaGo Zero: Google DeepMind supercomputer learns 3,000 years of human knowledge in 40 days". The Telegraph. Archived from the original on 19 October 2017. Retrieved 19 October 2017.
5. mnj12 (7 July 2021), mnj12/chessDeepLearning, retrieved 7 July 2021
6. "DeepMind AlphaGo Zero learns on its own without meatbag intervention". ZDNet. 19 October 2017. Archived from the original on 20 October 2017. Retrieved 20 October 2017.
7. https://www.idi.ntnu.no/emner/it3105/materials/neural/silver-2017b.pdf
8. Silver, David; Hubert, Thomas; Schrittwieser, Julian; Antonoglou, Ioannis; Lai, Matthew; Guez, Arthur; Lanctot, Marc; Sifre, Laurent; Kumaran, Dharshan; Graepel, Thore; Lillicrap, Timothy; Simonyan, Karen; Hassabis, Demis (5 December 2017). "Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm". arXiv:1712.01815 [cs.AI].
9. Knapton, Sarah; Watson, Leon (6 December 2017). "Entire human chess knowledge learned and surpassed by DeepMind's AlphaZero in four hours". The Telegraph. Archived from the original on 2 December 2020. Retrieved 5 April 2018.
10. Greenemeier, Larry. "AI versus AI: Self-Taught AlphaGo Zero Vanquishes Its Predecessor". Scientific American. Archived from the original on 19 October 2017. Retrieved 20 October 2017.
11. "Computer Learns To Play Go At Superhuman Levels 'Without Human Knowledge'". NPR. 18 October 2017. Archived from the original on 20 October 2017. Retrieved 20 October 2017.
12. "Google's New AlphaGo Breakthrough Could Take Algorithms Where No Humans Have Gone". Fortune. 19 October 2017. Archived from the original on 19 October 2017. Retrieved 20 October 2017.
13. "Data on Notable AI Models". Epoch AI. 19 June 2024. Retrieved 29 November 2024.
14. "This computer program can beat humans at Go—with no human instruction". Science | AAAS. 18 October 2017. Archived from the original on 2 February 2022. Retrieved 20 October 2017.
15. Gibney, Elizabeth (18 October 2017). "Self-taught AI is best yet at strategy game Go". Nature News. doi:10.1038/nature.2017.22858. Archived from the original on 1 May 2020. Retrieved 10 May 2020.
16. "The latest AI can work things out without being taught". The Economist. Archived from the original on 19 October 2017. Retrieved 20 October 2017.
17. Sample, Ian (18 October 2017). "'It's able to create knowledge itself': Google unveils AI that learns on its own". The Guardian. Archived from the original on 19 October 2017. Retrieved 20 October 2017.
18. "'It's able to create knowledge itself': Google unveils AI that learns on its own". The Guardian. 18 October 2017. Archived from the original on 19 October 2017. Retrieved 26 December 2017.
19. Knapton, Sarah (18 October 2017). "AlphaGo Zero: Google DeepMind supercomputer learns 3,000 years of human knowledge in 40 days". The Telegraph. Archived from the original on 15 December 2017. Retrieved 26 December 2017.
20. "How Google's new AI can teach itself to beat you at the most complex games". Australian Broadcasting Corporation. 19 October 2017. Archived from the original on 20 October 2017. Retrieved 20 October 2017.
21. "Go Players Excited About 'More Humanlike' AlphaGo Zero". Korea Bizwire. 19 October 2017. Archived from the original on 21 October 2017. Retrieved 21 October 2017.
22. "New version of AlphaGo can master Weiqi without human help". China News Service. 19 October 2017. Archived from the original on 19 October 2017. Retrieved 21 October 2017.
23. "【柯洁战败解密】AlphaGo Master最新架构和算法，谷歌云与TPU拆解" (in Chinese). Sohu. 24 May 2017. Archived from the original on 17 September 2017. Retrieved 1 June 2017.
24. Hardware used during training may be substantially more powerful

External links and further reading

• "AlphaGo Zero: Starting from scratch". Archived from the original on 3 January 2020.
• Singh, S.; Okun, A.; Jackson, A. (2017). "AOP". Nature. 550 (7676): 336–337. Bibcode:2017Natur.550..336S. doi:10.1038/550336a. PMID 29052631. S2CID 4447445.
• Silver, David; Schrittwieser, Julian; Simonyan, Karen; Antonoglou, Ioannis; Huang, Aja; Guez, Arthur; Hubert, Thomas; Baker, Lucas; Lai, Matthew; Bolton, Adrian; Chen, Yutian; Lillicrap, Timothy; Hui, Fan; Sifre, Laurent; Van Den Driessche, George; Graepel, Thore; Hassabis, Demis (2017). "Mastering the game of Go without human knowledge" (PDF). Nature. 550 (7676): 354–359. Bibcode:2017Natur.550..354S. doi:10.1038/nature24270. PMID 29052630. S2CID 205261034.
• AlphaGo Zero Games