The slow traversal of 'human-level'

By Katja Grace, 21 January 2015

Once you have normal-human-level AI, how long does it take to get Einstein-level AI? We have seen that a common argument for 'not long at all' based on brain size does not work in a straightforward way, though a more nuanced assessment of the evidence might. Before we get to that though, let's look at some more straightforward evidence (from our new page on the range of human intelligence).

In particular, let's look at chess. AI can play superhuman-level chess, so we can see how it got there. And how it got there is via about four decades of beating increasingly good players, starting at beginners and eventually passing Kasparov (note that a beginner is something like level F or below, which doesn't make it onto this graph):

Figure 1: Chess AI progress compared to human performance, from Coles 2002. The original article was apparently written before 1993, so note that the right of the graph (after ‘now’) is imagined, though it appears to be approximately correct.

Something similar is true in Go (where -20 on this graph is a good beginner score, and go bots are not yet superhuman, but getting close):

From Grace 2013.

Backgammon and poker AI's seems to have progressed similarly, though backgammon took about 2 rather than 4 decades (we will soon post more detailed descriptions of progress in board games).

Go, chess, poker, and backgammon are all played using different algorithms. But the underlying problems are sufficiently similar that they could easily all be exceptions.

Other domains are harder to measure, but seem basically consistent with gradual progress. Machine translation seems to be gradually moving through the range of human expertise, as does automatic driving. There are fewer clear cases where AI abilities took years rather than decades to move from subhuman to superhuman, and the most salient cases are particularly easy or narrow problems (such as arithmetic, narrow perceptual tasks, or easy board games).

If narrow AI generally traverses the relevant human range slowly, this suggests that general AI will take some time to go from minimum minimum wage competency to—well, at least to AI researcher competency. If you combine many narrow skills, each progressing gradually through the human spectrum at different times, you probably wouldn't end up with a much more rapid change in general performance. And it isn't clear that a more general method should tend to progress faster than narrow AI.

However, we can point to ways that general AI might be different from board game AI.

Perhaps progress in chess and go has mostly been driven by hardware progress, while progress in general AI will be driven by algorithmic improvements or acquiring more training data.

Perhaps the kinds of algorithms people really use to think scale much better than chess algorithms. Chess algorithms only become 30-60 Elo points stronger with each doubling of hardware, whereas a very rough calculation suggests human brains become more like 300 Elo points better per doubling in size.

In humans, brain size has roughly a 1/3 correlation with intelligence. Given that the standard deviation of brain size is about 10% of the size of the brain (p. 39), this suggests that a doubling of brain size leads to a relatively large change in chess-playing ability. On a log scale, a doubling is a 7 standard deviation change in brain size, which would suggest a ~2 standard deviation change in intelligence. It's hard to know how this relates to chess performance, but in Genius in Chess Levitt gives an unjustified estimate of 300 Elo points. This is what we would expect if intelligence were responsible for half of variation in performance (neglecting the lower variance of chess player intelligence), since a standard deviation of chess performance is about 2000/7 ~ 300 Elo. Each of these correlations is problematic but nevertheless suggestive.

If human intelligence in general scales much better with hardware than existing algorithms, and hardware is important relative to software, then AI based on an understanding of human intelligence may scale from sub-human to superhuman more quickly than the narrow systems we have seen. However these are both open questions.

(Image: The Chess Players, by Honoré Daumier)