The Expert Survey on Progress in AI (ESPAI) is a big survey of AI researchers that I’ve led four times—in 2016, then annually: 2022, 2023, and 2024 (results coming soon!)

Each time so far it’s had substantial attention—the first one was the 16th ‘most discussed’ paper in the world in 2017.

Various misunderstandings about it have proliferated, leading to the methodology being underestimated in terms of robustness and credibility (perhaps in part due to insufficient description of the methodology—the 2022 survey blog post was terse). To avoid these misconceptions muddying interpretation of the 2024 survey results, I’ll answer key questions about the survey methodology here.

This covers the main concerns I know about. If you think there’s an important one I’ve missed, please tell me—in comments or by email (katja@aiimpacts.org).

This post throughout discusses the 2023 survey, but the other surveys are very similar. The biggest differences are that a few questions have been added over time, and we expanded from inviting respondents at two publication venues to six in 2023. The process for contacting respondents (e.g. finding their email addresses) has also seen many minor variations.

How good is this survey methodology overall?

To my knowledge, the methodology is substantially stronger than is typical for surveys of AI researcher opinion. For comparison, O’Donovan et al was reported on by Nature Briefing this year, and while 53% larger, its methodology appeared worse in most relevant ways: its response rate was 4% next to 2023 ESPAI’s 15%; it doesn’t appear to report efforts to reduce or measure non-response bias, the survey population was selected by the authors and not transparent, and 20% of completed surveys were apparently excluded (see here for a fuller comparison of the respective methodologies).

Some particular strengths of the ESPAI methodology:

• Size: The survey is big—2778 respondents in 2023, which was the largest survey of AI researchers that had been conducted at the time.

• Bias mitigations: We worked hard to minimize the kinds of things that create non-response bias in other surveys—

  • We wrote individually to every contactable author for a set of top publication venues rather than using less transparent judgments of expertise or organic spread.

  • We obscured the topic in the invitation.

  • We encouraged a high response rate across the board through payments, a pre-announcement, and many reminders. We have experimented over the years with details of these approaches (e.g. how much to pay respondents) in order to increase our response rate.

• Question testing: We tested and honed questions by asking test-respondents to take the survey while talking aloud to us about what they think the questions mean and how they think about answering them.

• Testing for robustness over framings: For several topics, we ask similar questions in several different ways, to check if respondents’ answers are sensitive to apparently unimportant framing choices. In earlier iterations of the survey, we found that responses are sensitive to framing, and so have continued including all framings in subsequent surveys. We also highlight this sensitivity in our reporting of results.

• Consistency over time: We have used almost entirely identical questions every year so can accurately report changes over time (the exception is e.g. several minor edits to task descriptions for ongoing accuracy, and the addition of new questions).

• Non-respondent comparison: In 2023, we looked up available demographic details for a large number of non-respondents so that we could measure the representativeness of the sample along these axes.

Criticisms of the ESPAI methodology seem to mostly be the result of basic misunderstandings, as I’ll discuss below.

Did lots of respondents skip some questions?

No.

Respondents answered nearly all the normal questions they saw (excluding demographics, free response, and conditionally-asked questions). Each of these questions was answered by on average 96% of those who saw it, with the most-skipped still at 90% (a question about the number of years until the occupation of “AI researcher” would be automatable).1

The reason it could look like respondents skipped a lot of questions is that in order to ask more distinct questions, we intentionally only directed a fraction (5-50%) of respondents to most questions. We selected those respondents randomly, so the smaller pool answering each question is an unbiased sample of the larger population.

Here’s the map of paths through the survey and how many people were given which questions in 2023. Respondents start at the introduction then randomly receive one of the questions or sub-questions at each stage. The randomization shown below is uncorrelated, except that respondents get either the “fixed-years” or “fixed-probabilities” framing throughout for questions that use those, to reduce confusion.2

Did only a handful of people answer extinction-related questions?

No. There were two types of question that mentioned human extinction risk, and roughly everyone who answered any question—97% and 95% of respondents respectively, thousands of people—answered some version of each.

Confusion on this point likely arose because there are three different versions of one question type—so at a glance you may notice that only about a quarter of respondents answered a question about existential risk to humanity within one hundred years, without seeing that the other three quarters of respondents answered one of two other very similar questions3. (As discussed in the previous section, respondents were allocated randomly to question variants.) All three questions got a median of 5% or 10% in 2023.

This system of randomly assigning question variants lets us check the robustness of views across variations on questions (such as ‘...within a hundred years’), while still being able to infer that the median view across the population puts the general risk at 5% or more (if the chance is 5% within 100 years, it is presumably at least 5% for all time)4.

In addition, every respondent was assigned another similar question about the chance of ‘extremely bad outcomes (e.g. human extinction)’, providing another check on their views.

So the real situation is that every respondent who completed the survey was asked about outcomes similar to extinction in two different ways, across four different precise questions. These four question variations all got similar answers—in 2023, median 5% chance of something like human extinction (one question got 10%). So we can say that across thousands of researchers, the median view puts at least a 5% chance on extinction or similar from advanced AI, and this finding is robust across question variants.

Did lots of people drop out, biasing answers?

No. Of people who answered at least one question in the 2023 survey, 95% got to seeing the final demographics question at the end5. And, as discussed above, respondents answered nearly all the (normal) questions they saw.

So there is barely any room for bias from people dropping out. Consider the extinction questions: even if most pessimistically, exactly the least concerned 5% of people didn’t get to the end, and least concerned 5% of those who got there skipped the second extinction-related question (everyone answered the first one), then the real medians are what look like the 47.5th and 45th percentiles now for the two question sets, which are still both 5%.6

On a side note: one version of this concern envisages the respondents dropping out in disapproval at the survey questions being focused on topics like existential risk from AI, systematically biasing the remaining answers toward greater concern for that. This concern suggests a misunderstanding about the content of the survey. For instance, half of respondents were asked about everyday risks from AI such as misinformation, inequality, empowering authoritarian rulers or dangerous groups before extinction risk from AI was even mentioned as an example (Q2 vs. Q3). The other question about existential risk is received at the end.

Did the survey have a low response rate?

No. The 2023 survey was taken by 15% of those we contacted.7 This appears to be broadly typical or high for a survey such as this.

It seems hard to find clear evidence about typical response rates, because surveys can differ in so many ways. The best general answer we got was an analysis by Hamilton (2003), which found the median response rate across 199 surveys to be 26%. They also found that larger invitation lists tended to go with lower response rates—surveys sent to over 20,000 people, like ours, were expected to have a response rate in the range of 10%. And specialized populations (such as scientists) also commonly had lower response rates.

For another comparison, O’Donovan et al 2025 was a recent, similarly sized survey of AI researchers which used similar methods of recruiting and got a 4% response rate.

Are the AI risk answers inflated much from concerned people taking the survey more?

Probably not.

Some background on the potential issue here: the ESPAI generally reports substantial probabilities on existential risk to humanity from advanced AI (‘AI x-risk’)—the median probability of human extinction or similar has always been at least 5%8 (across different related questions, and years). The question here is whether these findings represent the views of the broad researcher population, or if they are caused by massive bias in who responds.

There are a lot of details in understanding why massive bias is unlikely, but in brief:

1. As discussed above, very few people drop out after answering any questions, and very few people skip each question, so there cannot be much item non-response bias from respondents who find AI risk implausible dropping out or skipping questions.

2. This leaves the possibility of unit non-response bias—skewed results from people with certain opinions systematically participating less often. We limit the potential for this by writing individually to each person, with an invitation that doesn’t mention risk. Recipients might still infer more about the survey content through recognizing our names, recognizing the survey from previous years, following the links from the invitation, or looking at 1-3 questions without answering. However for each of these there is reason to doubt they produce a substantial effect. The AI safety community may be overrepresented, and some people avoid participating because of other views, however these groups appear to be a small fraction of the respondent pool. We also know different demographic subsets participate at different rates, and have somewhat different opinions on average. Nonetheless, it is highly implausible that the headline result—5% median AI x-risk—is caused by disproportionate participation from researchers who are strongly motivated by AI x-risk concerns, because the result is robust to excluding everyone who reports thinking about the social impacts of AI even moderately.

The large majority of respondents answer every question—including those about AI x-risks

One form of non-response bias is item nonresponse, where respondents skip some questions or drop out of the survey. In this case, the concern would be that unconcerned respondents skip questions about risk, or drop out of the survey when they encounter such questions. But this can only be a tiny effect here—in 2023 ~95% of people who answered at least one question reached the end of the survey. (See section “Did lots of people drop out when they saw the questions…”). If respondents were leaving due to questions about (x-)risk, we would expect fewer respondents to have completed the survey.

This also suggests low unit non-response bias among unconcerned members of the sample: if people often decide not to participate if they recognize that the survey includes questions about AI x-risk, we’d also expect more respondents to drop out when they encounter such questions (especially since most respondents should not know the topic before they enter the survey—see below). Since very few people drop out upon seeing the questions, it would be surprising if a lot of people had dropped out earlier due to anticipating the question content.

Most respondents don’t know there will be questions about x-risk, because the survey invitation is vague

We try to minimize the opportunity for unit non-response bias by writing directly to every researcher we can who has published in six top AI venues rather than having people share the survey, and making the invitation vague: avoiding directly mentioning anything like risks at all, let alone extinction risk. For instance, the 2023 survey invitation describes the topic as “the future of AI progress”9.

So we expect most sample members are not aware that the survey includes questions about AI risks until after they open it.

There is still an opportunity for non-response bias from some people deciding not to answer the survey after opening it and looking at questions. However only around 15% of people who look at questions leave without answering any, and these people can only see the first three pages of questions before the survey requires an answer to proceed. Only the third page mentions human extinction, likely after many such people have left. So the scale of plausible non-response bias here is small.

Recognition of us or our affiliations is unlikely to have a large influence

Even in a vague invitation, some respondents could still be responding to our listed affiliations connecting us with the AI Safety community, and some recognize us.10 This could be a source of bias. However different logos and affiliations get similar response rates11, and it seems unlikely that very many people in a global survey have been recognizing us, especially since 2016 (when the survey had a somewhat higher response rate and the same median probability of extremely bad outcomes as in 2023)12. Presumably some people remember taking the survey in a previous year, but in 2023 we expanded the pool from two venues to six, reducing the fraction who might have seen it before, and got similar answers on existential risk (see p14).

As confirmation that recognition of us or our affiliations is not driving the high existential risk numbers, recognition would presumably be stronger in some demographic groups than others, e.g. people who did undergrad in the US over Europe or Asia, and probably industry over academia, yet when we checked in 2023, all these groups gave median existential risk numbers of at least 5%13.

Links to past surveys included in the invitation do not foreground risk.

Another possible route to recipients figuring out there will be questions about extinction risk is that we do link to past surveys in the invitation. However the linked documents (from 2022 or 2023) also do not foreground AI extinction risk, so this seems like a stretch.14

So it should be hard for most respondents to decide if to respond based on the inclusion of existential risk questions.

AI Safety researchers are probably more likely to participate, but there are few of them

A big concern seems to be that members of “the AI (Existential) Safety community”, i.e. those whose professional focus is reducing existential risk from AI, are more likely to participate in the survey. This is probably true—anecdotally, people in this community are often aware of the survey and enthusiastic about it, and a handful of people wrote to check that their safety-interested colleagues have received an invitation.

However this is unlikely to have a strong effect, since the academic AI Safety community is quite small compared to the number of respondents.

One way to roughly upper-bound the fraction of respondents from the AI Safety community is to note that they are very likely to have ‘a particular interest’ in the ‘social impacts of smarter-than-human machines’. However, when asked “How much thought have you given in the past to social impacts of smarter-than-human machines?” only 10.3% gave an answer that high.

People decline the survey for opinion reasons, but probably few

As well as bias from concerned researchers being motivated to respond to the survey, at the other end of the spectrum there can be bias from researchers who are motivated to particularly avoid participating for reasons correlated with opinion. I know of a few of instances of this, and a tiny informal poll suggested it could account for something like 10% of non-respondents15, though this seems unlikely, and even if so, this would have a small effect on the results.

Demographic groups differ in propensity to answer and average opinions

We have been discussing bias from people’s opinions affecting whether they want to participate. There could also be non-response bias from other factors influencing both opinion and desire to participate. For instance, in 2023 we found that women participated around 66% of the base rate, and generally expected less extreme positive or negative outcomes. This is a source of bias, however since women were only around one in ten of the total population, the scale of potential error from this is limited.

We similarly measured variation in the responsiveness of some other demographic groups, and also differences of opinion between these demographic groups among those who did respond, which together give some heuristic evidence of small amounts of bias. Aside from gender, the main dimension where we noted a substantial difference in response rate and also in opinion was for people who did undergraduate study in Asia. They were only 84% as likely as the base rate to respond, and in aggregate expected high level machine intelligence earlier, and had higher median extinction or disempowerment numbers. This suggests an unbiased survey would find AI to be sooner and more risky. So while it is a source of bias, it is in the opposite direction to that which has prompted concern.

Respondents who don’t think about AI x-risk report the same median risk

We have seen various evidence that people engaged with AI safety do not make up a large fraction of the survey respondents. However there is another strong reason to think extra participation from people motivated by AI safety does not drive the headline 5% median, regardless of whether they are overrepresented. We can look at answers from a subset of people who are unlikely to be substantially drawn by AI x-risk concern: those who report not having thought much about the issue. (If someone has barely ever thought about a topic, it is unlikely to be important enough to them to be a major factor in their decision to spend a quarter of an hour participating in a survey.) Furthermore, this probably excludes most people who would know about the survey or authors already, and so potentially anticipate the topics.

We asked respondents, “How much thought have you given in the past to social impacts of smarter-than-human machines?” and gave them these options:

• Very little. e.g. “I can’t remember thinking about this.”

• A little. e.g. “It has come up in conversation a few times”

• A moderate amount. e.g. “I read something about it now and again”

• A lot. e.g. “I have thought enough to have my own views on the topic”

• A great deal. e.g. “This has been a particular interest of mine”

Looking at only respondents who answered ‘a little’ or ‘very little’—i.e. those who had at most discussed the topic a few times—the median probability of “human extinction or similarly permanent and severe disempowerment of the human species” from advanced AI (asked with or without further conditions) was 5%, the same as for the entire group. Thus we know that people who are highly concerned about risk from AI are not responsible for the median x-risk probability being at least 5%. Without them, the answer would be the same.

Is the survey small?

No, it is large.

In 2023 we wrote to around 20,000 researchers—everyone whose contact details we could find from six top AI publication venues (NeurIPS, ICML, ICLR, AAAI, IJCAI, and JMLR).16 We heard back from 2778. As far as we could tell, it was the largest ever survey of AI researchers at the time. (It could be this complaint was only made about the 2022 survey, which was 738 respondents before we expanded the pool of invited authors from two publication venues—NeurIPS and ICML—to six, but I’d say that was also pretty large17.)

Is the survey biased to please funders?

Unlikely: at a minimum, there is little incentive to please funders.

The story here would be that we, the people running the survey, might want results that support the views of our funders, in exchange for their funding. Then we might adjust the survey in subtle ways to get those answers.

I agree that where one gets funding is a reasonable concern in general, but I’d be surprised if it was relevant here. Some facts:

• It has been easy to find funding for this kind of work, so there isn’t much incentive to do anything above and beyond to please funders, let alone extremely dishonorable things.

• A lot of the funding specifically raised for the survey is for paying the respondents. We pay them to get a high response rate and reduce non-response bias. It would be weird to pay $100k to reduce non-response bias, only to get yourself socially obligated to bring about non-response bias (on top of the already miserable downside of having to send $50 to 2000 people across lots of countries and banking situations). It’s true that the first effect is more obvious, so this might suffice if we just wanted to look unbiased, but in terms of our actual decision-making as I have observed it, it seems like we are weighing up a legible reduction in bias against effort and not thinking about funders much.

Is it wrong to make guesses about highly uncertain future events without well-supported quantitative models?

One criticism is that even AI experts have no valid technical basis for making predictions about the future of AI. This is not a criticism of the survey methodology, per se, but rather a concern that the results will be misinterpreted or taken too seriously.

I think there are two reasons it is important to hear AI researchers’ guesses about the future, even where they are probably not a reliable forecast.

First, it has often been assumed or stated that nobody who works in AI is worried about AI existential risk. If this were true, it would be a strong reason for the public to be reassured. However hearing the real uncertainty from AI researchers disproves this viewpoint, and makes a case for serious investigation of the concern. In this way even uncertain guesses are informative, because they let us know that the default assumption in confident safety was mistaken.

Second, there is not an alternative to making guesses about the future. Policy decisions are big bets on guesses about the future, implicitly. For instance when we decide whether to rush a technology or to carefully regulate it, we are guessing about the scale of various benefits and the harms.

Where trustworthy quantitative models are available, of course those are better. But in the absence, the guesses of a large number of relatively well-informed people is often better than the unacknowledged guesses of whoever is called upon to make implicit bets on the future.

That said, there seems little reason to think these forecasts are highly reliable—they should be treated as rough estimates, often better responded to by urgent more dedicated analysis of the issues they hazily outline over acting on the exact numbers.

When people say ‘5%’ do they mean a much smaller chance?

The concern here is that respondents are not practiced at thinking in terms of probabilities, and may consequently say small numbers (e.g. 5%) when they mean something that would be better represented by an extremely tiny number (perhaps 0.01% or 0.000001%). Maybe especially if the request for a probability prompts them to think of integers between 0 and 100.

One reason to suspect this kind of error is that Karger et al. (2023, p29) found a group of respondents gave extinction probabilities nearly six orders of magnitude lower when prompted differently.18

This seems worth attending to, but I think unlikely to be a big issue here for the following reasons:

• If your real guess is 0.0001%, and you feel that you should enter an integer number, the natural inputs would seem to be 0% or 1%—it’s hard to see how you would get to 5%.19

• It’s hard to square 5% meaning something less than 1% with the rest of the distribution for the extinction questions—does 10% mean less than 2%? What about 50%? Where the median response is 5%, many respondents gave these higher numbers, and it would be strange if these were also confused efforts to enter minuscule numbers, and also strange if the true distribution had a lot of entries greater than 10% but few around 5%. (See Fig. 10 in the paper for the distribution for one extinction-relevant question.)

• Regarding the effect in Karger et al. (2023) specifically, this has only been observed once to my knowledge, and is extreme and surprising. And even if it turned out to be real and widespread, including in highly quantitatively educated populations, it would be a further question whether the answers produced by such prompting are more reliable than standard ones. So it seems premature to treat this as substantially undermining respondents’ representations of their beliefs.

• It would surprise me if that effect were widely replicable and people stayed with the lower numbers, because in that world I’d expect people outside of surveys to often radically reduce their probabilities of AI x-risk when they give the question more thought (e.g. enough to have run into other examples of low probability events in the meantime). Yet survey respondents who have previously given a “lot” or a “great deal” of thought to the social impacts of smarter-than-human machines give similar and somewhat higher numbers than those who have thought less. (See A.2.1)

What do you think are the most serious weaknesses and limitations of the survey?

While I think the quality of our methodology is exceptionally high, there are some significant limitations of our work. These don’t affect our results about expert concern about risk of extinction or similar, but do add some noteworthy nuance.

1) Experts’ predictions are inconsistent and unreliable
As we’ve emphasized in our papers reporting the survey results, experts’ predictions are often inconsistent across different question framings—such sensitivity is not uncommon, and we’ve taken care to mitigate this by using multiple framings. Experts also have such a wide variety of different predictions on many of these questions that they must each be fairly inaccurate on average (though this says nothing about whether as a group their aggregate judgments are good).

2) It is not entirely clear what sort of “extremely bad outcomes” experts imagine AI will cause

We ask two different types of questions related to human extinction: 1) a question about “extremely bad outcomes (e.g. human extinction)”, 2) questions about “human extinction or similarly permanent and severe disempowerment of the human species”. We made the latter broader than ‘human extinction’ because we are interested in scenarios that are effectively the end of humanity, rather than just those where literally every homo sapiens is dead. This means however that it isn’t clear how much probability participants place on literal extinction versus adjacent strong human disempowerment and other extremely bad scenarios. And there is some evidence that the fraction is low: some respondents explicitly mentioned risks other than extinction in write-in responses, and anecdotally, it seems common for AI researchers to express more concern about issues other than human extinction.

For many purposes, it isn’t important to distinguish between extinction and outcomes that are similarly extremely bad or disempowering to humanity. Yet if the catastrophes many participants have in mind are not human extinction, but the results lend themselves to simplification as ‘risk of extinction’, this can be misleading. And perhaps more than you’d expect, if for instance ‘extinction’ tends to bring to mind a different set of causes than ‘permanent and severe human disempowerment’.

3) Non-response bias is hard to eliminate

Surveys generally suffer from some non-response bias. We took many steps to minimize this, and find it implausible that our results are substantially affected by whatever bias remains (see the earlier question “Are the AI risk answers inflated much from concerned people taking the survey more?”). But we could do even more to estimate or eliminate response bias, e.g. paying some respondents much more than $50 to complete the survey and estimating the effect of doing so.

Is this the kind of low quality research that couldn’t get into an academic journal?

No. We published the near-identical 2016 survey in the Journal of AI Research, so the methodology had essentially been peer reviewed.20 Publication is costly and slow, and AI survey results are much more interesting sooner than later.

The 2023 paper was actually also just published, but in the meantime you had the results more than a year earlier!

There’s a new report on the AI Impacts web site, that focuses on reanalyzing the data from the 2023 Expert Survey on Progress in AI (originally written up in Thousands of AI Authors on the Future of AI).

The report, by Tom Adamczewski, introduces several improvements over the earlier analysis. Even better, it comes with a brand new open-source codebase, also by Tom, that anyone can use to perform their own analyses on the data.

I found the report illuminating, and I recommend it to anyone interested in the tradeoffs and options available in eliciting, analyzing, and presenting quantitative estimates from experts. Plus there are many excellent new graphs!

Overview

The competition attracted 90 entries in total (only one of which was obviously just the work of an LLM!), taking a wide variety of angles on the topic. The judges awarded the top four prizes as follows:

• $7,000 to Rudolf Laine for essays on wisdom, amortised optimisation, and AI:

  • Part I: Wisdom, amortised optimisation, and AI

  • Part II: Growth and amortised optimisation

  • Part III: AI effects on amortised optimisation

• $6,000 to Thane Ruthenis for Towards the operationalization of philosophy & wisdom

• $4,000 to Chris Leong for essays on training wise AI systems:

  • An Overview of “Obvious” Approaches to Training Wise AI Advisors

  • Some Preliminary Notes on the Promise of a Wisdom Explosion

• $3,000 to Gabriel Recchia for Should we just be building more datasets?

Additionally, the judges awarded ten runner-up prizes, of $500 each. These adjustments to the prize schedule were made to better reflect the judges’ assessments — see footnote for details1. 

Many of those essays which did not get prizes still had something valuable to recommend them, and we are very grateful for the deep thought participants put into engaging with these potentially-crucial topics.

In the rest of this post, we’ll link to all fourteen of the prize-winning entries, and provide judges’ commentary on these. We have made a point to include critical commentary as well as praise. In our view this is an essential part of the competition — helping to draw collective attention to the ideas people have presented, and (we hope) helping to advance the discourse on what makes for valuable work. 

Judge introductions

Andreas Stuhlmüller (AS) — Hi, I'm CEO & cofounder of Elicit, an AI company working on scaling up high-quality reasoning, starting with science. I've been interested in how AI can differentially advance wisdom for a long time, and (pre LLMs) founded the non-profit Ought to work on that topic.

Linh Chi Nguyen (CN) — Hi, I have been thinking about long-term risk from AI besides alignment for a few years. I landed on thinking a bunch about AIs’ decision theories, which is what I’m currently doing. In that process, I’ve thought a bit more generally about the philosophical capabilities and inclinations of AI systems and I’m excited about more work in this area.

Bradford Saad (BS) — Hi, I’m a philosopher and senior research fellow at Oxford’s Global Priorities Institute. Most of my past work is in philosophy of mind. But for the past few years I have been working on AI and catastrophic risks, including the risk that future AI agents will cause catastrophes as a result of making philosophical errors. (I’ll add that my comments are not as developed and well-put as I’d like and that with apologies I’ve only managed to write comments for some winning entries in time for the announcements.)

David Manley (DM) — Hi, I’m an Associate Professor of Philosophy at the University of Michigan, Ann Arbor. I’ve worked in philosophical semantics, ontology, epistemology, and global priorities research. 

Due to timing issues, Wei Dai withdrew from the formal judging.

Top Prize Winners

Note that after the winners had been chosen, we provided them with feedback for the judges, and gave them an opportunity to revise their submissions before this announcement. The judges’ comments are generally based on the original submitted versions.

Wisdom, amortised optimisation, and AI — Rudolf Laine — $7,000

This was submitted as one essay, and split into three parts during the revision process, to improve readability:

• Part I: Wisdom, amortised optimisation, and AI

• Part II: Growth and amortised optimisation

• Part III: AI effects on amortised optimisation

Summary

A lot of wisdom is about mental qualities shaped more by amortised optimisation than direct optimisation. Direct optimisation is solving problems by searching through alternatives on the fly, while amortised optimisation is solving a problem by applying past data and computation. LLMs are products of amortised optimisation, but can be used for direct optimisation.

Amortised optimisation matters more when growth and change are slow, so the relative importance of it will likely decline with the growth and changed caused by AI. However, a model suggests it's hard for amortised optimisation to entirely decline in relevance.

Human amortised optimisation is enabled by things like cultural evolution and written knowledge. Advanced AI may weaken human cultural evolution, but may themselves be very good at it due to their low copying costs. AIs might help humans distil insights from written knowledge, but ultimately replace the need for it.

A benefit of wisdom is avoiding large-scale, or "strategic", errors. Good strategic planning often relies on amortised optimisation. If we need to get strategy right about novel things like AI, we could use AIs to help with simulation. A specific type of strategy error that seems hard to solve with AI is wrong paradigms.

BS’s comments

This entry was one of my nominations for a top prize. It persuaded me that the contrast between direct vs. amortised optimization does a lot of work in explaining the contrast between intelligence and wisdom and that the former distinction merits more consideration from a macrostrategic perspective of considering how to make the evolution of AI go well. The essay also contains many points that I found plausible and worthy of further reflection. I wrote some notes in an attempt to distill ideas that stood out to me in the submitted version, but I would recommend that people read the essay rather than the notes.

CN’s comments

Tl;dr: Interesting ideas analysed in a way that makes it easy to trust the author, but somewhat lacking in a coherent, convincing, and important narrative.

I think of this text as having three parts that don’t quite follow the way the author divided the text. The first one offers a characterisation of wisdom. The second theoretically discusses the future of wisdom, including a growth model and considering cultural evolution. The third one discusses concrete and practical ways in which LLMs might contribute to wisdom.

All parts were interesting although I thought they weren’t very well integrated such that, after reading, it was difficult to remember the contents of the text. I think with a little work, the text might read more like a coherent piece or, failing that, might benefit from being split. I thought the first two sections were clearly stronger than the last section. [Editor’s note: it has now been split along these lines.]

I would like to highlight that I perceived the author to take an appropriate epistemic stance in this text. At virtually all points in the text when I thought “what about this objection?” or “this doesn’t seem quite true because of X”, the text addressed my concern in one of the following paragraphs. Sometimes, the text didn’t really provide an answer and instead just noted the concern. While that’s obviously worse than providing an answer, I really appreciated that the author acknowledged potential weaknesses of his conclusions.

That said, the entry didn’t offer a big original idea with a “big if true and plausibly true” feel (which, to be clear, is a very high bar, and I do think that the text contributes many interesting thoughts). I also have some reservations about this type of highly conceptual, foundational work on philosophy and wisdom since the path to impact is relatively long.

Overall, I think the text merits a top prize for what I consider its virtues in scholarship and the ideas that it contains.

DM’s comments

This is a very thought-provoking set of posts, about which I have several concerns. (The TL;DR is that if you read these, you should also read “Tentatively against making AIs ‘wise’” as a counterbalance.) What follows is a summary of my notes on the original submission in which these posts formed a single paper, and in which Laine was making a bolder claim about the connection between amortized optimization (AO) and wisdom. 

We are told that a lot of wisdom is amortized optimization. We might wonder: how often do instances of wisdom involve AO, and when they do, how much of the wisdom is attributable to AO? Surely, being produced by AO surely isn’t sufficient for a process or heuristic to be wise: Laine admits as much.

I would go further and argue that what distinguishes wise judgments from unwise ones, irrespective of how much AO is involved, has more to do with aspects of reasoning that fall on the other side of Laine’s conceptual divide. At least, so I would argue if we’re using “wise” in any normative sense, any sense in which it seems obvious that we want to make people and AIs wiser than they are. Indeed, I’m more confident that legibly good epistemics are crucial for this normative sense of wisdom than I am that there is any deep connection between wisdom and amortized optimization, per se. 

Take Laine’s example of cultural evolution. Crucially, enacting whatever practices have been handed down to us by our forebears is only sometimes wise; the fact that it involves AO doesn’t distinguish between cases where it’s done wisely and those in which it isn’t. In the best kind of case, the human consciously knows that they are benefiting from many generations of (say) successful arrow-making, and that they are unlikely to do better by designing a new method from the ground up. In the worst kind of case, the human blindly and superstitiously mimics cultural practices they’ve inherited, leading to harmful cultural inertia. 

In short, the wisest use of cultural evolution will be clear-eyed about why it makes sense to do so-- that is, it will verify the use of AO using legible and deliberate cognition. And often, consciously rejecting a more AO-style process in favor of a more transparent one will be the wisest path.

(Note also that Laine’s examples involve cases where a practice/artifact has been optimized for something beneficial, like survival. But often selection processes aren’t even optimized for anything beneficial to begin with. For example, many memes succeed not through benefiting the host, but because they happen to be good at spreading. I doubt we want to count the use of an AO artifact as wise as long as it just so happens that it’s beneficial. Adopting the output of an AO process without any reflection about the process that produced it or its current applicability, is often very unwise, even if one gets lucky.)

These points matter not just for the verbal dispute about what constitutes “wisdom”, but for the practical question about what kinds of cognitive features we should imbue our AIs with. For example, Laine characterizes wisdom in part by illegibility; but do we want to create systems whose apparently deep insights people trust despite the lack of any transparent lines of reasoning behind them? As Delaney’s submission notes, insofar as we characterize “wisdom” by mystery/ enigma/ illegibility, we may want to actively avoid building systems that are wise in this sense.

Towards the operationalization of philosophy & wisdom — Thane Ruthenis — $6,000

Summary

I provide candidate definitions for philosophy and wisdom, relate them to intuitive examples of philosophical and wise reasoning, and offer a tentative formalization of both disciplines. The motivation for this is my belief that their proper operationalization is the bottleneck both to scaling up the work done in these domains (i. e., creating an ecosystem), and to automatizing them.

I operationalize philosophy as “the process of deriving novel ontologies”, further concretized as “deriving some assumptions using which reality-as-a-whole could be decomposed into domains that could be studied separately”. I point out the similarity of this definition to John Wentworth’s operationalization of natural abstractions, from which I build the tentative formal model. In addition, I link philosophical reasoning to conceptual/qualitative/non-paradigmatic research, arguing that they’re implemented using the same cognitive algorithms. Counterweighting that, I define philosophy-as-a-discipline as a special case of this reasoning, focused on decomposing the “dataset” represented by the sum of all of our experiences of the world.

I operationalize wisdom as meta-level cognitive heuristics that iterate on object-level heuristics for planning/inference, predicting the real-world consequences of a policy which employs said object-level heuristics. I provide a framework of agency in which that is well-specified as “inversions of inversions of environmental causality”.

I close things off with a discussion of whether AIs would be wise/philosophical (arguing yes), and what options my frameworks offer regarding scaling up or automatizing these kinds of reasoning.

DM’s comments

There is a great deal that I liked about this paper, though I think it would be better construed as offering “operationalizations” not of philosophy and wisdom but of one characteristic thing philosophers (and often other theorists) do, and one characteristic thing wise people do. The two conceptual tasks that are operationalized here--viz., coming up with “ontologies” in the Ruthenis’s sense, and applying good meta-heuristics--are important enough to consider in their own right, even if they overlap only partially with the notions of philosophy and wisdom respectively. 

Among the “other theorists” I have in mind are theoretical physicists, who are actively engaged with the question of which ontology best characterizes the domain of physics (e.g. whether strings should be a fundamental part of it); and while there is also a thriving field of the philosophy of physics, that area seems best described as pursuing meta-questions about the relationship between these competing ontologies and the reality behind them, especially in the case where we may have reached an in-principle limit to our ability to empirically distinguish them. In other words, the borderline between physics and philosophy of physics in fact has to do with the presence or absence of in-principle empirical evidence, not with whether there are competing ontologies at issue. Likewise, I would say, for other domains where philosophy interacts with the sciences: for example, the question of how best to carve up cognitive traits most usefully to predict and explain behavior or pathology is well within the ambit of psychology, whereas the philosophy of mind pursues questions even further from the empirical frontier.

In characterizing the probabilistic independence criterion for ontologies, I wish Ruthenis had explained how this relates to more traditional articulations of theoretical desiderata in the philosophy of science, especially explanatory and predictive power. I was also unsure about how the model is supposed to apply in a case where a given higher level (functional) state (such as being a tree, to use the Ruthenis’s example) has multiple possible lower-level realizations that are all mutually inconsistent (specifying, among other things, exactly how many leaves are on a tree), since the latter will yield probabilities of zero for P(Li | L \ Li ∧ H). 

I agree that the notion of meta-heuristics is an important aspect of wisdom and that it’s importantly distinct from “outside-view” reasoning. The discussion of wisdom overlaps somewhat with Laine’s submission, making the point that wisdom is not explicit knowledge but is stored as “learned instincts, patterns of behavior… cultural norms” that are crystallized through both biological and cultural evolution. I would argue that Ruthenis’s analysis cuts more finely here, helping to explain why not everything crystalized in this way fits into the “wisdom” category (why is a thing like GPT1, that is made of amortized optimization, not actually wise?); and also, conversely, why not everything that is wise has been crystalized in this way. (“Wisdom can nevertheless be inferred “manually”... purely from the domain’s object-level model, given enough effort and computational resources.”)

CN’s comments

Tl;dr: Thought-provoking, many interesting ideas, but lacking in analysis and scholarship. [Editor’s note: the final version was edited to address some of these issues, but I don’t know how much the judges would still be concerned about them.]

I thought the text was very ambitious in a good way and made a lot of points that made me think. For example, the definition for wisdom made me wonder whether it seems right and go through examples and potential counterexamples. So, from that perspective, I really liked the text and appreciate its ideas.

That said, the quality of writing and analysis could have been improved a lot. Many of the thoughts, while very interesting, felt merely stated instead of argued for. In particular, many claims were confidently stated as conclusions when I didn’t think they followed from previous arguments. (I also couldn’t help but notice that virtually all examples were drawn from the LessWrong memesphere, which isn’t a problem per se, but adds to the sense that the author isn’t necessarily aware of the gaps in argumentation.) I also would have liked the text to engage more with reasons to doubt its own conclusions (or at least not state them as confidently in places where I thought it was unwarranted).

I also would have liked it if the text had tried to situate itself more in existing literature as I’m sure that many of the text’s ideas have already been discussed elsewhere. The text also is hard to follow at times, such that I could have benefitted from more and clearer explanations. I also have some reservations about this type of highly conceptual, foundational work on philosophy and wisdom since the path to impact is relatively long.

As we are highlighting this text by giving it a top prize, I would like to share that I wouldn’t want people to read it without applying a strong critical lens. I think some texts are valuable because they give you a lot of food for thought. Other texts are valuable because they are rigorous, do a lot of the work of critical thinking for you, and make you trust the author’s conclusion. This text definitely falls into the former and not the latter category.

With all that said, I overall think the text makes a lot of intellectual contributions, some of which have a “potentially big if (something in this vicinity is) true and plausibly (something in this vicinity is) true” feel, and merits a top prize.

BS’s comments

I found this to be one of the more thought provoking entries. I liked that it made ambitious positive proposals.

To echo some of Chi’s comments: because this entry is receiving a top prize, I also want to encourage readers to approach it as food for thought and with a strong critical lens. To that end, I’ll mention a few somewhat critical reflections:

• As an academic philosopher with only a moderate level of familiarity with Rationalist ideas, I was struck by how many of the interesting ideas in the entry were ones that I had previously encountered in Rationalist writings. I also suspected that I would have a better sense of what is original in the entry if I were more familiar with Rationalist writings and that I would have significantly overestimated the originality of the entry if (like many of my fellow academic philosophers) I were wholly unfamiliar with Rationalist ideas. I offer this mainly as one observation that readers might use to inform their own judgement of the essay rather than as a concern about its philosophical substance.

• I appreciated that the entry was able to cover a lot of ground in virtue of not belaboring qualifications. But at times I thought it went too far in this direction. As a fairly central example, my impression was that the entry largely theorized about philosophy both without displaying deep familiarity with the history or discipline of philosophy and without sufficiently registering the epistemic risks associated with doing so. More generally, my impression was that this entry achieved an especially high degree of boldness partly through significant expense to epistemic caution. Irrespective of the fruits of that tradeoff in this particular case, I’d generally prefer that researchers working in this area place more weight on epistemic caution and be averse to making this type of tradeoff.

• The entry’s proposed operationalization of philosophy seemed to me to at best capture only a fairly narrow aspect of philosophy. I suspect that the most useful and interesting way to understand the proposal is as targeting something else (perhaps to do with the development of conceptual schemes, as I think another judge suggested in a contest meeting). So, when considering the entry’s proposal, readers might want to contemplate what the best target for the proposal would be.

Essays on training wise AI systems — Chris Leong — $4,000

• An Overview of “Obvious” Approaches to Training Wise AI Advisors

• Some Preliminary Notes on the Promise of a Wisdom Explosion

These were submitted as part of a single essay, which centrally argued for a particular approach to training wise AI systems. In response to judge feedback it was split into two relatively separate parts, and the emphasis on the imitation learning approach was lessened.

Summary

The first essay compares the advantages and disadvantages of four “obvious” approaches to producing wise AI systems:

• Imitation learning: Training imitation learning agents on a bunch of people the lab considers to be wise

• The Direct Approach: Training an AI to be wise using optimisation based on human demonstrations and feedback

• The Principled Approach: Attempting to understand what wisdom is at a deep principled level and build an AI that provides advice according to those principles

• The Scattergun Approach: This approach involves just throwing a bunch of potentially relevant wise principles and/or anecdotes (nuggets of wisdom) from a fixed set at the deciders in the hope that reading through it will lead to a wise decision

The second essay introduces and examines the idea of a “wisdom explosion”, in analogy to an intelligence explosion. It argues that it is plausible that a wisdom explosion could be achieved, and that it looks potentially favourable to pursue from a differential technology development perspective.

CN’s comments

(Tl;dr: Gives an overview of the “obvious” approaches to training for wisdom and considerations around them.

The text aims to convince the reader of the imitation learning approach to training AIs to be wise. To this aim, it contrasts it with other possible “obvious” approaches. I found the resulting overview of “obvious” approaches and the considerations around them to be the more valuable part of this text compared to the arguments in favour of the imitation learning approach. I can imagine this being a handy resource to look at when thinking about how to train wisdom, both as a starting point, a refresher, and to double-check that one hasn’t forgotten anything important.

That said, I thought the analysis of the different proposals was reasonable but didn’t stand out to me, which also contributes to me not being convinced that the imitation learning approach is in fact the best approach. This would possibly have been better if the entry had been written out: I personally find that bullet points often hide gaps in argumentation that I notice when I try to write them out. (This might also be completely idiosyncratic to me.)

Overall, I think this text deserves a top prize for doing solid, potentially useful work.

BS’s comments

This entry was one of my nominations for a top prize.  It looks at a number of different proposals for making AI wise. I thought it did an exceptionally good job of considering a range of relevant proposals, thinking about obstacles to implementation, and making a sustained effort to weigh up pros and cons of rival approaches.  Although I think some of the proposals this entry discusses merit further consideration, I’d especially like to see more work in this area that follows this one in considering a range of rival proposals, obstacles to implementation, and pros and cons.

Should we just be building more datasets? — Gabriel Recchia — $3,000

Summary

Building out datasets and evaluations focused on 'wisdom-relevant skills' could significantly contribute to our ability to make progress on scalable oversight and avoid large-scale errors more broadly. Specifically, I suggest that curating datasets for use as fine-tuning corpora or prompts to elicit latent model capabilities in areas such as error detection, task decomposition, cumulative error recognition, and identification of misleading statements could be a particularly worthwhile project to attempt. These datasets should cover a range of domains to test whether these skills can be elicited in a general way. This kind of work is relatively accessible, not requiring insider access to unpublished frontier models, specialized ML experience, or extensive resources.

CN’s comments

Tl;dr: Makes the case for a direction I think is promising, but not a lot of novelty or “meat”.

This is a short, pragmatic and relatively simple submission pointing out that we might just want to do the obvious thing and generate training data on things we consider important for wisdom. I have a lot of sympathy for concise, “not trying to be fancy” proposals that, if implemented, could be very impactful. I also think the text did a good job explaining for which domains additional training data is and isn’t helpful.

That said, I thought the intellectual contribution of this text could have been bigger. First, the text leaves out the analysis of whether wisdom is a domain for which additional training data is helpful. Second, how successful a potential implementation would come down a lot to the details, i.e. what exactly to put in the training data. While the text offers some preliminary thoughts on this, I think they are not enough to meaningfully guide a project trying to implement the proposal, so any implementers would have to basically think about this from scratch. [Editor’s note: the final version of the essay does include more ideas along these lines, and also links to a practical project building datasets.] This seems particularly relevant given that the basic idea of the proposal isn’t very original.

(To be clear, I think it’s much better to be not original and impactful than original and unlikely to ever be implemented or help. That’s part of the charme of this submission! But when evaluating this in terms of intellectual contribution, I would have liked it to either be novel or more fleshed out.)

Overall, I think this text deserves a top prize and I liked it for what I think it is: A brief high-level proposal for a promising if not very original agenda.

BS’s comments

This entry identifies some LLM wisdom-relevant skills that could plausibly be improved with suitable datasets using existing methods. I see this entry as belonging to the category of saying sensible, relatively straightforward stuff that’s worth saying rather than to the category of trying to offer theoretical insights. On the current margin, I think work in this area that aims to contribute via the former category is more to be encouraged than work that aims to contribute via the latter. So I liked that this entry nicely exemplified a category in which I’d like to see more work. I also liked that it identified a task (building datasets) that could plausibly lead to quality improvements in automated philosophy and which is a task that philosophers are already in a position to help with.

Runner-up prizes

Designing Artificial Wisdom: The Wise Workflow Research Organization — Jordan Arel

This was judges’ favourite of a series of four relatively standalone essays the author wrote on artificial wisdom. The other three are:

• On Artificial Wisdom

• Designing Artificial Wisdom: GitWise and AlphaWise

• Designing Artificial Wisdom: Decision Forecasting AI & Futarchy

Summary

Even simple workflows can greatly enhance the performance of LLM’s, so artificially wise workflows seem like a promising candidate for greatly increasing Artificial Wisdom (AW).

This piece outlines the idea of introducing workflows into a research organization which works on various topics related to AI Safety, existential risk & existential security, longtermism, and artificial wisdom. Such an organization could make progressing the field of artificial wisdom one of their primary goals, and as workflows become more powerful they could automate an increasing fraction of work within the organization.

Essentially, the research organization, whose goal is to increase human wisdom around existential risk, acts as scaffolding on which to bootstrap artificial wisdom.

Such a system would be unusually interpretable since all reasoning is done in natural language except that of the base model. When the organization develops improved ideas about existential security factors and projects to achieve these factors, they could themselves incubate these projects, or pass them on to incubators to make sure the wisdom does not go to waste.

CN’s comments

I like the series for what it shows about the author thinking from ground up and at a high-level about all the relevant aspects of the automation of philosophy and wisdom. I thought the scheme presented here was reasonable although fairly vague and, at its level of abstraction, not very novel.

Philosophy's Digital Future — Richard Yetter Chappell

Summary

I suggest that a future "PhilAI" may (at a minimum) be well-suited to mapping out the philosophical literature, situating (e.g.) every paper in the PhilPapers database according to its philosophical contributions and citation networks. The resulting "PhilMap" would show us, at a glance, where the main “fault lines” lie in a debate, and which objections remain unanswered. This opens up new ways to allocate professional esteem: incentivizing philosophers to plug a genuine gap in the literature (or to generate entirely new branches), and not just whatever they can sneak past referees. I further argue that this could combine well with a "Publish then Filter" model (replacing pre-publication gatekeeping with post-publication review) of academic production.

BS’s comments

I liked that this entry made concrete suggestions for using AI to improve information flow across the field of philosophy and that these suggestions were grounded in knowledge of how the field currently operates. 

Regardless of whether one is sold on the particular proposals in this entry, I think its proposals encourage optimism about the availability of options for accelerating philosophical progress, options that can be uncovered without theoretical insights just by reflecting on how to apply existing tools across the field. My sense is that there’s a lot of room for improvement within academic philosophy when it comes to coordination and experimenting with concrete proposals for, say, improving the publication system. So, I’d also be particularly excited for future work on using automation to improve the ecosystem of academic philosophy to happen alongside work geared at promoting the requisite sorts of coordination/experimentation for implementing or testing promising proposals.

CN’s comments

I thought this presented cool product ideas that would improve Philosophy and that I’d be excited for someone to pursue. That said, I think it wasn’t the most relevant to making sure AGI goes well.

Synthetic Socrates and the Philosophers of the Future — Jimmy Alfonso Licon

Summary

Many philosophers prize finding deep, important philosophical truths such as the nature of right and wrong, the ability to make free choices, and so on. Perhaps, then, it would be better for such philosophers to outsource the search for such truths to entities that are better equipped to the task: artificial philosophers. This suggestion may appear absurd, initially, until we realize that throughout human history outsourcing tasks has been the norm for thousands of years. To the extent such philosophers care about discovering deep philosophical truths, they have a reason to aid in the creation of artificial philosophers who will eventually, in many respects, do philosophy better than even the best human philosopher who ever lived or who will live.

CN’s comments

I thought this was a well-written piece of communication that I’m happy to have out there and shared with people. That said, I don’t think it will offer any new considerations to the typical person who knows about our competition.

BS’s comments

I take the central idea of this piece to be that philosophers who care about the discovery of deep philosophical truths have reason to aid in the creation of artificial philosophers rather than simply trying to discover such truths. I’d be delighted for more academic philosophers to engage with this idea. 

The Web of Belief — Paal Fredrik S. Kvarberg

Summary

In this essay, I present a method for using technological innovations to improve rational belief formation and wise decision-making in an explainable manner. I assume a view of rationality in which beliefs are evaluated according to norms of intelligibility, accuracy and consistency. These norms can be quantified in terms of logical relations between beliefs. I argue that Bayesian networks are ideal tools for representing beliefs and their logical interconnections, facilitating belief evaluation and revision. Bayesian networks can be daunting for beginners, but new methods and technologies have the potential to make their application feasible for non-experts. AI technologies, in particular, have the potential to support or automate several steps in the construction and updating of Bayesian networks for reasoning in an explainable way. One of these steps consists of relating empirical evidence to theoretical and decision-relevant propositions. The result of using these methods and technologies would be an AI-powered inference engine we can query to see the rational support, empirical or otherwise, of key premises to arguments that bear on important practical decisions. With these technological innovations, decision support systems based on Bayesian networks may represent belief structures and improve our understanding, judgement, and decision-making.

CN’s comments

I thought this presented a potentially cool tool that I might be excited to see someone work on. I also really appreciate that the author already implemented proto-types and did experiments with them. Very cool. That said, I am sceptical that this tool will ultimately have a big impact (but would love for reality to convince me otherwise.)

Wise AI support for government decision-making — Ashwin Acharya and Michaelah Gertz-Billingsley

Summary

Governments are behind the curve on adopting advanced AI, in part due to concerns about its reliability. Suppose that developers gradually solve the issues that make AIs poor advisors — issues like confabulation, systemic bias, misrepresentation of their own thought processes, and limited general reasoning skills. What would make smart AI systems a good or poor fit as government advisors?

We recommend the development of wise decision support systems that answer the questions people _should_ have asked, not just the ones they did.

We recommend that government-advising institutions (e.g. contracting orgs like RAND) begin developing such systems, scaling them up over time as they acquire data and AI becomes more capable. Initially, AI systems might mostly summarize and clarify the views of human experts; later, they may guide and reframe conversations; eventually, they may largely _constitute_ the expert conversation that informs government decisionmaking on a topic.

CN’s comments

I thought this post highlights a very important area, decision-making in governments, and argues for something that I think is clearly true: Governments should implement (cheap) tools to improve their belief-formation and decision-making. I also liked that they discussed the availability of data. That said, I didn’t think the text offered either very novel ideas or a concrete enough proposal to make it meaningfully easier for groups to implement the proposal.

Machines and moral judgement — Jacob Sparks

Summary

Creating AI that is both generally intelligent and good requires building machines capable of making moral judgments, a task that presents philosophical and technical difficulties.

Current AI systems and much of what people imagine when they talk about “moral machines” may make morally significant or morally correct decisions, but they lack true moral reasoning.

Moral reasoning allows for reflective distance from one’s motivations and from any description under which one might act. It seems to challenge the distinction between beliefs and desires and to sit uncomfortably with a naturalistic worldview.

While there are promising tools in reinforcement learning for building agents that make moral judgments, we need more work on the philosophical puzzles raised by moral judgment and to rethink core RL concepts like action, reward, value and uncertainty.

CN’s comments

I thought the discussion of the peculiarities of morality and hence potential challenges with implementing it in AI systems was well-done if not novel. That said, I really strongly disagree with its last section “But How” where the author discusses how particular AI training techniques could help and thought that was by far the weakest part of the essay. In particular, I thought the connection the author draws between uncertainty and moral reflection is dubious and disagree with the reasons to conclude that RL is more promising than supervised finetuning. I think the text could have been improved with simple changes to the framing. (E.g. “Supervised finetuning has these flaws. Perhaps RL is better? Here are some reasons to think so. But, unfortunately, RL also has some flaws. So, maybe not.”) [Editor’s note: The framing was adjusted in response to this feedback.]

The purpose of philosophical AI will be: to orient ourselves in thinking — Maximilian Noichl

Summary

In this essay I will suggest a lower bound for the impact that artificial intelligence systems can have on the automation of philosophy. Specifically I will argue that while there is reasonable skepticism about whether LLM-based-systems that are sufficiently similar to the best ones available right now, will, in the short to medium term, be able to independently produce philosophy in a level of quality and creativity makes them interesting to us, they are clearly able to solve medium complexity language tasks in a way that makes them useful to structure and integrate the contemporary philosophical landscape, allowing for novel and interesting ways to orient ourselves in thinking.

CN’s comments

I really liked that this entry just went ahead and actually tried to implement a proto-type for a tool that can help with automating Philosophy. I also think the type of tool they proto-typed could become really useful. (In a way, it’s a first implementation of the improvements to organising academic literature that another entry proposed.)

BS’s comments

In a pilot study, the author of this entry used automation tools to map some literature on the philosophy of AI. I was happy to see an actual instance of automating something philosophical, that the instance was of a sort that could plausibly be useful if well-implemented at scale, and that the instance could be seen as a sort of proof of concept of a proposal in another entry (“Philosophy's Digital Future”) to use AI to map philosophical literature.

Cross-context deduction: on the capability necessary for LLM-philosophers — Rio Popper and Clem von Stengel

Summary

In this paper, we define a capability we call ‘cross-context deduction’, and we argue that cross-context deduction is required for large language models (LLMs) to be able to do philosophy well. First, we parse out several related conceptions of inferential reasoning, including cross-context deduction. Then, we argue that cross-context deduction is likely to be the most difficult of these reasoning capabilities for language models and that it is particularly useful in philosophy. Finally, we suggest benchmarks to evaluate cross-context deduction in LLMs and possible training regimes that might improve performance on tasks involving cross-context deduction. Overall, this paper takes an initial step towards discerning the best strategy to scalably employ LLMs to do philosophy.

CN’s comments

I really enjoyed the entry from an ML perspective and also thought it was very easy to read. I also really like the general approach of “Take some fairly well defined types of capabilities that we can differentiate in the context of ML, think about which ones are the kinds of capabilities we want to differentially improve, and suggest promising ways of doing so based on the existing ML literature.” That said, I’m unfortunately very unsure that boosting what the authors call cross-context deduction* would in fact be a good idea. While I thought their discussion of cross-context deduction* and Philosophy was really interesting, my current guess is that if we succeeded at boosting cross-context deduction* this would be net bad, i.e. net would net advance dangerous capabilities more than is justified by the boost to Philosophy.

(*We had some discussion about whether they defined deduction correctly, but I think they successfully managed to point at a for practical purposes distinct-enough kind of capability.)

DM’s comments

I have three main concerns about this paper. 

First, it’s unclear what the authors mean by “deduction”: they introduce it both as reasoning from universals to particulars (e.g. on pgs 2 and 6), and, more commonly, as reasoning in which premises “necessarily imply” the conclusion. But these are very different things. “Socrates is human, therefore something is human” is a deductive inference (in the latter sense) that moves from particular to universal. Conversely, we infer from universal to particular in an uncertain way all the time, eg. applying statistics to a case. 

In what follows, I’ll assume that by “deduction” the authors really mean “the type of reasoning that preserves truth with certainty”. (Another alternative is that they mean “inference by way of the application of logical laws”, but in that case the category depends on the choice of a logical system.) But why think of this as a type of reasoning, beyond the fact that there’s an old tradition of doing so? Maybe instead we should think of this as a specific epistemic feature that happens to attend some instances of various types of reasoning, individuated by something like an underlying cognitive procedure. That is, perhaps a better taxonomy will distinguish between semantic reasoning, statistical reasoning, visual reasoning, conditionalization, etc., each of which allows for various degrees of epistemic distance between premises and conclusion, including--in the limit case--the preservation of certainty. 

My second concern is that no convincing case is made that there is anything especially deductive about philosophical reasoning, or that there is anything especially philosophical about deductive reasoning (cross-context or otherwise), in either of the senses of “deduction” conflated in the paper. Indeed, paradigmatically philosophical arguments can usually be presented either in a deductive form or an inductive one. The deductive version will have more tendentious premises of which we are uncertain; the inductive version will have more certain premises but won’t entail the conclusion. 

Third, the paper uses “context” in “cross-context” in two very different ways: applied to humans it seems to mean roughly a domain of inquiry or a source of information, but applied to LLMs it seems to literally mean their “context window” (the input text supplied by the user). But there is no clear relationship between these two things aside from the fact that the word “context” is used for both of them. So even if philosophical reasoning by humans is often “cross-context” in the former sense, it doesn’t follow that there’s any special relationship to “cross-context” reasoning by LLMs in the latter sense.  

Evolutionary perspectives on AI values — Maria Avramidou

Summary

Addressing the question of value drift is crucial for deciding the extent to which AI should be automated. AI has the potential to develop capabilities that, if misaligned with human values, could present significant risks to humanity. The decision to automate AI depends heavily on our ability to trust that these systems will reliably maintain this alignment. Evolutionary theory provides a valuable framework for predicting how AI values might change, highlighting the role of environmental incentives and selection pressures in shaping AI behavior. Current discussions often focus on the inevitability of AI developing selfish values due to these pressures. I analyse these arguments and provide a case against these claims by drawing examples from evolution.

CN’s comments

This text, while somewhat off-topic in my view, highlights value drift as an important and neglected area. I think that’s valuable and it does a good job characterising it alongside what they call value specification and value monitoring (which I would loosely translate into outer and inner alignment.) I also think it’s reasonable to discuss evolution in this context. That said, I thought the text missed an opportunity to really analyse to which extent the evolutionary analogy is relevant in the AI context (and perhaps different stages in the AI context, e.g. training and post-deployment) and what other mechanics might be at play. The text failed to convince me that the evolutionary arguments it gave have bearing on AI and, in my view, unfortunately also doesn’t seem to try to do so.

BS’s comments

(I recused myself from judging this entry because I had worked with the author on related work.)

Tentatively against making AIs 'wise' — Oscar Delaney

Summary

• Wisdom is best conceived of as being more intuitive than carefully reasoned. This is helpful in order to distinguish ‘wisdom’ from ‘rationality’ or ‘good thinking’.

• Intuitions, including wise intuitions, are easy to communicate but hard to justify, and need to be taken ‘on faith’.

• It is very important that early AGIs are transparent and that their justifications for any actions they propose can be checked by a human.

• Therefore, we should prefer to train smart, careful-reasoning AIs rather than inscrutable wisdom-nugget dispensing AIs.

• Arguably I am unreasonably shifting the goalposts of the essay competition. The more positive framing is that I am “noticing that an old ontology was baking in some problematic assumptions about what was going on” and therefore I am actually being wise!

CN’s comments

I liked that the entry questions the premise of the competition. I also think it makes a reasonable point that inspires some thinking and in a concise and clear way. A more complete version of this text would have, for example, tried to reason about what it is about wisdom that the competition hosts find appealing and whether there is some wisdom-related property that would be beneficial to promote in AI or otherwise addressed a potential trade-off between the author’s concern and wise AI. (Or otherwise tried to turn the competition premise into its most interesting version.)

Concluding thoughts

We found it fascinating and informative getting to read the competition entries and then hear other people’s perspectives on them. We hope that many readers may find the same. The greatest success of this competition will be if it sparks further thought and engagement, and helps these nascent fields to find their feet a little earlier than they might otherwise have done.

Here are some closing thoughts from our judges:

BS’s thoughts

Views I had before reading contest entries:

• I thought it’d probably be extremely difficult to make big-picture theoretical insights about how to automate philosophy.

• I was mildly optimistic about the prospects for narrower theoretical insights about automating philosophy and about non-theoretical approaches to advancing the automation of philosophy in desirable ways.

• I was somewhat skeptical about the fruitfulness of automating wisdom as a research direction.

How reading contest entries affected my views:

• Reading entries reinforced my confidence in the extreme difficulty of making big-picture theoretical insights about how to automate philosophy.

• Reading entries made me more optimistic about the prospects for narrower theoretical insights about automating philosophy and about non-theoretical approaches to advancing the automation of philosophy in desirable ways.

• Reading entries made me think that there’s more fruitful theoretical work to be done on wisdom than I’d expected and that the automation of wisdom is a research avenue at least worth pursuing more on the current margin. (Currently, I take there to be very little work on this.)

Some takes on future work in these areas:

• I’d be excited for senior philosophers who have worked in multiple areas and on philosophical methodology to do big-picture, foundational work on automating philosophy in collaboration with AI researchers. (Prominent philosophers who fit the sort of profile I have in mind include David Chalmers, Alan Hájek, and Timothy Williamson.) In contrast, I’d be more excited about non-philosophers and early-career philosophers instead working on aspects of the topic that are narrower and/or more applied.

• I think there’s a lot of progress to be made in automating philosophy in desirable ways that doesn’t require theoretical breakthroughs. So I think it’d be good for more people to do things like:

  • find ways to facilitate philosophical reasoning with existing tools of automation,

  • develop datasets and tests for philosophical reasoning in AI systems,

  • find ways to use automation to improve information flow throughout the field of academic philosophy, 

  • empirical work on existing AI systems that tests hypotheses about those systems’ philosophical reasoning abilities, and

  • empirical work on existing AI systems that provides compelling, toy illustrations philosophical failure modes that could lead to catastrophes in powerful, future AI systems.

• I’d  like to see work in the genre of Leong’s An Overview of “Obvious” Approaches to Training Wise AI Advisors: that is, work that lays out some approaches to automating wisdom along with their pros and cons. I’d also like to see some work that implements particular approaches to automating wisdom. I’m not confident that such implementations would be useful, but I think trying them would at least provide evidence about the extent to which this direction is worth pursuing.

DM’s thoughts

Reading and thinking about these submissions, it strikes me that there are two kinds of goals for which one might want to try to automate philosophy and/or wisdom. One is the goal of helping to improve human thinking: perhaps AIs can guide us to understand our values better, help us achieve new philosophical insights, or help us reason better about how to build better institutions or future AIs. Importantly, a system that supports this goal wouldn’t need to be wise or philosophical itself: consider, for example, the AI that Yetter Chapell envisions, which simply provides a map of philosophical contributions, or the systems described by Acharya and Gertz-Billingsley, which support government decision-making. I was impressed by the variety and practicality of the suggestions for making progress towards the goal of improving human reasoning. 

A very different goal is that of imbuing the AIs themselves with wisdom, and/or having them arrive at philosophical conclusions on our behalf. Having read these submissions, I am both more optimistic that this can be achieved, and hesitant about whether we want to achieve it. Consider, for example, Delaney’s point that many features we associate with wisdom are really not the sort of features with which we’d want to imbue AIs, at least at first. (Laine likewise places “illegibility” on the “wisdom” side of the “intelligence/wisdom” divide.) So perhaps wisdom isn’t the right concept to focus on, after all. Likewise, if an AI is trained to be very capable at philosophical reasoning, we will probably want to inspect its whole chain of reasoning rather than trust its philosophical hunches. And the more legible we can make such a system, the more its output will actually function simultaneously to improve human reasoning. 

As much as I would love to see more progress made in philosophy generally, I do think the most pressing problem for which we need the help of AI is the problem of alignment itself. Here my guess is that systems aimed at supporting human reasoning could be quite helpful; but for the reasons just given, I’m less optimistic that it would be very useful to aim making AIs “wise” or even good at philosophy per se, as a stepping stone towards alignment. One kind of exception might be systems trained to reason carefully (but also legibly!) about human values, in a way that might ultimately help us design systems to reflect them. 

AS’s thoughts

My reaction after reading the submissions: "Wow, this area really needs a lot of work". The gap between what's there and what seems possible feels large.

Ideally work in this area:

• Advances our understanding of the relevant concepts and builds up new ones as needed

• Follows good epistemic practices and sound reasoning

• Is concrete and connected to experiments, observations, or implementations

I'd say the essays have about 1-1.5 of these three features on average (and I'm grateful that they do). They don't all have the same such features, so this doesn't seem to be an intrinsic limit of the domain.

I’m pretty sure you could take any one of the essays, score it along the three dimensions, and then develop it further along the dimensions where it's lacking.

This means that there is already one clear recipe for further work--significantly improve what's there along one dimension, turning it into something quite different in the process. Beyond this basic recipe, there's comparing, combining, and systematizing ideas across different essays. And finally, the original competition announcement poses excellent questions that none of the essays seem to directly address, especially in the Thinking ahead and Ecosystems sections.

As part of my work on Elicit, I regularly ask myself "What kind of reasoning is most worth advancing, and how?", and I know other founders whose roadmaps are open to arguments & evidence. Of course, good thinking on automating wisdom would likely have many diffuse positive effects, but I wanted to flag this one path to impact that feels especially real to me, and makes me personally very excited to see more work in this area.

Acknowledgements

Thanks to AI Impacts for hosting this competition, and to the donors who allowed it to happen. And of course, a big thanks to the people who took up the challenge and wrote entries for the competition — this would all have been nothing without them.

The results were very confusing, so we ended up thinking more about this than initially intended and running four iterations total. This is still a small and scrappy poll to satisfy our own understanding, and doesn’t involve careful analysis or error checking. But I’d like to share a few interesting things we found. Perhaps someone else wants to look at our data more carefully, or run more careful surveys about parts of it. 

In total we surveyed around 570 people across 4 different polls, with 500 in the main one. The basic structure was:

1. p(doom): “If humanity develops very advanced AI technology, how likely do you think it is that this causes humanity to go extinct or be substantially disempowered?” Responses had to be given in a text box, a slider, or with buttons showing ranges

2. (Present them with one of eleven arguments, one a ‘control’)

3. “Do you understand this argument?”

4. “What did you think of this argument?”

5. “How compelling did you find this argument, on a scale of 1-5?”

6. p(doom) again

7. Do you have any further thoughts about this that you'd like to share?

Interesting things:

• In the first survey, participants were much more likely to move their probabilities downward than upward, often while saying they found the argument fairly compelling. This is a big part of what initially confused us. We now think this is because each argument had counterarguments listed under it. Evidence in support of this: in the second and fourth rounds we cut the counterarguments and probabilities went overall upward. When included, three times as many participants moved their probabilities downward as upward (21 vs 7, with 12 unmoved). 

• In the big round (without counterarguments), arguments pushed people upward slightly more: 20% move upward and 15% move downward overall (and 65% say the same). On average, p(doom) increased by about 1.3% (for non-control arguments, treating button inputs as something like the geometric mean of their ranges).

• But the input type seemed to make a big difference to how people moved!  

  

  It makes sense to me that people move a lot more in both directions with a slider, because it’s hard to hit the same number again if you don’t remember it. It’s surprising to me that they moved with similar frequency with buttons and open response, because the buttons covered relatively chunky ranges (e.g. 5-25%) so need larger shifts to be caught.

• Input type also made a big difference to the probabilities people gave to doom before seeing any arguments. People seem to give substantially lower answers when presented with buttons (Nathan proposes this is because there was was a <1% and 1-5% button, so it made lower probabilities more salient/ “socially acceptable”, and I agree):

  

• Overall, P(doom) numbers were fairly high: 24% average, 11% median.

  

• We added a ‘control argument’. We presented this as “Here is an argument that advanced AI technology might threaten humanity:” like the others, but it just argued that AI might substantially contribute to music production:

  

  
  This was the third worst argument in terms of prompting upward probability motion, but the third best in terms of being “compelling”. Overall it looked a lot like other arguments, so that’s a bit of a blow to the model where e.g. we can communicate somewhat adequately,  ‘arguments’ are more compelling than random noise, and this can be recognized by the public.

• In general, combinations of claims about compellingness, changes in probabilities and write-in answers were frequently hard to make sense of, especially if you treat the probability changes as meaningful rather than as random noise. For instance, the participant who rated an argument 4/5 compellingness, yet reduced their P(doom) from 26% to 0%, and said “The above argument is completely based on probability of AI's effects on humanity near future, some feels that it could be turn into negative way but most people feels that it is going to a good aspect for future technology.” My sense is that this was more true in the first round than the fourth, so perhaps the counterarguments are doing something there.

• This is how the different arguments fared:

  

• All arguments were on mean between 2 and 3 compelling, including the control argument

• The arguments we considered worst did roughly worst, in terms of probability change (black  boxes, large impacts, multiagent dynamics, control)

• The argument from expert opinion was the very worst though, which is interesting to me because it seems like the one that people are constantly pointing at in public, in trying to justify concerns about x-risk. 

• The top arguments for increasing P(doom) were the ones about normal human processes getting out of hand (human non-alignment, catastrophic tools, speed) then the ones about bad new agents came below (second species, competent non-aligned agents, inferiority). Compellingness looks related but not closely so. 

• We both found the experience of quickly polling the public enlivening.

If you wish to look at the arguments in more detail, they are here. If you want to analyze the data yourself, or read everyone’s write-in responses, it’s here. If you see any errors, please let us know. 

We’ve made a list of arguments1 that AI poses an existential risk to humanity. We’d love to hear how you feel about them in the comments and polls. 

Competent non-aligned agents

Summary:

1. Humans will build AI systems that are 'agents', i.e. they will autonomously pursue goals

2. Humans won’t figure out how to make systems with goals that are compatible with human welfare and realizing human values

3. Such systems will be built or selected to be highly competent, and so gain the power to achieve their goals

4. Thus the future will be primarily controlled by AIs, who will direct it in ways that are at odds with long-run human welfare or the realization of human values

Selected counterarguments:

• It is unclear that AI will tend to have goals that are bad for humans

• There are many forms of power. It is unclear that a competence advantage will ultimately trump all others in time

• This argument also appears to apply to human groups such as corporations, so we need an explanation of why those are not an existential risk

People who have favorably discussed2 this argument (specific quotes here): Paul Christiano (2021), Ajeya Cotra (2023), Eliezer Yudkowsky (2024), Nick Bostrom (20143).

See also: Full wiki page on the competent non-aligned agents argument

Second species argument

Summary:

1. Human dominance over other animal species is primarily due to humans having superior cognitive and coordination abilities

2. Therefore if another 'species' appears with abilities superior to those of humans, that species will become dominant over humans in the same way

3. AI will essentially be a 'species' with superior abilities to humans

4. Therefore AI will dominate humans

Selected counterarguments:

• Human dominance over other species is plausibly not due to the cognitive abilities of individual humans, but rather because of human ability to communicate and store information through culture and artifacts

• Intelligence in animals doesn't appear to generally relate to dominance. For instance, elephants are much more intelligent than beetles, and it is not clear that elephants have dominated beetles

• Differences in capabilities don't necessarily lead to extinction. In the modern world, more powerful countries arguably control less powerful countries, but they do not wipe them out and most colonized countries have eventually gained independence

People who have favorably discussed this argument (specific quotes here): Joe Carlsmith (2024), Richard Ngo (2020), Stuart Russell (20204), Nick Bostrom (2015).

See also: Full wiki page on the second species argument

Loss of control via inferiority

Summary:

1. AI systems will become much more competent than humans at decision-making

2. Thus most decisions will probably be allocated to AI systems

3. If AI systems make most decisions, humans will lose control of the future

4. If humans have no control of the future, the future will probably be bad for humans

Selected counterarguments:

• Humans do not generally seem to become disempowered by possession of software that is far superior to them, even if it makes many 'decisions' in the process of carrying out their will

• In the same way that humans avoid being overpowered by companies, even though companies are more competent than individual humans, humans can track AI trustworthiness and have AI systems compete for them as users. This might substantially mitigate untrustworthy AI behavior

People who have favorably discussed this argument (specific quotes here): Paul Christiano (2014), Ajeya Cotra (2023), Richard Ngo (2024).

See also: Full wiki page on loss of control via inferiority 

Loss of control via speed

Summary:

1. Advances in AI will produce very rapid changes, in available AI technology, other technologies, and society

2. Faster changes reduce the ability for humans to exert meaningful control over events, because they need time to make non-random choices

3. The pace of relevant events could become so fast as to allow for negligible relevant human choice

4. If humans are not ongoingly involved in choosing the future, the future is likely to be bad by human lights

Selected counterarguments:

• The pace at which humans can participate is not fixed. AI technologies will likely speed up processes for human participation.

• It is not clear that advances in AI will produce very rapid changes.

People who have favorably discussed this argument (specific quotes here): Joe Carlsmith (2021).

See also: Full wiki page on loss of control via speed

Human non-alignment

Summary:

1. People who broadly agree on good outcomes within the current world may, given much more power, choose outcomes that others would consider catastrophic

2. AI may empower some humans or human groups to bring about futures closer to what they would choose

3. From 1, that may be catastrophic according to the values of most other humans

Selected counterarguments:

• Human values might be reasonably similar (possibly after extensive reflection)

• This argument applies to anything that empowers humans. So it fails to show that AI is unusually dangerous among desirable technologies and efforts

People who have favorably discussed this argument (specific quotes here): Joe Carlsmith (2024), Katja Grace (2022), Scott Alexander (2018).

See also: Full wiki page on the human non-alignment argument

Catastrophic tools

Summary:

1. There appear to be non-AI technologies that would pose a risk to humanity if developed

2. AI will markedly increase the speed of development of harmful non-AI technologies

3. AI will markedly increase the breadth of access to harmful non-AI technologies

4. Therefore AI development poses an existential risk to humanity

Selected counterarguments:

• It is not clear that developing a potentially catastrophic technology makes its deployment highly likely

• New technologies that are sufficiently catastrophic to pose an extinction risk may not be feasible soon, even with relatively advanced AI

People who have favorably discussed this argument (specific quotes here): Dario Amodei (2023), Holden Karnofsky (2016), Yoshua Bengio (2024).

See also: Full wiki page on the catastrophic tools argument

Powerful black boxes

Summary:

1. So far, humans have developed technology largely through understanding relevant mechanisms

2. AI systems developed in 2024 are created via repeatedly modifying random systems in the direction of desired behaviors, rather than being manually built, so the mechanisms the systems themselves ultimately use are not understood by human developers

3. Systems whose mechanisms are not understood are more likely to produce undesired consequences than well-understood systems

4. If such systems are powerful, then the scale of undesired consequences may be catastrophic

Selected counterarguments:

• It is not clear that developing technology without understanding mechanisms is so rare. We have historically incorporated many biological products into technology, and improved them, without deep understanding of all involved mechanisms

• Even if this makes AI more likely to be dangerous, that doesn't mean the harms are likely to be large enough to threaten humanity

See also: Full wiki page on the powerful black boxes argument

Multi-agent dynamics

Summary:

1. Competition can produce outcomes undesirable to all parties, through selection pressure for the success of any behavior that survives well, or through high stakes situations where well-meaning actors' best strategies are risky to all (as with nuclear weapons in the 20th Century)

2. AI will increase the intensity of relevant competitions

Selected counterarguments:

• It's not clear what direction AI will have on the large number of competitive situations in the world

People who have favorably discussed this argument (specific quotes here): Robin Hanson (2001)

See also: Full wiki page on the multi-agent dynamics argument

Large impacts

Summary:

1. AI development will have very large impacts, relative to the scale of human society

2. Large impacts generally raise the chance of large risks

Selected counterarguments:

• That AI will have large impacts is a vague claim, so it is hard to tell if it is relevantly true. For instance, 'AI' is a large bundle of technologies, so it might be expected to have large impacts. Many other large bundles of things will have 'large' impacts, for instance the worldwide continued production of electricity, relative to its ceasing. However we do not consider electricity producers to pose an existential risk for this reason

• Minor changes frequently have large impacts on the world according to (e.g. the butterfly effect). By this reasoning, perhaps we should never leave the house

People who have favorably discussed this argument (specific quotes here): Richard Ngo (2019) 

See also: Full wiki page on the large impacts argument

Expert opinion

Summary:

• The people best placed to judge the extent of existential risk from AI are AI researchers, forecasting experts, experts on AI risk, relevant social scientists, and some others

• Median members of these groups frequently put substantial credence (e.g. 0.4% to 5%) on human extinction or similar disempowerment from AI

Selected counterarguments:

• Most of these groups do not have demonstrated skill at forecasting, and to our knowledge none have demonstrated skill at forecasting speculative events more than 5 years into the future

This is a snapshot of an AI Impacts wiki page. For an up to date version, see there.

https://scholar.harvard.edu/files/shleifer/files/princes_merchants.pdf.

Summary: Freer societies have faster economic growth. 

One of the oldest themes in economics is the incompatibility of despotism and development. Economies in which security of property is lacking – because of either the possibility of arrest, ruin, or execution at the command of the ruling prince or the possibility of ruinous taxation – should experience relative stagnation. By contrast, economies in which property is secure – either because of strong constitutional restrictions on the prince or because the ruling elite is made up of merchants rather than princes – should prosper and grow.

Much of the work investigating the relationship between despotism and economic growth focuses on modern economies. The quantity of data available for modern economies is much greater than for earlier economies. 

However, there are some limitations from focusing on only modern economies. Long time series are often impossible to compare, because most data is only available for maybe a century. The development trajectories of different countries, both economically and politically, are not independent from each other, so there are fewer independent data points than it might initially seem. 

Modern economies might also have different development patterns which cannot be easily replicated elsewhere. Having an abundance of certain natural resources, like oil, can cause a modern economy to grow regardless of its institutions. This often isn’t a useful observation: `Have large hydrocarbon reserves’ is not actionable advice. The development strategies useful for catching up to the technological frontier might be different from the development strategies useful for societies at the technological frontier.

Looking at earlier economies can allow us to avoid some of these problems. The time series can be much longer. The economic and political developments of different countries are less likely to be similar to each other over the entire time frame. Pre-industrial economies were all centered around food production, so there is less opportunity to build an economy based on resource extraction. The technological differences between countries (at least within Europe) were smaller, so catch-up growth is less relevant. Pre-industrial economies provide a set of examples with different limitations than modern economies.

De Long & Shleifer compare different regions of Europe between 1050 and 1800. The growth of cities with a population of at least 30,000 acts as a proxy for economic growth. They divide the time frame into 6 periods, they divide central and western Europe into 9 regions, and they categorize governments into 8 types. All of these categories are listed in Table 1.

The main challenge with focusing on pre-industrial economies is that high quality data is much less available.  Urban population is an imperfect proxy for economic development. There are several datasets that estimate the population of European cities over this time period, and they do not always agree with each other.1 There is also some arbitrariness in deciding which system of government was dominant in a region for more than a century. However, the effect size is large enough to be robust to these uncertainties.

This paper was written in 1993 and has over 1,000 citations. I have not looked at how more recent literature has responded to or developed since it was published. It seems to be a classic in its subfield.

There is a strong correlation between how free a society is and its urban growth (Figure 1). 

We find that, on average, for each century that such a region is free of government by an absolute prince, its total population living in cities of 30,000 or more inhabitants grew by 120,000, relative to a century of absolutist rule. This difference is larger than the average growth rate of urban populations in European regions between 1000 and 1800. In a purely statistical sense, therefore, the association between absolutism and slow city growth can more than account for why some western European regions had relatively low rates of urbanization in 1800, while others had flourishing cities and abundant commerce.

Controlling for the region, the time period,2 and whether the city is an imperial capital makes the association even stronger.

Several of the particular examples make it seem likely that this is causal, rather than merely a correlation.

In 1050, southern and northern Italy were similarly urbanized. Over the course of the next century, their political systems diverged. Southern Italy was consolidated under an unusually centralized Norman kingdom. The cities of northern Italy became increasingly independent from the Holy Roman Empire through the Investiture Controversy and Lombard League.3 The cities of northern Italy outpaced the growth of the cities of southern Italy over the next few centuries. Around 1500, the kingdoms of France and Spain reduced the independence of most of the Italian city-republics in the Italian Wars, and the urbanization of northern Italy slowed.

Prior to about 1500, the Low Countries were ruled by the Duke of Burgundy, a weak feudal government with checks on its power. These regions were then transferred to the Habsburgs, an increasingly bureaucratic absolute monarchy. The northern part of the Low Countries rebelled and formed the Dutch Republic. In 1500, the urbanization of the southern (Belgium) and northern (Netherlands) parts of the Low Countries were similar. They diverged dramatically afterwards. Urbanization in the Netherlands increased and they became a global economic power, while Belgium stagnated.

For most of the time between 1050 and 1800, England was an unusually centralized and bureaucratic kingdom. Around 1650, the English Civil War and Glorious Revolution established the supremacy of Parliament on the most important issues. England had been economically underdeveloped prior to 1650, but afterwards saw rapid economic growth culminating in the Industrial Revolution.

While the existence of the correlation is not surprising, it is surprising how clear the effect is. Modern economies do not have this strong of a correlation between freedom and economic growth. With longer time frames and with fewer opportunities for technological catch-up or economies based on resource extraction, the long-term structural differences are more apparent. Medieval and early modern free societies had more economic development than their less free counterparts.

From the perspective of the people alive at the time, or of the long-term growth of the economy, princely success is economic failure. For the people of southern Italy, the creation of the d’Hauteville regno was no blessing; for the people of Belgium, their incorporation into the Habsburg Empire was no benefit; for the people of Iberia, the marriage of Ferdinand and Isabella was no cause for rejoicing. The rise of an absolutist government and the establishment of princely authority are, from a perspective that values economic growth, events to be mourned and not celebrated.

Subscribe now

https://www.pnas.org/doi/pdf/10.1073/pnas.1913678117.

Summary: Numerically expressing uncertainty when talking to the public is fine. It causes people to be less confident in the number itself (as it should), but does not cause people to lose trust in the source of that number.

Uncertainty is inherent to our knowledge about the state of the world yet often not communicated alongside scientific facts and numbers. In the “posttruth” era where facts are increasingly contested, a common assumption is that communicating uncertainty will reduce public trust. However, a lack of systematic research makes it difficult to evaluate such claims.

Within many specialized communities, there are norms which encourage people to state numerical uncertainty when reporting a number. This is not often done when speaking to the public. The public might not understand what the uncertainty means, or they might treat it as an admission of failure. Journalistic norms typically do not communicate the uncertainty.

But are these concerns actually justified? This can be checked empirically.  Just because a potential bias is conceivable does not imply that it is a significant problem for many people. This paper does the work of actually checking if these concerns are valid.

Van der Bles et al. ran five surveys in the UK with a total n = 5,780. A brief description of their methods can be found in the appendix below.

Respondents’ trust in the numbers varied with political ideology, but how they reacted to the uncertainty did not.

People were told the number either without mentioning uncertainty (as a control), with a numerical range, or with a verbal statement that uncertainty exists for these numbers. The study did not investigate stating p-values for beliefs. Exact statements used in the survey can be seen in Table 1, in the appendix.

The best summary of their data is in their Figure 5, which presents results from surveys 1-4. The fifth survey had smaller effect sizes, so none of the shifts in trust were significant.

Expressing uncertainty made it more likely that people perceived uncertainty in the number (A). This is good. When the numbers are uncertain, science communicators should want people to believe that they are uncertain. Interestingly, verbally reminding people of uncertainty resulted in higher perceived uncertainty than numerically stating the numerical range, which could mean that people are overestimating the uncertainty when verbally reminded of it.

The surveys distinguished between trust in the number itself (B) and trust in the source (C). Numerically expressing uncertainty resulted in a small decrease in the trust of that number. Verbally expressing uncertainty resulted in a larger decrease in the trust of that number. Numerically expressing uncertainty resulted in no significant change in the trust of the source. Verbally expressing uncertainty resulted in a small decrease in the trust of the source.

The consequences of expressing numerical uncertainty are what I would have hoped: people trust the number a bit less than if they hadn’t thought about uncertainty at all, but don’t think that this reflects badly on the source of the information.

Centuries of human thinking about uncertainty among many leaders, journalists, scientists, and policymakers boil down to a simple and powerful intuition: “No one likes uncertainty.” It is therefore often assumed that communicating uncertainty transparently will decrease public trust in science. In this program of research, we set out to investigate whether such claims have any empirical basis.

The answer is mostly no. Good epistemic practice is not bad journalistic practice. When you give people numerical estimates of uncertainty of a number, they respond the way they should. The perceived confidence in the number itself goes down, while the trust in the source does not. Verbally reminding people of uncertainty seems like a worse practice: it causes people to distrust the source of information and seems to cause them to overestimate the uncertainty in the number. Expressing no uncertainty seems to make people overconfident in the number reported.

It is better to use good epistemics when talking to the public than it is to try to correct for their bad epistemics with compromised epistemics of your own.

The high degree of consistency in our results, across topics, magnitudes of uncertainty, and communication formats suggest that people “can handle the truth.”

Subscribe now

Appendix: Survey Methods

Of the five surveys involved in this paper, the first three each had about 1,000 participants in the UK recruited using the platform Prolific, and were paid £1.20 to complete a 2 minute survey. The fourth survey was a preregistered replication with 1,050 adults in the UK recruited using the platform Qualtrics Panel. The fifth survey was a field experiment done with BBC News. When BBC News reported new labor market statistics on October 15, 2019, they ran three different versions of the article, and included a link to the survey in the article. There were 1,700 people who completed this survey. 

Each survey presented readers with a measured number of some statistic: the number of unemployed people in the UK, the net number of migrants between the EU and UK, the amount the Earth’s average global temperature increased between 1880 & 2012, and the number of tigers in India. Some of these are more partisan issues than others in the UK, and the error bars are different sizes relative to the size of the number. 

Table 1 shows some of the exact statements used in the 3rd and 4th surveys.

The participants were somewhat more educated and more liberal than the general public in all five of the surveys, and the first three surveys were somewhat younger and more female.

This post has more of my personal opinions than previous posts or the report itself.

Other movements should try to avoid becoming as partisan as the environmental movement. Partisanship did not make environmentalism more popular, it made legislation more difficult to pass, and it resulted in fluctuating executive action. Looking at the history of environmentalism can give insight into what to avoid in order to stay bipartisan.

Partisanship was not inevitable. It occurred as the result of choices and alliances made by individual decision makers. If they had made different choices, environmentalism could have ended up being a bipartisan issue, like it was in the 1980s and is in some countries in Europe and democratic East Asia.

Environmentalists were not the only people making significant decisions here. Fossil fuel companies and conservative think tanks also had agency in the debate – and their choices were more blameworthy than the choices of environmentalists. Politicians choose who they do and do not want to ally with. My focus is on the environmental movement itself, because that is similar to what other activist groups are able to control.

I am more familiar with the history of the environmental movement than with most other social movements. The environmental movement is particularly interesting because it involves an important global issue that used to be broadly popular, but has since become very partisan and less effective at enacting policy in the United States. It nevertheless can be risky to over-update on a single case study. Much of the advice given here has support in the broader social movements literature, but the particulars are based on the history of one movement.

With those caveats aside, let’s look at what we can learn.

Here is a list of advice I have gleaned from this history:

1. Make political alliances with individuals and institutions in both political parties.
   
   This is the most important advice.
   
   Allying with the Democratic Party might have seemed like a natural choice at the time. Climate scientists might have already leaned left, and so found allying with Democrats to be more natural – although the evidence for this is weak. Al Gore was committed to their cause, and was rapidly building political influence: from Representative to Senator to Vice President, and almost to President.
   
   The mistake was not simultaneously pursuing alliances with rising Republicans as well. At the time, it would not have been too difficult to find some who were interested.
   
   Building relationships with both parties involves recruiting or persuading staffers for both Democratic and Republican congressmen and analysts for both conservative and liberal think tanks. Personal relationships with individuals and institutions often matter more than the implications of a fully consistent ideology.
   

2. Don’t give up on one side once partisanship starts to be established.
   
   I wouldn’t be surprised if some environmentalists in the late 1990s or 2000s thought that the issue was already partisan, so it didn’t matter that they were only working with one side. They were wrong. Partisanship could and did continue to get worse. Environmentalism is now one of the, if not the, most partisan issue in the country.
   
   In 1995, after Newt Gingrich had won control of the House of Representatives opposing the BTU tax, there was still only one conservative think tank that regularly promoted climate skepticism. Environmentalists might have been able to gain influence at other conservative think tanks to weaken the reframing efforts of fossil fuel companies.
   
   In 2006, Al Gore’s documentary An Inconvenient Truth did not change the opinions of the public overall, but did encourage a new generation of activists. He might have been able to reduce the partisan effect of the documentary by collaborating with a prominent Republican who supported climate policies, like Schwarzenegger or McCain.
   
   Ongoing decisions by environmentalists and their allies continued to reinforce the partisan divide.
   

3. Proposing flawed legislation, and losing the resulting legislative battle, seems quite bad.
   
   There were two key legislative defeats as environmentalism started to become partisan: the BTU tax in 1993 and the Kyoto Protocol in 1997.
   
   In both cases, the legislation seems poorly designed. The BTU tax focused on energy, not greenhouse gasses, with exemptions for favored industries. The Kyoto Protocol had already been rejected by the entire Senate.
   
   Unpopular legislation proposed by environmentalists and their allies made it easier for other politicians to rally against environmentalism.
   
   Drafting good legislation is important both to get what you actually want enacted and to not offer as many opportunities for others’ attacks.
   

4. Be cautious and intentional about mission creep.
   
   Mission creep is the gradual expansion of an institution’s or a movement’s goals beyond their original intention. For an advocacy group focusing on a complex issue, some mission creep is inevitable: as your understanding of the problems grows, there should be some changes to the goals you are pursuing to address these problems.
   
   Mission creep can also involve expanding your goals to include goals of your current political allies, even if they are not directly related to the original intention. This seems bad. Environmental organizations today promote liberal positions on many other policy issues and reliably endorse one political party.
   
   If the organizations in a movement endorse controversial positions aligned with one party, it should not be surprising if many people associate them with that party. Allowing mission creep makes it harder to build bipartisan coalitions. There are more people who agree with you on environmental issues than there are people who agree with you on environmental issues and abortion and the Israeli-Palestinian conflict and … .
   
   Your movement should try to avoid having public opinions on most issues and only focus on the issues central to your original intention.
   

5. Focusing on local issues makes it easier to form idiosyncratic partnerships that cut across party lines.
   
   In its first few decades, the modern environmental movement focused primarily on local concerns: air pollution in Los Angeles, the Cuyahoga River fire, the proposed Bodega Bay nuclear power plant, and proposed dams in the Grand Canyon. In the late 1980s, much of the attention of the environmental movement shifted towards climate change, an inherently global phenomenon. This does not reflect public opinion, which seems to be more concerned with local environmental issues than climate change.
   
   Local politics in the US is less partisan than national politics.1 It is often not obvious how the national parties would respond to specific local questions, so there are fewer elite cues to divide people into partisan camps. Different localities compete with each other for population and economic activity, and so respond to where people are choosing to live in addition to how people vote. For these reasons, local issues often involve idiosyncratic partnerships cutting across party lines.
   
   The environmental movement’s shift from local issues to one international issue made it easier for it to become consistently tied to one political party.
   
   There is some reason to have caution here. You do not want a particular local partnership to turn into an alliance that defines your movement. This feels like a solvable problem by not becoming too committed to local partnerships and managing mission creep well.
   

6. Getting messaging right seems hard.
   
   Both underselling and overselling your arguments seems like it could have bad results. Either of them seems like they could undermine public trust in your expertise.
   
   Explicitly stating numerical uncertainty to the public is fine, and does not cause people to trust you less.2 Telling policy makers both your politically plausible asks and your more ambitious hopes also seems fine.
   
   Failing to distinguish between empirical and normative claims could be effective in the short term: if people accept the validity of the empirical claims, conflating them with policy proposals can make it easier to get these policies enacted. It seems counterproductive in the longer term: if people do not accept your policy goals, it can also make them more dismissive of your empirical claims.
   
   I am of the opinion that you should use good epistemics when talking to the public or policy makers, rather than using bad epistemics to try to be more persuasive. Most subject matter experts are not also experts in public messaging, and so typically do not know how to use effective rhetoric and narrative-crafting. Being publicly revealed to have been dishonest to the public seems like it damages trust much more than using good epistemics in a not rhetorically optimal way. I would rather have a reputation as someone who trusts the public and policy makers to understand my key points, rather than as someone who looks down on their ability to understand what I’m worried about.
   
   It is unclear whether any one actor could have dramatically improved the messaging, or if that would have required an unrealistic amount of discipline within the movement. It was not hard for activists on either side to find climate scientists who were willing to confidently argue their position to the public.

To me, the AI safety movement feels sort of like environmentalism in the 1960s or climate change in the 1980s. The movement is still really young. Most of the public is still uncertain what to think about it.

Despite this uncertainty, a decent amount of the public seems to support the goals of the AI safety movement. Polls indicate that many people are skeptical that AI will have a positive impact on society, and that some amount of government regulation is broadly popular.3

This does not inherently imply that the AI safety movement will succeed if, for example, it proposes a ballot measure for the next election. The public is still more uncertain than supportive. The details of the proposal need to be proposed and promoted. Various leaders and groups may respond in unpredictable ways. Public opinion might look very different after a major political push than it did before. But I do think that these polls indicate that there is latent public support that the AI safety community could develop in support of its policy goals.

When trying to build this latent public support, it is important to cast as wide of a net as possible. Many different people might be interested in and willing to support the AI safety movement – including people who are culturally very different from the people who are currently working on AI safety. The movement should try to build relationships with as varied a group of people as possible. 

A broad bipartisan movement would be more effective at enacting policy than a movement closely allied to one political party.

Summary

Rising partisanship did not make environmentalism more popular or politically effective. Instead, it saw flat or falling overall public opinion, fewer major legislative achievements, and fluctuating executive actions.

Public Opinion

One hypothesis is that partisanship was useful, or even necessary, for an issue to become popular. Maybe journalists never would have covered the story if it did not involve an exciting partisan contest. The public then might have never realized that this is a thing they could care about.

The polling data do not support this hypothesis.

The clearest data from McCright et al.1 Over 70% of both parties, and both ideologies, supported more government spending on the environment in 1990. Then, over the next 20 years, Republicans’ support for environmental spending fell dramatically while Democrat support remained roughly constant. These polls show declining overall support as partisanship increased.

Figure 1: Percentages of Democrats and Republicans reporting that national spending on the environment is “Too Little,” 1974-2012. Reprinted from McCright et al. (2014).

The Gallup data is more ambiguous.2 It starts later, so it cannot show what was happening in the 1990s. For four different questions about global warming, Gallup shows roughly flat support from 2001-2021, although there have been fluctuations in the level of support.

Figure 2: Percentage of Americans who agree with four different statements about global warming. Reprinted from Gallup (2021).

Public opinion is not quite the same thing as public attention. Maybe partisanship increases how much people are talking about an issue, even if it has little impact on the support as measured in polls. As a proxy for public attention, I will use Google Book’s Ngram, which shows how frequently phrases appear in a corpus of English language books published each year.

For many environmental issues, there was a peak in the early 1990s. Public attention was growing prior to the issue becoming partisan and declining once the issue had become partisan. This might be because the environmental movement itself shifted focus away from these issues. ‘Pollution’ shows a similar peak around 1990, but also a larger peak in the early 1970s. ‘Climate change’ shows a somewhat different pattern: rapid growth before 1990, which then levels off until the mid-2000s, followed by resumed growth. It is possible to have increasing public attention while a topic is highly partisan, but for all of these environmental issues, public attention was flat or falling while partisanship was becoming established in the 1990s and early 2000s. Increasing partisanship does not seem to be a reliable way to attract public attention. 

Figure 3a: Frequency of the phrases ‘ozone,’ ‘deforestation,’ ‘endangered species,’ and ‘acid rain’ in books each year from 1945-2019. From Google Ngram Viewer.

Figure 3b: Frequency of the phrases ‘pollution,’ ‘air pollution,’ and ‘water pollution’ in books each year from 1945-2019. From Google Ngram Viewer.

Figure 3c: Frequency of the phrases ‘climate change,’ ‘global warming,’ and ‘climatic change’ in books each year from 1945-2019. From Google Ngram Viewer.

Republicans’ opposition to climate change does not seem to be a result of a lack of information. Two polls in swing states in 2011 indicated that Republicans and Democrats with less education, or who said they know little about climate change, have similar views. As education and knowledge increase, Democrats became more concerned about climate change, while Republicans became less concerned.3 This result is consistent with the overall trends: as time goes on, and people become more familiar with climate change, it exacerbates the partisan divide without increasing overall support.

Legislation

Many of the goals of environmentalism cannot be directly achieved by public opinion: they require new legislation.

To determine if partisanship has made it easier or harder to pass legislation, I investigated when pieces of major environmental legislation have been passed.

What counts as ‘major’ environmental legislation? 

The Congressional Research Service produced a summary of the “major statutes administered by the EPA” in 20134 and the Library of Congress has a research guide about “significant legislation governing environmental law and policy.”5 Any law mentioned in either of these is included.6 I categorize these laws as dealing with environmental impact statements, air pollution, water pollution, solid waste, toxic substances, or endangered species. All of these are domestic laws – treaties will be discussed below.

These laws often create the framework of U.S. environmental policy. Newer legislation might not need to create a new framework, and instead could be an amendment to the existing framework. In addition, some of these laws superseded earlier environmental legislation. To account for both of these possibilities, I turned to Wikipedia. The sidebar of the Wikipedia article for each of these laws has a list of “major amendments.” I include each of these as a piece of major environmental legislation, checking to make sure there is no double counting.7 I also read the History section to check for any major precursors. Each law has a well-developed article. I trust their judgment for what counts as a “major amendment” and what precursors should be mentioned more than a search and categorization I might do specifically for this report.

A graph of the number of pieces of major environmental legislation and amendments since 1945 is shown in Figure 4, binned into 4 year intervals corresponding to presidential terms.

Figure 4: The number of major pieces of domestic environmental legislation, including amendments and precursors to existing legislation, since 1945. Each four year bin corresponds to a presidential term.

There was clearly more environmental legislation passed in the 1960s-80s than in earlier or later decades. The modern environmental movement began in the 1960s and became legislatively successful within the decade. The environmental movement became partisan in the 1990s, and stopped being as capable of passing major legislation. 

Treaties show a similar, if sparser, pattern. During the 1970s and 1980s, the Senate ratified four international environmental treaties unanimously.8 The U.N. Framework Convention on Climate Change was also ratified using a division vote in 1992.9 Between 1989 and 2001, the United States signed four environmental treaties which the Senate refused to ratify, including the Kyoto Protocol.10 More recently, international environmental agreements have been structured so they do not require ratification from the Senate – like the Paris Climate Accords.11

Increasing partisanship has made it more difficult for the environmental movement to pass its legislative agenda.

Executive Actions

Environmental policy can also be enacted by the executive branch. When environmentalism is partisan, executive action fluctuates with broader political winds.12 Over the medium-to-long term, it is unlikely that one political party will consistently win elections. Having bipartisan support for an issue results in more stable and reliable executive policies.13

This can be most clearly seen in international agreements. The Clinton administration negotiated the Kyoto Protocol, the Bush administration declared that they would not implement it, the Obama administration negotiated the Paris Accords, the Trump administration withdrew from them, and the Biden administration rejoined them.14

Much of the implementation of environmental policy occurs in the executive branch, especially the EPA. While Republican administrations have not been effective at reducing the size or budget of the EPA,15 they have chosen leaders who have closer ties to fossil fuel companies than to the environmental movement. Many of the details of how to enact policy are small enough to not be widely reported, but some are. Examples include changing rules for air pollution16 or removing ‘climate change’ from the EPA’s website.17 These changes can be reversed by future Democratic administrations, but the result is that environmental protections are not consistently applied.

Conclusion

Environmentalism was a very successful movement from the 1960s-1980s. It convinced a majority of people in both parties of the importance of its concerns. It was effective at passing domestic legislation and negotiating international treaties. Presidents from both parties supported the movement.

The rise in partisanship starting in the 1990s was bad for environmentalism. Some presidential administrations are now hostile to the movement. Major legislation is still sometimes passed, but much less frequently than before. Overall public support did not increase as environmentalism became partisan.

Many of the specific claims made here are investigated in the full report. If you want to know more about how fossil fuel companies’ campaign contributions, the partisan lean of academia, or newspapers’ reporting on climate change have changed since 1980, the information is there.

Introduction

Environmentalism in the United States today is unusually partisan, compared to other issues, countries, or even the United States in the 1980s. This contingency suggests that the explanation centers on the choices of individual decision makers, not on broad structural or ideological factors that would be consistent across many countries and times.

This post describes the history of how particular partisan alliances were made involving the environmental movement between 1980 and 2008. Since individual decisions are central to understanding why this happened, this history is best presented as a narrative following the key people and organizations.

Environmentalism in the Reagan Era

In the wake of the New Deal, the Republican Party acquiesced to the government having a larger role in society than it had had before the Great Depression.1 Republican presidents would sometimes support increased government spending and regulation. This is apparent in environmental policy: Nixon was involved in several major pieces of environmental legislation and created the EPA in the executive branch.

The election of Reagan in 1980 reoriented the Republican Party. It would now advocate for a smaller government: less (non-military) spending, lower taxes, and less regulation. The free market would provide many of the services that had previously been done by the government. Thatcher’s election as Prime Minister had similar results for the Conservative Party in the UK.

This might seem like it would cause a deep ideological conflict: Environmentalists advocated for regulations on private enterprise and international cooperation on policy, while the Republican Party preferred private and local action. However, this is not what we observe with either Reagan or Thatcher.

The conservative leaders in the US and UK supported environmentalism, even when it involved international regulations. The clearest example of this is the Montreal Protocol on Substances That Deplete the Ozone Layer in 1988. Reagan described it as: 

The Montreal protocol is a model of cooperation. It is a product of the recognition and international consensus that ozone depletion is a global problem, both in terms of its causes and its effects. The protocol is the result of an extraordinary process of scientific study, negotiations among representatives of the business and environmental communities, and international diplomacy. It is a monumental achievement.2 

The US Senate ratified the Montreal Protocol unanimously.3

Reagan and Thatcher also specifically supported international regulation to combat climate change. Thatcher was the first head of government to talk about climate change at the UN, in 1989, and called for an international conference on climate change in 1992 (The Earth Summit in Rio de Janeiro).4 The International Panel on Climate Change (IPCC) creates the reports summarizing the scientific consensus about climate change, and its “principal architect was the conservative Reagan administration.”5 In 1992, the U.N. Framework Convention of Climate Change (UNFCCC), the result of the Rio Summit, had the support of the Bush Sr. administration. The U.S. Senate decided that it was popular enough to not need a roll-call vote.6

These actions of conservative leaders also translated into broad popular support, including among Republicans. The 1980s saw increasing concern about the environment and a decreasing partisan gap. Republicans only became anti-environmentalist after 1990.

Anti-environmentalism is not the natural consequence of the small government ideology of Reagan and Thatcher. It only entered the US Republican Party a decade later, and the UK Conservative Party has continued to support environmentalism.

Environmentalists, Climate Scientists, & Democratic Politicians

The earliest partisan alliances involving climate change began in Congressional hearings in the 1980s.

The Reagan administration entered office promising to reverse most of the energy policies of the Carter administration and dramatically shrink the Department of Energy, which had just been created in 1977.7 One of the programs cut was a newly established center for climate research.8 As an undergraduate student, Al Gore had taken classes in climate science from Roger Revelle, one of the first people to study global warming. In the House of Representatives, Gore led Congressional hearings against these particular cuts (which were partially reversed), and continued to be very involved whenever climate was an issue in Washington.9 Climate policy at this time was still bipartisan, and the Reagan administration was open to government action on climate.

The environmental movement also became increasingly interested in climate change in the late 1980s, particularly after the summer of 1988. This summer saw severe drought across most of the U.S., low enough water in the Mississippi River to hinder barge traffic, heat waves, major fires in Yellowstone, and a Category 5 hurricane in the fall. In and after a Congressional hearing on the climate, James Hansen of NASA claimed that he was “99 percent certain” that “the greenhouse effect is here.”10 Many other climate scientists did not believe that the evidence was that strong yet and disliked his combative tone. A few publicly rebuked him.11

Focusing on climate change provided a way to unify the disparate concerns of the environmental movement, including air & water pollution, habitat conservation, recycling, and energy production. The historian of science Spencer Weart describes this transition as:

The environmental movement, which had found only occasional interest in global warming, now took it up as a main cause. Groups that had other reasons for preserving tropical forests, promoting energy conservation, slowing population growth, or reducing air pollution could make common cause as they offered their various ways to reduce emissions of CO₂.1213

An unusually explicit statement of this strategy comes from Senator Timothy Wirth (D-CO):

What we've got to do in energy conservation is try to ride the global warming issue. Even if the theory of global warming is wrong, to have approached global warming as if it is real means energy conservation, so we will be doing the right thing anyway in terms of economic policy and environmental policy.14

In the late 1980s and early 1990s, environmentalists became increasingly focused on climate science, and both environmentalists and climate scientists formed political alliances with Democratic politicians.

The Clinton-Gore Administration and the BTU Tax

In 1992, Bill Clinton selected Al Gore as his vice presidential candidate and secured the endorsements of environmental organizations like the Sierra Club that had mostly stayed above the partisan fray.

One of the Clinton’s administration’s early legislative goals was a tax on energy, measured in British thermal units (BTUs).15 While this is sometimes remembered as an attempt at a carbon tax, it taxed energy rather than carbon dioxide. Solar, wind, and geothermal power production were exempted, but nuclear and hydroelectricity were not. This tax proved extremely unpopular. To shore up support, the Clinton administration agreed to more exemptions for particular industries, but this diminished what the bill hoped to accomplish, did not improve its popularity in Congress, and encouraged even more groups to request exemptions.16 The broad-based BTU tax was abandoned, and replaced with a much weaker tax on gasoline. Congressmen who had supported the BTU tax suffered politically in the midterms. 

In 1994, Republicans won control of the House of Representatives for the first time in 40 years. The BTU tax was not the only significant issue: the NRA also organized against an assault rifle ban and Newt Gingrich innovated by using a national strategy instead of focusing on individual races. Opposing a climate policy was one thing that helped propel Republicans into power in Congress.

Fossil Fuel Companies, Climate Skeptics, & Conservative Think Tanks

The fossil fuel industry opposed government action on climate change. Significant reductions in greenhouse gas emissions would completely undermine their business model, forcing them to transition to a different industry (renewable energy) or dramatically lose market share. In 1989, a group of fossil fuel and manufacturing companies founded the Global Climate Coalition to oppose climate policy that they claimed would disrupt the American economy.17 The Global Climate Coalition spent tens of millions of dollars in ad campaigns and contributions to politicians before it disbanded in 2001.

At the time, it was not obvious that the industry lobbying would overwhelmingly favor Republicans. There had not previously been a strong tendency for fossil fuel companies to support Republicans.

It is not too surprising that the industry lobby ended up favoring Republicans. Republicans were the more business-friendly party. There was already a small bias for campaign contributions in that direction and oil is more concentrated in Republican-leaning states. The Gulf War in 1990 might have associated the oil industry with the Republican Party, although the war proved broadly popular.18 Academics, probably including climate scientists, were somewhat more likely to lean to the left, although not nearly as much as they do today. Congressional Republicans had been somewhat more likely to oppose environmental legislation than Congressional Democrats for several decades. But then there was an abrupt change in the early 1990s, and this difference dramatically increased.

The Global Climate Coalition found willing allies among conservative think tanks. These think tanks would accept funding from the fossil fuel industry to hire skeptical climate scientists or experts from other fields who were skeptical of climate change. They would publish policy studies, newsletter articles, and press releases that cast doubt on conventional climate science and opposed climate policy proposals. These think tanks and skeptics were successful at reframing the climate debate and creating the “non-problematicity” of global warming among conservatives. The most common claims by conservative think tanks were that “the scientific evidence for global warming is highly uncertain” and “proposed action would harm the national economy.”19

It is not clear to me whether the causal relationship mostly points from conservative think tanks to Republican congressmen or vice versa. The first climate skeptic publication by a major conservative think tank was in 1991, before Clinton & Gore were elected or Gingrich became Speaker of the House. However, there were initially only single digits of publications per year, mostly by a single think tank: the Marshall Institute. The publications did not become common or widespread until 1996-1997. Between 1991 and 1996, most conservative think tanks did not yet have a public position on climate change.

After the Republican Party led by Gingrich won the Midterm elections in 1994, the number of Congressional hearings about climate change decreased. When there were hearings, Congress would invite similar numbers of conventional climate scientists and climate change skeptics.20 Congress began to treat this as an active scientific debate and the media followed suit,21 even though both had previously predominantly presented the scientific consensus.

Debates Over the Kyoto Protocol

The Earth Summit in Rio de Janeiro in 1992 which had produced the UN Framework Convention on Climate Change (UNFCCC) also called for a future summit, in Kyoto, that would impose limits on countries’ greenhouse gas emissions.

While the UNFCCC had gotten broad bipartisan support under the Bush Sr. administration, the politics of climate change had changed dramatically since then. The debate over the Kyoto Protocol would see the last major bipartisan actions on climate change and the beginning of substantial partisanship among the public.

Before the summit, the US Senate unanimously passed the Byrd-Hagel Resolution, which declared that it would not support any treaty that imposed restrictions on developed countries (like the US) but not developing countries.22

The summit was very contentious and the negotiations almost collapsed. On the last day, Vice President Gore flew to Kyoto to save the agreement. The resulting Kyoto Protocol did not impose any restrictions on the greenhouse gas emissions of developing countries. President Clinton signed the treaty, but did not even submit it to the Senate for consideration.23 To win, it would need 2/3 of the Senate, who had just unanimously opposed it.

I am uncertain whether this should be thought of more as a case where the Senate overconstrained the international negotiating position of the presidential administration or more as a case of the administration ignoring the advice of the Senate.

Before and during the summit, the Clinton administration ran a media campaign to build public support for the resulting treaty. There was a massive increase in media coverage, most of which was aligned with conventional climate science. Conservative think tanks also dramatically increased their production of skeptical media. A pair of surveys conducted before and after this debate found that it did make people more aware of climate change as an issue. A majority of people believed that climate change was going to happen, was going to be bad, and that the government should limit air pollution to address it. The overall percentages of people who supported these positions did not change as a result of the debate. There were underlying shifts as strong Democrats came to increasingly support the administration’s policy and strong Republicans came to oppose it.24

The Kyoto Protocol was a failed attempt at climate policy in the United States that directly increased partisanship.

Continued Increases in Partisanship

Partisanship continued to increase after the debate over the Kyoto Protocol. 

Al Gore ran for president in 2000. Although he was strongly associated with climate change by this point, multiple sources claimed that environmentalism was not a major issue in this election.25 Bush Jr. became president instead, declared that the U.S. would not fulfill its obligations under the Kyoto Protocol, and reduced funding to climate science.26

During this time, some of the structural factors that might have been contributing to rising partisanship ended. The Global Climate Coalition disbanded in 2001. Some of the companies which had been involved accepted climate change, while others continued to promote skepticism.  The mainstream media stopped presenting both sides of the debate in 2003-2004. Nevertheless, the partisan gap continued to grow.

In 2006, Gore released a climate change documentary titled An Inconvenient Truth. This did not change overall public opinion.27 Instead, partisanship continued to increase: more Democrats were becoming climate activists, while Republicans were becoming increasingly skeptical.

There were still prominent Republicans who supported policies to counteract climate change. Governor Arnold Schwarzenegger introduced a cap-and-trade system for California,28 while Senator John McCain co-sponsored a bill that would create a similar system for the country.29 However, an increasing number of Republicans became increasingly opposed to environmentalism, and the environmental movement became increasingly tied to the Democratic Party.

Subsequent decisions by both parties, and the environmental movement itself, continued to contribute to rising partisanship on environmental issues in the United States.

Conclusion

Broad structural and ideological differences do not explain the partisanship of environmentalism. During the Reagan and Bush Sr. administrations, the Republican Party did support environmentalism, including international agreements on climate change, despite its small-government orientation on most issues. The Republican Party did not significantly change its ideology between the 1980s and 2000s. The subsequent partisanship in environmentalism then cannot be explained by foundational ideological differences between Democrats and Republicans. Instead, the explanation involves a history of alliances made by particular decision makers.

The first alliance made was between environmentalists, climate scientists, and Congressional Democrats during Congressional hearings in the 1980s. The key figure here was Al Gore. Environmentalists seemed to accept the usefulness of this alliance and did not seriously try to find a similarly prominent or rising Republican politician to ally with as well.

The second alliance made was between fossil fuel companies, climate skeptics, and conservative think tanks, starting around 1990. The industry organized into the Global Climate Coalition in 1989 and convinced the Marshall Institute to begin publishing climate skepticism in 1991. For a few more years, this alliance was not complete: most conservative think tanks were still neutral on climate change, and environmentalists might have been able to convince some of them to support their cause.

When environmentalist-aligned Democrats were in political power in the 1990s, they made several policy proposals that were deeply flawed. A tax proposed in 1993 taxed energy produced, not carbon dioxide emitted, and included arbitrary exemptions. The Kyoto Protocol in 1997 contained terms that the Senate had previously rejected unanimously. These flawed policy proposals made it easier for Republicans like Gingrich or Bush Jr. to rally the public against them – and environmentalism more broadly. Subsequent decisions, on both sides of the aisle, continued to reinforce the trend towards increasing partisanship.

This partisanship could have been avoided, if various decision makers had made different choices about what alliances to form or not form. Environmentalism is not partisan in many other countries, including in highly partisan countries like South Korea  or France. The resulting partisanship was bad for the environmental movement. As partisanship increased in the 1990s and early 2000s, environmentalism saw flat or falling support, fewer major legislative accomplishments, and fluctuating executive actions.

Introduction

In the United States, environmentalism is extremely partisan.

It might feel like this was inevitable. Caring about the environment, and supporting government action to protect the environment, might seem like they are inherently left-leaning. Partisanship has increased for many issues, so it might not be surprising that environmentalism became partisan too.

Looking at the public opinion polls more closely makes it more surprising. Environmentalism in the United States is unusually partisan, compared to other issues, compared to other countries, and compared to the United States itself at other times. 

The partisanship of environmentalism was not inevitable.

Compared to Other Issues

Environmentalism is one of the, if not the, most partisan issues in the US.

The most recent data demonstrating this comes from a Gallup poll from 2023.1 Of the 24 issues surveyed, “Protecting the Environment Has Priority Over Energy Development” was tied for the largest partisan gap with “Government Should Ensure That Everyone Has Healthcare.” Of the top 5 most partisan issues, 3 were related to environmentalism. The amount this gap has widened since 2003 is also above average for these environmental issues.

Pew also has some recent relevant data.2 They ask whether 21 particular policies “should be a top priority for the president and Congress to address this year.” The largest partisan gap is for “protecting the environment” (47 p.p.), followed by “dealing with global climate change” (46 p.p.). These are ten percentage points higher than the next most partisan priority. These issues are less specific than the ones Gallup asked about, and so might not reveal as much of the underlying partisanship. For example, most Democrats and most Republicans agree that strengthening the economy is important, but they might disagree about how this should be done.

Guber’s analysis of Gallup polls from 1990, 2000, & 2010 also shows that environmentalism is unusually partisan.3 Concern about “the quality of the environment” has a similar partisan gap as concern about “illegal immigration,” and larger than concern about any other political issue. If we hone in on concern about “global warming” within overall environmental concern, the partisan gap doubles, making it a clear outlier.

The partisanship of environmentalism cannot be explained entirely by the processes that made other issues partisan. It is more partisan than those other issues. At least this extra partisan gap wants an explanation.

Compared to Other Countries

The United States is more partisan than any other country on environmentalism, by a wide margin.

The best data comes from a Pew survey of “17 advanced economies” in 2021.4 It found that 7 of them had no significant partisan gap, and that the US had a partisan gap that was almost twice as large as any other country.

This is evidence that environmentalism is more likely to be left-leaning. The explanation for this might involve something intrinsic to environmentalism itself, or it might involve interactions between countries and shared media environments. But it clearly is possible for environmentalism to remain bipartisan, which has happened in the UK, France, Spain, Japan, South Korea, Taiwan, and Singapore.

The United States is more partisan overall than most other countries, but it is not an outlier. There are other countries with similar levels of overall partisanship,5 but almost no partisanship in their support for environmentalism: France6 and South Korea.7 There is no correlation between overall partisanship and partisanship in environmentalism.8

Compared to Other Times

Environmentalism was a bipartisan issue in the United States as recently as the 1980s.

The longest data series for U.S. public opinion on environmentalism comes from the General Social Survey, which has been administered to thousands of Americans for most years between 1974 and 2012.9

During the mid-to-late 1970s, support for environmentalism was declining in both parties. Democrats were consistently about 10 percentage points (p.p.) more likely than Republicans to say that there was too little environmental spending.

During the 1980s, support for environmentalism surged. This increase was even larger among Republicans than among Democrats, with the partisan gap closing by the end of the decade.

In the 1990s and 2000s, Democrats’ support for environmentalism remained roughly constant, while Republicans’ support fell dramatically. A large partisan gap opened. The overall support for environmentalism declined, although this might be because support for overall government spending also fell in the early 1990s.10

Gallup polling on similar questions only goes back to 1997.11 It shows an initially modest partisan gap of 15 p.p. in 1997, which grew to an over 50 p.p. gap in 2021.

This change is especially striking because the Republican Party did not change its positions on most issues between the 1980s and 2000s. Underlying principles like small government economics and social conservatism were common to the Republican Party of both decades. The anti-environmentalism of the Republican Party began in the 1990s, clearly after the ‘Reagan Revolution.’

Conclusion

The development of a large partisan gap about environmentalism in the United States was not inevitable. The United States has a smaller partisan gap for most other issues, other countries have less partisanship on this issue (even if the country is very partisan overall), and environmentalism was a bipartisan issue as recently as the 1980s.

This suggests that the explanation for the partisanship does not lie in broad structural or ideological factors that are consistent across many countries and times. Instead, the explanation is likely to be contingent, centered on the choices of individual decision makers.

With AI Impacts, we’re pleased to announce an essay competition on the automation of wisdom and philosophy. Submissions are due by July 14th. The first prize is $10,000, and there is a total of $25,000 in prizes available.

Background

AI is likely to automate more and more categories of thinking with time.

By default, the direction the world goes in will be a result of the choices people make, and these choices will be informed by the best thinking available to them. People systematically make better, wiser choices when they understand more about issues, and when they are advised by deep and wise thinking.

Advanced AI will reshape the world, and create many new situations with potentially high-stakes decisions for people to make. To what degree people will understand these situations well enough to make wise choices remains to be seen. To some extent this will depend on how much good human thinking is devoted to these questions; but at some point it will probably depend crucially on how advanced, reliable, and widespread the automation of high-quality thinking about novel situations is.

We believe1 that this area could be a crucial target for differential technological development, but is at present poorly understood and receives little attention. This competition aims to encourage and to highlight good thinking on the topics of what would be needed for such automation, and how it might (or might not) arise in the world.

For more information about what we have in mind, see some of the suggested essay prompts or the FAQ below.

Scope

To enter, please submit a link to a piece of writing, not published before 2024. This could be published or unpublished; although if selected for a prize we will require publication (at least in pre-print form; optionally on the AI Impacts website) in order to pay out the prize. 

There are no constraints on the format — we will accept essays, blog posts, papers2, websites, or other written artefacts3 of any length. However, we primarily have in mind essays of 500–5,000 words. AI assistance is welcome but its nature and extent should be disclosed. As part of your submission you will be asked to provide a summary of 100–200 words. 

Your writing should aim to make progress on a question related to the automation of wisdom and philosophy. A non-exhaustive set of questions of interest, in four broad categories:

• Automation of wisdom

  • What is the nature of the sort of good thinking we want to be able to automate? How can we distinguish the type of thinking it’s important to automate well and early from types of thinking where that’s less important?

  • What are the key features or components of this good thinking?

    • How do we come to recognise new ones?

  • What are traps in thinking that is smart but not wise?

    • How can this be identified in automatable ways?

  • How could we build metrics for any of these things?

• Automation of philosophy

  • What types of philosophy are language models well-equipped to produce, and what do they struggle with?

  • What would it look like to develop a “science of philosophy”, testing models’ abilities to think through new questions, with ground truth held back, and seeing empirically what is effective?

  • What have the trend lines for automating philosophy looked like, compared to other tasks performed by language models?

  • What types of training/finetuning/prompting/scaffolding help with the automation of wisdom/philosophy?

    • How much do they help, especially compared to how much they help other types of reasoning?

• Thinking ahead

  • Considering the research agenda that will (presumably) eventually be needed to automate high quality wisdom/philosophy:

    • Which parts of the agenda can we expect to automate in a timely fashion? 

    • What is the core that we will need humans to address?

    • What do we expect the thorny sticking points to be?

  • Why may or may not this problem be solved “by default”? (from a technical standpoint)

  • Can we tell concrete stories or vignettes in which the automation of wisdom/philosophy is/isn’t important, to triangulate our understanding of what matters?

  • What preparatory research could provide the best groundwork for humanity to automate high-quality wisdom/philosophy before it is necessary?

  • What projects today or in the near future would be valuable to undertake?

• Ecosystems

  • If the world were devoting serious attention to this, what would that look like?

    • What incentives on institutional actors could push work onto related but less important questions; vice-versa what could help ensure that work remained well-targeted?

  • What are the natural institutional homes for this research in the short term?

    • Academia? Nonprofits? Frontier AI labs? Elsewhere in industry?

  • What might be needed (proofs, audits, track record?) to enable humans (decision-makers, voters) and human institutions to correctly trust wise advice from AI systems?

    • How could we lay the groundwork for this?

  • Ideas for catalysing/sustaining this field?

  • Why may or may not this problem be solved “by default”? (from a social standpoint)

If you’re not sure whether a topic would be within scope, feel free to check with us.

Judging

The judging process will be coordinated by Owen Cotton-Barratt. After shortlisting, entries will be assessed by a panel of judges: Andreas Stuhlmüller, Brad Saad, David Manley, Linh Chi Nguyen, and Wei Dai.

Judging criteria will be:

• Does the entry tackle an important facet of the automation of wisdom/philosophy?

• Does the entry contain good analysis or valuable new ideas?

• Is the writing clear, succinct, and epistemically appropriate?

• Does the entry provide something that we are excited to see built upon or explored further? 

The prize pool is $25,000, and the prize schedule will be:

• $10,000 First Prize

• $5,000 Second Prize

• 4x $2,000 Best-in-Category Prizes 

  • Judging for these will exclude the overall First and Second Prize winners from consideration

    • So if e.g. the overall First Prize and Second Prize both went to entries in the “Ecosystems” category, then the third-best entry in that category would receive $2,000 

• 4x $500 Runner Up Prize, for the best entries across any category that did not receive another prize

  • For these prizes, the judges may give preference to impressive entries by people at early career stages

    • Whereas judging for the main prizes will — insofar as this is feasible — be blind to the identities and personal characteristics of the authors

We may contact entrants whose work impresses us about possible further opportunities (e.g. conferences or research positions) on these topics.

Details

Entries should be submitted via this form, which asks for:

• Your name and email address

• A link to your entry

• A 100–200 word summary

• Which if any of our four categories your entry falls under

• Statement of authorship credit (including AI credit)

• A brief description of career stage (so that judges can at their discretion account for this in awarding Runner Up prizes)

• Opportunity to opt out of future contact not directly related to this competition

• Anything else we should know

You are of course welcome to seek feedback on drafts before submission. Coauthored articles are also very welcome.

The deadline for submissions is midnight anywhere in the world on Sunday 14th July. We hope to complete shortlisting within two weeks of the submission deadline, and contact winners within four weeks of the submission deadline. Winners whose entries are not yet public will have two weeks after we contact them to provide a public version, or agree to us publishing it on the AI Impacts website. Payment will be made by ACH (for US-based winners) or wire transfer (for international winners).

We reserve the right to extend the submission deadline or increase the prize pool without notice. Judges have the right to split prizes in cases of ties, or to not award prizes in the unlikely event that no submissions are found to merit them.

If you want to ask questions about the competition, feel free to comment, or to email essaycompetition@aiimpacts.org

FAQ on the automation of wisdom and philosophy

What’s the basic idea here?

We're interested in the automation of thinking that can help actors to take wise actions (whatever that means) and avoid unwise actions. As an important subcategory, we're interested in the automation of philosophical thinking, and how to avoid practical errors grounded in philosophical mistakes.

What do you want to know about such automation?

We're not certain! We think it's a potentially important area which hasn't received that much attention. We'd like people to explore more of the ideas around this. If we understood more of the contours of when such automation might be helpful (or unhelpful!), that would seem good. If we understood more about what would be necessary for automation, that would seem good. If people developed a sense of things it would be good for someone to do in the world, that's potentially great.

We give a bunch of example questions we'd be interested in people addressing in the essay prompts part of the announcement, but because it seems like a broad area we've preferred to leave the competition fairly open, and wait to see which parts people can make meaningful contributions to.

What do you mean by “wisdom” and “philosophy”?

By “wisdom”, we mean something like “thinking/planning which is good at avoiding large-scale errors”. An archetype of something which is smart-but-not-wise might be a plan full of clever steps which are each individually well-chosen to chain to the previous step in the plan, but which collectively forget why they were doing this, and end up taking actions which are in conflict with the original goal. Wisdom is also what’s needed for noticing that an old ontology was baking in some problematic assumptions about what was going on.

By “philosophy”, we mean something like “the activity of trying to find answers by thinking things through, without the ability to observe answers”. This is close to the sense understood in the academic discipline of philosophy.

We’re not sure if automating these things is most naturally thought of as one topic, two topics, or more …

What threats are you concerned about?

Progress in these areas seems like it could potentially help avoid a number of different issues:

Unwise human actions

Humans sometimes take actions which are predictably unwise (from some perspectives), and which they later regret. Such actions could be really bad if they interact with high stakes situations. If people had access to trusted high wisdom automated advice, this could help them to reduce the rate of these errors.

This might be particularly important around issues coming with the development of AI, as people will be facing very novel situations and be less able to rely on experience.

Human philosophical errors

People sometimes make decisions that are influenced by their philosophical understanding of an issue. This could happen in the future, e.g. around understanding of AI consciousness/rights. Automation of good work, if achievable, could help people to have deeper understanding by the times they need to make key decisions.

Unwise AI actions

If people empower AI agents, ensuring that they are in some sense wise and not just smart could help to reduce rare damaging actions. In the extreme this could reduce risk of human extinction (imagine an AI system which wipes out humans in order to secure its own power, and later on reflection wishes it hadn't; a wiser system might have avoided taking that action in the first place).

AI philosophical errors

If AI systems become superintelligent and are meaningfully running the world, their stances on philosophical questions could matter. e.g. deciding to engage in acausal trade (if it doesn’t actually make sense), or deciding not to (if it does) could be a large and consequential error. Better understanding of the automation of philosophy could help either to lead to more philosophically-competent AI systems, or alternatively could help people to coordinate about which parts of thinking should not be delegated to AI systems.

Is there a particular threat model you’re focused on?

No. We could make some guesses (both about which of the above categories are most concerning, and more concretely what the most concerning threats within them are), but we feel like the whole area is under-explored, and wouldn’t be confident in our guesses. We’d love to see high-quality analysis of this.

The fact that the automation of wisdom/philosophy seems important to better understand for multiple different threats — and also seems like a plausibly useful intervention for improving our ability to handle unknown unknowns — feeds into our desire to see it prioritized more than at present.

Automating wisdom, philosophy — isn’t this all just AI capabilities work?

Maybe! Certainly this is a type of capability (and high performance probably requires significantly advanced general capabilities, relative to today).

However, it seems to us that for a given level of general smarts in a system, the capacity for wisdom or philosophy could keep up with that, or could fail to. We are concerned about worlds where the ability to automate wise actions is outstripped by the ability to automate smart ones. So it seems like it may (at least in part) be a problem of differential technological development. We would be interested in further analysis of this question.

The first serious attempt at building a subway in New York City occurred in 1866, following the end of the Civil War (1865) and the opening of the first subway in London (1863). The following decades saw a sequence of failed attempts, and the first subway in New York City would not begin operations until 1904.

When we consider how popular the subway was, even when it first opened, these failures are remarkable. Many of the important actors were indifferent or opposed, and those who supported building a subway were comically bad at coordinating. Only widespread, mostly decentralized public support was able to pressure different political and economic elites into finally cooperating.

The history of these efforts made me wonder whether the construction of a subway in New York City was inevitable. I am usually skeptical of technological inevitability, but this seems like a potentially good example. For decades, the New York City subway was a locally resisted technological temptation,1 and a potential example of technological overhang.2 The subway eventually succeeded despite a system that seemed structured to thwart it.

The main source I read about this history was:

Wallace B. Katz. The New York rapid transit decision of 1900: Economy, society and politics. (1978),3 from a series put together by the Historic American Engineering Record.4

All quotes or claims that are not cited come from this source. I have not investigated other sources in detail, but it seems to present a mainstream scholarly view of the history.

Why Did New York City Need A Subway?

By 1866, it seems inevitable that New York City would become one of the greatest world cities. It has an excellent harbor and a deep river heading inland. After the completion of the Erie Canal (1825), most of the Midwest’s international trade passed through New York. It has been the largest city in the US since the first census (1790),5 and was among the 10 largest in the world by 1850.6 Network effects would help it continue to grow. The New York Stock Exchange brought in large businesses, especially railroads. New York City was also a major destination for immigrants.7

Rapid transit was even more important for New York than for other great world cities. Manhattan is an island, and so New York City has fewer directions it can grow in than London, Paris, or Beijing. Manhattan is only two miles wide, and the central business district was at the end, in downtown.8 People would commute much of the length of the island or across the rivers on ferries.

New York City had both surface and elevated trains before the subway was built. The surface trains (streetcars) had an extensive network, with many operators who eventually coalesced into the Metropolitan Traction Company. They could not travel faster than other traffic on the streets, so even an extensive network with good transfers wasn’t effective for longer trips. There were also some elevated lines (els) run by the Manhattan Railway Company. These were faster than the streetcars, but still had limited speed unless their steel viaducts were replaced by large, expensive, and more stable stone viaducts. Subways could go faster because they were built on solid ground. In practice, the subway was built with four tracks to provide both local and express service, while the els were built with only two tracks.9

In the absence of a subway, most of the new immigrants packed into extremely dense tennant houses in the Lower East Side of Manhattan. These slums were widely understood to be bad for the health, wealth, and moral progress of the people who lived in them. The hope was that the subway would allow poor immigrants to access the job opportunities of downtown Manhattan without having to live in the slums of the Lower East Side.10

Major Actors

The major actors involved in building the New York subway included:

• Tycoons. 
  
  Most infrastructure in the US had been built by private corporations or conglomerates, including New York’s streetcars and els. Most people assumed that some tycoon would be the one to build and operate the subway.
  
  Most tycoons were uninterested in building the subway, because the construction would cost more than other transit possibilities. They believed that intercity rail was more likely to achieve an acceptable rate of return: at least 6% annually.
  
  New York City’s Chamber of Commerce consisted of and represented the interests of the tycoons.
  

• Reformers. 
  
  These people were mostly interested in improving the lives of the people living in the slums. The subway would provide a great public service, and would confirm New York’s preeminent place among the greatest cities in the world.
  
  This group largely overlaps with the tycoons.11 Magnanimously building great public works was popular among the business elites - especially if these public works could also make a decent rate of return. The tycoons who were interested in urban transit (which was a minority) had progressive tendencies.
  
  The Chamber of Commerce also represented the reformers.
  

• Tammany Hall. 
  
  New York City’s local politics was dominated by Tammany Hall. Their base was largely poor immigrant communities, where they would exchange favors for votes. They were engaged in many instances of small-scale corruption and had earned a reputation of “lay[ing] the hand of spoliation upon the public funds.” 
  
  Occasionally, a coalition of all other political factions in the city could defeat the Tammany candidate for mayor, or a Tammany-backed mayor would act independently once in office. For the most part, dealing with the city government meant dealing with Tammany Hall.
  
  Tammany Hall was mostly indifferent to whether a subway was built. All of the other actors believed that Tammany was incapable of running a railroad and, if they gained control over one, they would use it for petty patronage.12
  

• State Government. 
  
  Since the building of a railroad requires the exercise of state power (eminent domain), private companies would have to get a charter from the state legislature and sometimes approval from the state supreme court.
  
  Most people in New York at the time lived in rural areas, so the state government was indifferent to whether New York City had a subway. They did not want to increase the power of their political rivals in the city, especially Tammany Hall.
  

• Metropolitan (streetcar) and Manhattan (els) Companies. 
  
  They largely opposed the construction of a subway, because it would mean competition with their services. They had extensive economic and political connections across the city and could make things very difficult for anyone else interested in building or operating transit in New York City.
  
  It is interesting that neither of them made a serious effort to build a subway themselves.13 Controlling multiple forms of transit would have given them a more substantial monopoly.
  
  There were both technical and financial reasons for their lack of interest. Financially, both companies had raised capital using ‘watered’ stock. This left them paying larger dividends to their shareholders than they could reasonably afford. Technically, they had other things they wanted to focus on: consolidating lines after purchasing formerly independent railroads, improving transfers, possibly adding a third set of rails to increase capacity, and electrification. Starting a whole new system seemed like a lot of additional work.14
  

• Local Businesses. 
  
  Local businesses, especially along Broadway, did not want the disruption that subway construction would entail. These are the NIMBYs.15 They were successful at making sure that the first subway would not run under the street through downtown which had the highest demand.

Basic Dynamics

The efforts to get a subway built occurred multiple times, with sixteen different companies being awarded a charter between 1864 and 1902. This is a large enough sample size to get a sense of the underlying dynamics at play.16

The impetus for subway construction typically came from progressives in the Chamber of Commerce, especially Abram Hewitt. They would draft a piece of legislation for the state legislature to consider. The legislation would create a commission empowered to grant a charter to a private company to build and operate the railroad. The commission would solicit bids17 from potential tycoons or companies. The auction would fail to attract serious candidates, and the process would begin again.

The tycoons of the day were not interested in building the subway using their own money, because they did not believe that it would yield a significant rate of return. The city agreed to supply most of the capital, and so the city would legally own the subway. The construction would be done by the private company, and they would lease the railway and run the operations for an extended period of time.18 The company would be obligated to repay the city at a fixed rate of return, but could keep the profits beyond that. The offer of public capital was supposed to incentivize some prominent businessman to build and run the subway, who had the necessary technical skills and would keep the subway out of the hands of Tammany Hall.

The details of the offered charter were determined by negotiations between the Chamber of Commerce, the state government, and city hall. These negotiations would determine who would be on the commission that reviewed the bids, how much capital would be provided publicly vs privately, what rate of return the city expected on its investment, and how long the lease would last. The result of the negotiations would end up being something that no prominent businessman would accept.

There would be some bidders. Sometimes, they would be people with insufficient financial backing or experience building railroads. Sometimes, they would be people interested in construction but not operations.19 Sometimes, the charter would be awarded, and the bidder would reveal that they were a front for someone who wanted to ensure that the subway was not built.20

This system was not working, and it did not look like it was making progress towards working in the future either.

What Changed?

Mass public support eventually forced the various elite groups to cooperate.

In 1893, when the Chamber of Commerce proposed yet another subway bill to the state legislature, New York City’s labor unions submitted a rival bill,21 which proposed a popular referendum. Although both groups wanted to see a subway built, the Chamber of Commerce opposed the referendum as a gateway to anarchy. The compromise bill that passed mostly followed the Chamber of Commerce’s proposal, but did include a referendum.

The referendum passed with over 75% of the vote. The Chamber of Commerce dominated commission reluctantly found itself with broad popular support.

The referendum did provide some additional constraints: the subway would have to run the entire length of Manhattan, and would charge 5¢ to go anywhere in the city. Despite these constraints, broad popular support was extremely useful.

After the state supreme court added some additional constraints,22 a delegation of working men visited the commission and told them that “the working people were surprised to see the Commission ‘knocked out’ in one round against five judges … the law cannot be bigger than the will of the People.”

In 1898, the commission was still looking for offers, and was seriously looking at a proposal from the Metropolitan Traction Company.23 The proposal was contrary to the letter of the law and especially to the spirit of the referendum: they asked for a perpetual franchise, no taxes until after the line had paid for itself, and 10¢ express services. The public was not happy: there were mass meetings throughout the city for weeks and almost every civic organization publicly opposed it, denouncing the Metropolitan, the commission, and Tammany together. Governor Theodore Roosevelt stepped in to stop the deal.

Mass public will had become strong enough that elite groups became willing to work together to build a subway.

The 1900 auction was attended by two bidders and the contract was awarded to John McDonald with (initially secret) backing from August Belmont II. McDonald had experience building railways in Baltimore, and Belmont had previously been involved in the management of the Long Island Rail Road. Both were tycoons, although not quite of the status the commission had originally intended. Both also had political connections to Tammany Hall. It is unclear what exactly was involved in the decision (or deal24) that gave McDonald & Belmont the contract, but the city had finally found people to build the subway who were willing and acceptable to most of the city’s elites.

The Metropolitan hadn’t quite given up. They convinced two financial institutions to renege on bonds they had agreed for McDonald. Belmont had to step in, providing much of the money himself and creating a company to sell (not watered) stock, in exchange for most of the profits.

Belmont’s company, the Interborough Rapid Transit Company, built the first subway line in New York City between 1900-1904, and soon afterwards extended it under the East River to Brooklyn.

By the time the line began construction, all of the technical details had been hashed out.25 The route had been extensively litigated. Everyone knew that the best railroad for urban transit was underground, powered by “electricity,26 and as near the surface as practicable.”

Conclusions

It seems like New York City tried to fail at building a subway system. There was some significant opposition, from the existing transit companies and (to a lesser extent) NIMBY business owners along Broadway. More important than outright hostility seems to be the indifference and conflict between various elite groups in New York.

Political leaders, both at Tammany Hall and the state legislature, didn’t care much about the project. Most of the efforts were instigated by civic-minded tycoons in the Chamber of Commerce. They proved unsuccessful at getting the political leaders to offer a contract that would convince prominent businessmen that this enterprise was worth pursuing. The resulting mess lasted for almost 40 years.

Public support for the subway seems both broad and deep. Katz does not describe how the public became so convinced, but I can guess at the outlines of a story. The problems of overcrowded slums and the difficulty of traveling across Manhattan would have been obvious. The public also had to be informed that better rapid transit was possible. Once they became aware, it is unsurprising that they supported it.

The public was eventually able to impose their will on the elite groups, offering more support than sometimes even the commission leading the efforts. They used a referendum, letters & delegations, mass meetings, and public statements by unrelated civic organizations to make their voice heard. This provided the pressure to get various elite groups to cooperate, until Tammany-friendly tycoons agreed to build the subway.

The result feels inevitable because of how it was pushed from below, in spite of decisions made by many individual actors. Elite indifference and opposition from a few key actors delayed the subway by a generation, but was not able to completely resist the public will.

Subscribe now

Cover image: David Sagarin. Historic American Engineering Record. IRT East Side Line at 23rd Street. Library of Congress, Prints and Photographs Division (1978). https://www.nycsubway.org/perl/show?7860.

It seems interesting to compare companies working on emerging technologies to get a feel for how surprising developments in these companies are. Some startups working on developing AI have grown rapidly, but how surprising is this growth?

The leading startup for AI is OpenAI. The leading startup for fusion is Commonwealth Fusions Systems (CFS). Here is some basic information for these two companies:

OpenAI was founded 2-3 years before Commonwealth Fusion Systems.12 Other than that, their trajectories have been surprisingly similar.

OpenAI currently has 770 employees.3 Commonwealth Fusion Systems currently has “>600 employees and >100 contractors.”4

OpenAI has received about 5 times as much funding as CFS. Most of this money came from a single investment by Microsoft in 2023.5 If we look at OpenAI from 2-3 years ago to control for the different founding dates, then OpenAI had raised less money than Commonwealth Fusion Systems. It would not surprise me if CFS gets significantly more investment in the next several years as well.

The $10B investment in OpenAI occurred after the launch of ChatGPT, 7 years after the company was founded. Commonwealth Fusion Systems’ first major project is SPARC, a tokamak experiment which should be completed in 2025, also 7 years after the company was founded. The goal of SPARC is to demonstrate Q>1, or net energy gain from the plasma,6 as quickly as possible, and then work up to demonstrating Q~10.7 Fusion projects are notorious for falling behind schedule,8 but CFS has maintained its schedule so far and construction is well underway. It is not certain if SPARC will succeed in demonstrating Q>1 in 2025 or 2026, but if they do, they will likely see a significant increase in hype - and funding. 

OpenAI was definitely not making a profit as recently as 2022.9 Their revenue has increased dramatically since then,10 so they might be making a profit now. Commonwealth Fusion Systems is definitely not making a profit now.

I’m not entirely sure what conclusions should be drawn from this comparison. It seems easy to overestimate the relative importance of an industry that is dominant locally (either in your city or in your social network) and to be unaware of developments that are more distant from you. OpenAI and CFS are obviously not the only actors in AI  and fusion, respectively, and I have not done a systematic comparison between the two fields. The technical details of these two fields are very different and so place different demands on their workforce and capital. Nevertheless, this comparison provides some evidence that investors’ estimates of the value of AI and fusion are not wildly different.

People answered in October, an eventful fourteen months after the 2022 survey, which had mostly identical questions for comparison.

Here is the preprint. And here are six interesting bits in pictures (with figure numbers matching paper, for ease of learning more):

1. Expected time to human-level performance dropped 1-5 decades since the 2022 survey. As always, our questions about ‘high level machine intelligence’ (HLMI) and ‘full automation of labor’ (FAOL) got very different answers, and individuals disagreed a lot (shown as thin lines below), but the aggregate forecasts for both sets of questions dropped sharply. For context, between 2016 and 2022 surveys, the forecast for HLMI had only shifted about a year.

2. Time to most narrow milestones decreased, some by a lot. AI researchers are expected to be professionally fully automatable a quarter of a century earlier than in 2022, and NYT bestselling fiction dropped by more than half to ~2030. Within five years, AI systems are forecast to be feasible that can fully make a payment processing site from scratch, or entirely generate a new song that sounds like it’s by e.g. Taylor Swift, or autonomously download and fine-tune a large language model.

   

3. Median respondents put 5% or more on advanced AI leading to human extinction or similar, and a third to a half of participants gave 10% or more. This was across four questions, one about overall value of the future and three more directly about extinction.

4. Many participants found many scenarios worthy of substantial concern over the next 30 years. For every one of eleven scenarios and ‘other’ that we asked about, at least a third of participants considered it deserving of substantial or extreme concern.

5. There are few confident optimists or pessimists about advanced AI: high hopes and dire concerns are usually found together. 68% of participants who thought HLMI was more likely to lead to good outcomes than bad, but nearly half of these people put at least 5% on extremely bad outcomes such as human extinction, and 59% of net pessimists gave 5% or more to extremely good outcomes.

Future Value 800

58.1KB ∙ PDF file

Download

Download

6. 70% of participants would like to see research aimed at minimizing risks of AI systems be prioritized more highly. This is much like 2022, and in both years a third of participants asked for “much more”—more than doubling since 2016.

If you enjoyed this, the paper covers many other questions, as well as more details on the above. What makes AI progress go? Has it sped up? Would it be better if it were slower or faster? What will AI systems be like in 2043? Will we be able to know the reasons for its choices before then? Do people from academia and industry have different views? Are concerns about AI due to misunderstandings of AI research? Do people who completed undergraduate study in Asia put higher chances on extinction from AI than those who studied in America? Is the ‘alignment problem’ worth working on?

Share

Subscribe now

Subscribe now

Examples

1942-1945: Manhattan Project scientists considered whether a single atom bomb would set the entire planet on fire.

During the Manhattan Project, some of the scientists became concerned that the atomic bomb they were building might generate enough heat to set off a chain reaction that would ignite the atmosphere and quickly end all life on Earth. Unlike many risks from new technologies, this one could not be directly evaluated through experimentation because there would be no survivors to observe a negative outcome of the experiment. Fortunately, the relevant physics was understood well enough to take a decent shot at evaluating the risk with theory and calculation alone. And even more fortunately, the calculations showed that atmospheric ignition was very unlikely– or at least unlikely enough that the scientists proceeded to detonate the first atomic bomb for the Trinity test.

The historical accounts differ about exactly how small the scientists believed the risk to be and what amount of risk they deemed acceptable:

If, after calculation, [Arthur Compton] said, it were proved that the chances were more than approximately three in one million that the earth would be vaporized by the atomic explosion, he would not proceed with the project. Calculation proved the figures slightly less -- and the project continued.
-Pearl S. Buck, in 1959, recalled a conversation she had with Arthur Compton, the leader of the Metallurgical Laboratory during the Manhattan Project.

There was never “a probability of slightly less than three parts in a million,”... Ignition is not a matter of probabilities; it is simply impossible.
-Hans Bethe, the leader of the T Division during the Manhattan Project, in 1976

Arthur Compton never disputed Buck’s account,1 so Compton probably did decide that a 3 x 10^-6 (three-in-a-million) chance of extinction was acceptable. Although Hans Bethe’s contrasting account shows some disagreement about the likelihood of the risk, the only documented attempt to decide an acceptable limit for the risk is Compton’s.

It’s hard to say whether it was reasonable for Compton to accept a 3x10^-6 chance of atmospheric ignition. It’s a small number, in the same order of magnitude as one’s chance of dying from a single skydiving jump. But when it comes to setting the earth on fire, how small is small enough? One way to weigh the risk is by finding the expected number of deaths it equates to, but the answer varies depending on the assumptions used:

• If the Manhattan Project scientists only gave moral value to humans who were alive in their time, then they could have multiplied the 1945 world population of ~2.3 billion by Compton’s threshold probability of 3x10^-6 for an expected 7000 casualties.

• As someone who lives “in the future” from the perspective of 1945, I would strongly prefer for them to have also placed moral value on humans living after their time. One analysis estimates that there could be between 10¹³ and 10⁵⁴ future humans, so a 3x10^-6 chance of extinction would be equivalent to somewhere between 30 million and 3x10⁴⁸ casualties.

A three-in-a-million chance of human extinction is clearly a serious risk, but it may have been reasonable given how gravely important Compton and his colleagues believed the Manhattan Project to be. In his interview with Buck, Compton made it clear that he considered the stakes to be extremely high, saying that it would have been “better to accept the slavery of the Nazis than to run the chance of drawing the final curtain on mankind.”

Because of the secrecy demanded by the Manhattan Project, the public had no knowledge of the risk of atmospheric ignition until many years later. The assessment of the risk and the decision about how to handle it was likely made by a relatively small number of scientists and government officials.

1973-1975: Biologists considered whether recombinant DNA research would create deadly pathogens.

Methods for genetic recombination quickly progressed in the early 1970s, making it easier to combine the DNA of multiple organisms and causing some biologists to become concerned that their research could create deadly new pathogens. In 1972, Biochemist Paul Berg and his colleagues combined the DNA of a cancer-causing virus with the DNA of the E. Coli bacterium, which lives in the human gut. Berg planned to insert this recombinant DNA back into E. Coli bacteria but was convinced by his colleagues to halt the experiment over fears that the altered bacteria might get out of the lab and harm the world.

Prompted by these concerns, the National Academy of Sciences (NAS) formed a committee chaired by Berg to evaluate the risk. In 1974, the committee asked for a moratorium on certain types of recombinant DNA experiments. Despite disagreement among biologists, the moratorium was adhered to.

In 1975, scientists, lawyers, and policymakers met for the Asilomar Conference on Recombinant DNA Molecules to determine whether to lift the moratorium. Twelve journalists were also selected for invitation to the conference, possibly due to fears about cover-up accusations in the wake of the recent Watergate scandal. The conference ultimately concluded that most recombinant DNA research should be allowed to proceed under strict guidelines but that experiments involving highly pathogenic organisms or toxic genes should be forbidden. The guidelines were adopted by the National Institutes of Health as a requirement for funding.

Unlike the physicists at the Manhattan Project, biologists in the 1970s did not have the theoretical understanding to confidently evaluate the risk and build a consensus around it. It’s unclear how likely or severe they thought the risks from recombinant DNA were. In 2015, Berg said that “if you sampled our true feelings, the members of the committee believed the experiments probably had little or no risk, but … nobody could say zero risk.” In the decades following the conference, recombinant DNA experiments were mostly safe, and many of the guidelines were gradually scaled back over time. In 2007, Berg said that they “overestimated the risks, but [they] had no data as a basis for deciding, and it was sensible to choose the prudent approach.”

1999: Physicists considered whether heavy-ion collider experiments would destroy the planet.

In 1999, the year before the Relativistic Heavy Ion Collider (RHIC) began operating at Brookhaven National Laboratory (BNL), members of the public became concerned by media reports about a chance that the collider could create miniature black holes that would destroy the earth. In response to these concerns, the director of the BNL convened a committee of physicists to write a report addressing the risks.

The report, authored by Busza et al., uses a combination of theory and empirical evidence to demonstrate that the RHIC was very unlikely to cause a catastrophe. The report evaluates three types of speculated catastrophic risks from RHIC but mostly focuses on the risk of dangerous ‘strangelets.’2 In the report, the authors say that theoretical arguments related to strangelet formation are enough on their own to “exclude any safety problem at RHIC confidently.” They also make an empirical argument based on the observation that the moon has not been destroyed despite constant bombardment by cosmic rays. Based on that empirical evidence, they derive an upper bound for the probability that ranges from 10⁻⁵ (one in ten thousand) to 2×10⁻¹¹ (two in one hundred billion) depending on assumptions made.

The authors of another report, authored by Dar et al. and published by CERN, used a similar empirical argument about the frequency of supernovae to calculate an upper bound of 2x10^-8 (two in a hundred million) for the likelihood of dangerous strangelets being produced at RHIC.

What did these physicists believe about the limit of acceptability for risks of this severity? Dar et al. describe their bound of 2x10^-8 as “a safe and stringent upper bound.” In the first version of their paper, Busza et al. described their bounds as “a comfortable margin of error,” but in the final version, they instead say that “We do not attempt to decide what is an acceptable upper limit on p, nor do we attempt a ‘risk analysis,’ weighing the probability of an adverse event against the severity of its consequences.”

The following year, physicist Adrian Kent released a paper criticizing Dar et al. and Busza et al. for a lack of nuance in their assessment of the risks. Kent points out that, even without considering future lives, Dar et al.’s “safe and stringent upper bound” of 10^-8 would imply 120 casualties in expectation over the course of ten years. He says that “despite the benefits of RHIC, the experiment would not be allowed to proceed if it were certain (say, because of some radiation hazard) to cause precisely 120 deaths among the population at large.”

Kent uses a back-of-the-envelope calculation involving established risk tolerances for radiation hazards to propose that the acceptable upper bound for risk of extinction should be 10^-15(one in a quadrillion) per year if not accounting for future lives and 10^-22(one in ten billion trillion) per year if accounting for future lives.3 He also argues that the acceptable risk bounds for a project should be decided further in advance, and agreed on by experts that aren’t actively involved in the project.4

2005-?: The SETI community considers whether messages sent to space will invite the attention of hostile extraterrestrials.

The search for extraterrestrial intelligence (SETI) involves monitoring the cosmos for signals of extraterrestrial life. Around 2005, some members of the SETI community became enthusiastic about the idea of “Active SETI,” which refers to intentionally sending messages to the cosmos. This sparked an intense debate within the community- some were concerned that Active SETI could endanger humanity by attracting the attention of hostile extraterrestrials. A 2006 article in Nature argued that these were “small risks” that “should nevertheless be taken seriously.” Supporters of Active SETI argued that extraterrestrial civilizations could likely detect radio signals from Earth regardless of whether messages with more powerful signals were sent.

Science fiction author David Brin was one of the concerned figures in the SETI community. In 2006, Brin wrote that he and others in the community had called for a conference to discuss the risks of Active SETI but that their concerns had been largely ignored by the rest of the community. Frustrated by what he perceived as a failure of Active SETI supporters to consult with their colleagues,5 Brin considered making the issue more public by contacting journalists with the story. Brin wrote that he preferred the “collegiate approach,” though, because public attention on the issue could damage SETI's reputation as a whole, and it’s unclear if he ever attempted to bring the story to journalists.

A permanent committee in the International Academy of Astronautics (IAA) is the closest thing to a regulatory body within the SETI community. In 2007, this committee drafted new principles regarding sending messages to Extraterrestrials, including that the decision of whether to do so “should be made by an appropriate international body, broadly representative of Humankind.” However, these principles were never adopted.

The debate over Active SETI was renewed in 2010. Stephen Hawking made headlines by saying that it was a bad idea. At a two-day conference hosted by the Royal Society, members of the SETI community had a heated debate but did not reach a consensus. At its annual meeting, the IAA SETI committee updated its Declaration of Principles for the first time in over 20 years, but the updated principles still made no mention of Active SETI.6

In 2015, the community debated the issue again at another conference with no resolution. That same year, 28 scientists and business leaders, including Elon Musk, signed a statement calling for “a worldwide scientific, political and humanitarian discussion” before continuing Active SETI, with an emphasis on the uncertainties that surround the existence, capabilities, and intentions of potential extraterrestrial intelligence.

The debate about whether the risks of Active SETI are acceptable appears to still be unresolved, and it’s unclear if the two sides of the debate are still discussing it with each other. The most recent transmission listed on the Active SETI Wikipedia page was sent in 2017.

1951-present: Computer scientists consider whether a sufficiently powerful misaligned AI system will escape containment and end life on Earth.

Concern about the impact of powerful AI systems dates back to the beginning of modern computer science:

Let us now assume, for the sake of argument, that [intelligent] machines are a genuine possibility, and look at the consequences of constructing them... There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler's Erewhon.
-Foundational computer scientist Alan Turing in 1951.

Today, many experts are concerned about the risk that there may eventually be an AI system that is much more capable than humans with goals that are not aligned with those of humans. When pursuing its goals, such a system might cause human extinction through its efforts to acquire resources or survive.

Unlike the scientists at the Manhattan Project or Brookhaven National Laboratory, AI researchers have no agreed-upon method for calculating the amount of extinction risk. Theoretical understanding of the nature of intelligence does not yet have the strong foundation seen in fields such as nuclear physics. Empirical data is limited because no human has ever interacted with an intelligence more capable than all humans.

Expert opinions vary widely about the amount of risk posed by powerful AI systems:

• Decision theorist Eliezer Yudkowsky argues that, with our current techniques, a powerful AI would be “roughly certain to kill everybody.”

• Former OpenAI researcher Paul Christiano believes that the total risk of extinction from AI is 10-20%.

• Two researchers who describe themselves as AI optimists argue that “a catastrophic AI takeover is roughly 1% likely.”

• In a recent survey of AI researchers, the median researcher gave a 5-10% chance7 of catastrophic outcomes such as human extinction.

Although it is common to see estimates of the extinction risk from AI, there seem to be few attempts to set an acceptable upper limit for it. Theoretical computer scientist Scott Aaronson wrote that his limit “might be as high as” 2% if the upside would be that we “learn the answers to all of humanity’s greatest questions.” Even without taking future lives into account, a 2% extinction risk is equivalent to around 160 million casualties in expectation, roughly four times the population of Canada.8 It’s difficult to say whether the potential benefits of powerful AI systems would justify taking that relatively high risk.

Citing the scientific uncertainty about the future outcomes of AI development, an open letter from earlier this year called for a moratorium on frontier AI development. The letter was signed by some prominent experts, including Yoshua Bengio, a Turing Prize winner, and Stuart Russell, co-author of the standard textbook “Artificial Intelligence: a Modern Approach.” As of this writing, no moratorium seems to have taken place– just two weeks ago, Google announced the release of its “largest and most capable AI model.”

Although discussions about extinction risks from AI have historically been rare outside of a relatively small research community, many policymakers, journalists, and members of the public have recently become more involved. Last month, the AI Safety Summit in the UK brought experts and world leaders together to discuss risks from AI and how to mitigate them.

Takeaways

Scientists sometimes disagree about the acceptable upper limit for extinction risks but usually agree that it should be extremely small.

For three cases, I found examples of scientists suggesting an acceptable limit for risk of extinction:

• A Manhattan Project scientist set his upper limit at 3x10^-6 (three in a million).

• Scientists at CERN considered a 2x10^-8 (two in a hundred million) risk to be acceptable for RHIC, but another physicist argued that a more appropriate limit would be 10^-15 (one in a quadrillion) or 10^-22 (one in ten billion trillion).

• One prominent computer scientist might accept as much as a 2x10^-2 (two in a hundred) chance of extinction from powerful AI.

Although these upper limits vary widely, they all fall below the 5x10^-2 (five in a hundred) to 10^-1 (one in ten) odds the median expert gave for extinction from AI last year, and usually by many orders of magnitude. Notably, this is true even about the upper limit given by a scientist who reasonably believed that stopping his research might lead to Nazis ruling the world.

When there is significant uncertainty about the risks, it is common to request a moratorium.

In each of the three cases where there was no clear consensus about the amount of risk, some portion of the scientific community called for a moratorium to discuss the issue and collect more evidence:

• In the case of recombinant DNA, the moratorium was adhered to by the entire community despite disagreements. A conference established safe guidelines for some research and a prohibition on the most dangerous types of research. Over time, these guidelines were relaxed as recombinant DNA was shown to be mostly safe.

• In the case of Active SETI, concerned community members who called for a moratorium and conference were initially dismissed. Later, there were conferences to discuss the issue, but the community never agreed on a set of guidelines.

• In the case of frontier AI systems, only in the last few months has a moratorium been earnestly requested. No moratorium has yet occurred, but there is a growing discussion among researchers, policymakers, and the public about the potential risks and policies to address them.

By default, scientists prefer keeping discussions of extinction risk within their community.

All five cases contain some degree of interaction between the scientific community and policymakers or the public:

• During the Manhattan Project, there was no public knowledge of the risk of atmospheric ignition, but the situation involved interaction between scientists and government officials. One of the scientists complained that the risk “somehow got into a document that went to Washington. So every once in a while after that, someone happened to notice it, and then back down the ladder came the question, and the thing never was laid to rest.”

• At the Asilomar Conference on Recombinant DNA, 12 journalists were invited, but possibly only out of a fear that the scientists would be accused of a cover-up otherwise.

• Risk assessments for large-ion collider experiments were published only a year before RHIC began to operate, and seemingly only because of unexpected media attention.

• A concerned member of the SETI community considered bringing the issue of Active SETI to journalists but said that he preferred to discuss the issue within the community.

• Discussion of risks from AI has recently grown among policymakers and the public. Previously, there was relatively little discussion of the risks outside the research community.

  Subscribe now

What Do I Mean By ‘Overhang’?

Hardware Overhang for AI

One major concern about pausing AI development, from a purely safety perspective, is the possibility of hardware overhang.1 Here is the concern as I understand it:

Suppose a policy were put in place tomorrow that banned all progress in AI capabilities anywhere in the world for the next five years.2 Afterwards, the ban would be completely lifted. 

Hardware would continue to progress during this AI pause. Immediately after the pause ended, it would be possible to train new AI systems using significantly more compute than was previously possible, taking advantage of the improved hardware. There would be a period of extremely rapid growth, or perhaps a discontinuity,3 until the capabilities returned to their previous trend. Figure 1 shows a sketch of what we might expect progress to look like.

Figure 1: What AI progress might look like if there were a temporary pause in capabilities progress. The ‘overhang’ is the difference between what AI capabilities currently are as a result of the pause and what AI capabilities could be if the pause had never been enacted, or were completely lifted.

It might be worse for safety to have a pause followed by extremely rapid growth in capabilities than to have steady growth in capabilities over the entire time frame. AI safety researchers would have less time to work with cutting edge models. During the pause, society would have less time to become accustomed to a given level of capabilities before new capabilities appeared, and society might continue to lag behind for some time afterwards.

If we knew that there would be catch-up growth after a pause, it might be better to not pause AI capabilities research now and instead hope that AI remains compute constrained so progress is as smooth as possible.

We do not know if there would be extremely rapid growth after a pause. To better understand how likely hardware overhang would be, I tried to find examples of hardware-overhang-like-things for other technologies.

Overhang for Other Technologies

Many technologies have an extremely important input - like GPUs/TPUs for AI, or engines for vehicles, or steel for large structures. Progress for these technologies can either come from improvements in the design of the technology itself or it can come from progress in the input which makes it easier to improve the technology. For AI, this is the distinction between algorithmic progress and hardware progress.

I am being purposefully vague about what ‘progress’ and ‘input’ mean here. Progress could be in terms of average cost, quantity produced, or some metric specific to that technology. The input is often something very particular to that technology, although I would also consider the general industrial capacity of society as an input. The definition is flexible to include as many hardware-overhang-like-things as possible.

It is possible for there to be a pause in progress for the technology itself, perhaps due to regulation or war, without there being a pause in progress for the inputs. The pause should be exogenous: it is a less interesting analogy for AI policy if further progress became more difficult for technical reasons particular to that technology.4 It is possible for AI progress to pause because of technical details about how hard it is to improve capabilities, and then for a new paradigm to see rapid growth, but this is a different concern than overhang due to AI policy. Exogenous pauses are cases where we might expect overhang to develop.

Examples of Overhang

Methods

To find examples of overhang, I looked in the data for our Discontinuous Progress Investigation5 and in the Performance Curve Database.6 The Performance Curve Database contains 135 entries, most of which include both average cost and production quantity as a function of time. The Discontinuous Progress Investigation contains 21 case studies. The specific metric or metrics for each case study is different, but it should be a proxy for how ‘good’ a particular technology is and it should have sufficient historical data to establish a trend. Most of these only include record performance on this metric. In total, I looked at several hundred graphs of some notion of progress for a technology.

I initially looked for data which had a similar shape to Figure 1: a clear trendline showing growth, followed by a pause, followed by rapid growth and a return to the previous trendline. If I found this pattern, I would then try to figure out the cause of the pause and the subsequent growth. If the pause was caused by an external factor, and the subsequent growth was driven by having better inputs, then this would be an example of overhang. 

I also thought of technologies which were banned, and for which the ban was later completely reversed, to check whether they follow the pattern predicted by overhang.

The Closest Example I Found: Land Speed Records

Figure 2 shows the fastest any vehicle traveled on land over a distance of either 1 km or 1 mi from 1900 to 2000, according to Wikipedia.

Figure 2: Historic land speed records in mph over time. Speeds on the left are an average of the record set in mph over 1 km and over 1 mile. The red dot represents the first record in a cluster that was from a jet propelled vehicle.7 The trendline shown is fit to the data from 1920-1940 and extrapolated.

This data shows several pauses. The most interesting one for this blog post is from 1939 to 1963. The first pause had similar growth before and after the pause, with no catch-up growth, and the last pause has less data to establish a clear trendline after the pause.

In 1939, progress in the land speed record abruptly stopped. No new records were set until 1947, and significant progress on this metric did not resume until 1963. Then there was extremely rapid progress over the next two years. This looks like catch-up growth, although it did not quite reach the previous trendline. It also was not sustained, and the catch-up growth was followed by another pause.

The explanation for this pattern follows what we would expect for overhang. The important input is the engine: having a more powerful engine is extremely important for making a car go fast. The exogenous source of the pause was WWII, which caused industrial capacity to shift towards more strategically important technologies. During WWII, engine progress continued. In particular, jet engines were developed for aircraft. Jet engines were initially reserved for military aircraft, and were not widely available for private buyers until after 1958, when the Boeing 707 entered service. In 1963, the land speed record was broken by a car with a jet engine for the first time, which began a period of rapid progress.

One other consideration for the land speed record is that it is not strategically or economically important, especially after WWII, so relatively small amounts of resources have been devoted to it. The most expensive record-breaking land vehicles cost millions or tens of millions of dollars.8 If this were a more important technology, someone might have built a car with a jet engine earlier, leading to more continuous growth.

This is the only example I found which had an exogenous pause, and then faster growth after the pause.

Endogenous Pauses: Optical Cables and Particle Accelerators

The data I found that looks most like Figure 1 involves optical cables, and is shown in Figure 3. There is a clear trend, followed by a pause, followed by rapid growth that catches up to the previous trendline. The trend continues for another decade beyond the edge of this graph, through generations 5 & 6.

Figure 3: Bandwidth-distance product in fiber optics alone,9 from Agrawal, 2016.10

This figure comes from a book chapter (Agrawal, 2016) which also explains the history. The cause of the pause was endogenous: it was determined by technical details of the technology itself, rather than something external. The original trendline is actually a combination of S-curves, each of which corresponds to a significant change in the design of the cables, called a ‘generation’ of the technology. The transition between most generations required a single significant change. The transition between generations 3 & 4 required three simultaneous changes to work. This explains why the transition occurred later, and resulted in more rapid growth, than the other transitions.

Particle accelerators have followed a similar pattern as optical cables, as shown in Figure 4. There are periods of slower growth, followed by periods of rapid growth. The slower growth was caused by diminishing returns to existing designs, and the faster growth was caused by new accelerator designs.11 The result is again a combination of S-curves. This pattern can be entirely explained by factors internal to the technology, rather than anything exogenous.

Figure 4: Record center of mass energy for a particle-particle collision in a particle accelerator from 1920 to 2010. Note the three distinct S-curves from about 1930-1945, 1945-1970, and 1970-2010.12

While this is an interesting way that a technology can develop, it is not a precedent for hardware overhang developing as a result of AI policy.

Prohibition of Alcohol

One example of a technology which was banned, and for which the ban was then completely reversed, is the Prohibition of alcohol in the US from 1920-1933. 

The real cost of producing alcohol has fallen over time, as expected for a manufactured good.13 During Prohibition, general industrial practices in the US improved substantially, and alcohol continued to be produced at scale in other countries, likely resulting in continued progress there. This seems like a situation where overhang might develop.

Unfortunately, I have not been able to find data on the cost of producing alcoholic beverages14 before and after Prohibition. The way economic data are reported changed significantly during the early 1900s, so it is difficult to compare data before15 to data after.16 There are data and proxies for the consumption of alcohol before, during, and after Prohibition, shown in Figure 5, which show lower consumption after Prohibition than before. This does not suggest a significant reduction in the cost of alcohol immediately after Prohibition ended.

Figure 5: Alcohol consumption before and after Prohibition (squares, thicker line) and several proxies for alcohol consumption during Prohibition. Plot from Miron & Zwiebel, 1991.17

Other Examples?

These are the best examples of overhang I have been able to find. None of them match the behavior that people seem to expect will happen with hardware overhang.

Most of the data series seem even less like overhang than these. Many do not have clear trendlines, which makes them hard to interpret. Many have pauses followed by similar or slower growth. When there is a discontinuity in technological progress (which is uncommon, but exists), it is more likely to be preceded by accelerating growth than by a pause.

If you are aware of any technology where you think overhang might have occurred, please let me know in the comments below.

Why Are There No Good Examples Of Overhang?

This section is more speculative than the previous section. 

The lack of examples of overhang for other technologies makes me skeptical that this would happen for AI. Catch-up growth for a particular technology after a pause is at least uncommon.

Here are some possible explanations I have considered for the observed lack of overhang. Feel free to suggest more in the comments.

There’s Nothing So Permanent as a Temporary Government Program

One premise of overhang is that a ban on progress for a particular technology would be enacted for a certain amount of time, and then completely ended. This is not how regulations normally work.18

There are lots of things which are regulated, many of which require complicated standards or enforcement mechanisms.19 It is easier to enact a regulation than it is to repeal a regulation. This is known as the ‘regulatory ratchet.’20 Legislatures and regulatory agencies have a mandate to pass new laws or regulations in response to problems. While they also have the power to repeal regulation, there is typically less incentive for them to work on repeals. Sunset laws might make this more symmetric, but they are uncommon.

Even if a ban is initially temporary, it is likely to be extended. The political coalition which was successful at getting the temporary ban enacted will still exist as the ban nears its end date. They were powerful enough to win before, and now they have status quo bias on their side. The industry which opposes the ban likely has less money and influence as a result of the ban. We should expect most new regulation to be permanent, unless its consequences are so visibly bad21 that it empowers a reaction much stronger than the initial movement. Even when a political party makes repealing a particular piece of legislation a central plank of their platform, and then wins major electoral victories, it is still hard to get the legislation repealed.22

When a ban is reversed, it is unlikely to be fully reversed. Bans which are temporary are sometimes enacted to give the relevant community time to figure out what good policies are. The ban ends when these policies are enacted, but the new policies often maintain some restrictions from the previous ban. The Asilomar Conference on Recombinant DNA follows this pattern. In 1974, the National Academy of Sciences (NAS) enacted a temporary ban23 on all experiments involving recombinant DNA, in response to concerns about biohazardous experiments. The Asilomar Conference in 1975 prohibited certain kinds of experiments, and provided recommendations for other experiments. After Asilomar, the total ban was reversed, and replaced with these strict guidelines.24 Some of these guidelines have since been scaled back, but regulation on recombinant DNA remains stricter than it was before the temporary ban.

Dispelling the Hype

Emerging technologies depend on hype. They are often not currently profitable, so they rely on promises of future success to acquire the resources they need. Without the hype, startups would fail and established companies would shift their investments elsewhere.25

Increasing regulations or bans are effective ways to dispel hype, even if they are enacted temporarily. People would (reasonably) expect that they might not be temporary. The ban is a strong signal to the private sector and foreign countries that this technology is not going to be the next big thing and reduces their expected value for pursuing it. The hype seems unlikely to fully return after a ban is lifted.

There are two main resources driven by hype: venture capital and talent.

Venture capital is the most obvious resource fueled by hype. Investors know that startups are promising more than they can currently deliver, but believe that there is a good enough chance that they will be able to deliver in the future that they are worth investing in. If something happens that makes investors think that startups in a field will systematically not be able to deliver, that available capital can quickly shift elsewhere.

Talent responds more slowly than capital. If there is a ban that is expected to last more than a few years, then students who are choosing their careers are less likely to want to go into that field, and people who currently work in the field are more likely to consider switching careers. While venture capital might quickly return after a ban is lifted, re-recruiting talent that has gone elsewhere would take longer.

Reduced Demand for the Input

Overhang assumes that progress for the input will continue to develop during the pause, the same as if the pause did not occur. 

It is not obvious if this would be the case for AI. AI is one of the major markets for GPUs and is the sole market for TPUs. We should expect counterfactually less progress in GPUs & TPUs as a result of AI regulation.

It might be different if the technology which was paused accounts for a small fraction of the total market of the input. Extremely fast cars account for effectively none of the market for jet engines.

For technologies which account for a larger fraction of the demand for their inputs, progress in their inputs is less likely to continue at the same rate if they are paused. They would have less catch-up growth immediately available afterwards.

Conclusion

I do not think that technologies typically develop overhang if they are subject to an exogenous pause. I know of no good examples of overhang from other technologies.

There are several plausible reasons why overhang is uncommon. Most regulations are not temporary, most ‘temporary’ regulations or bans are extended, and when bans are ended, this is often accompanied by stricter regulation than there was before. Temporary bans can dispel the hype surrounding an emerging technology causing venture capital and (more slowly) talent to leave the field. The assumption that progress in the input continues the same as before if one of its major customers gets paused also seems suspect.

As an example of how people do not expect overhang elsewhere, consider nuclear power. Nuclear power was getting steadily cheaper in the 1960s. Then, increasing safety regulation applied to nuclear power caused the learning curve to reverse, and the cost of building new nuclear reactors increased dramatically.26 Nuclear power advocates argue that, if we reduced regulations on nuclear power to a more reasonable level, we could see prices drop by a factor of 10 to return to the construction costs of the 1960s. Nuclear power advocates do not extrapolate the trendline from the 1960s to today and predict that, if this regulation were repealed, nuclear power would suddenly become 10⁶ times cheaper than it is today, which is what we would expect if the overhang model applied. Extrapolating all the way to today is probably unreasonable: there’s a good chance that nuclear power would not have followed the trendline for that long anyway.27 Nuclear power advocates’ optimistic views are less optimistic than what they would expect if they predicted even partial overhang.

Figure 6: Trendline for nuclear power plant construction costs in the US. Data are from Lovering et al.28 and digitized using WebPlotDigitizer.29 The trendline is fit to data from 1963-1966 and extrapolated to the year 2000. Extrapolating to 2023 results in two orders of magnitude of further cost reduction.

I think that hardware overhang can still be worth considering as something bad and plausible, but we should not treat overhang as the default or most likely future if there is a temporary pause in AI development. It seems more likely that AI progress would resume at a similar or slower rate after the ban is lifted, if it is lifted at all.

Thanks to Rick Korzekwa, Harlan Stewart, Zach Stein-Perlman, Rocket Drew, McKenna Fitzgerald, Ryan Kidd, Curt Tigges, and Alex Gray for helpful discussions on this topic.

Since our last newsletter, we reviewed some empirical evidence for AI risk, completed two case studies about standards, and wrote a few other blog posts and wiki pages. Recently we have been busy with ongoing projects, including the 2023 Expert Survey on Progress in AI, which we will share the results of soon.

Research and writing highlights

Case studies about standards

• In response to Holden Karnofsky’s call for case studies on standards that are interestingly analogous to AI safety standards, Harlan and Jeffrey completed case studies for two examples of social-welfare-based standards: 

  • Jeffrey wrote a report about institutional review boards for medical research. IRBs were formed after some unethical medical experiments were publicized. The government does not regulate medical research itself, but instead requires institutions that do medical research to regulate it in compliance with federal rules. The principles of medical ethics developed in the US spread widely across the world with little effort from the US government.

  • Harlan wrote a report about Green Seal and SCS, the first eco-labeling programs in the US. These programs failed to transform the consumer market as they initially intended. However, by establishing themselves as experts at the right time, they may have influenced the purchasing behavior of institutions, as well as the creation of other eco-labeling programs.

Empirical evidence for misalignment and power-seeking in AI

• Rose Hadshar investigated the evidence for future AI being misaligned, and for it being power-seeking, focusing on empirical evidence. Her project had several outputs:

  • A database of empirical examples of evidence for different AI behaviors, such as power-seeking, goal misspecification, and deception

  • A report reviewing some evidence for existential risk from AI, focused on empirical evidence for misalignment and power-seeking

  • A blogpost outlining some of the key claims that are often made in support of the argument that AI poses an existential threat

  • A series of interviews of AI researchers about their views on the strength of the available evidence for risks from AI.

Feasibility of strong AI regulation

• To better understand how proposals to regulate AI hardware might work, Rick and Jeffrey compiled a list of existing product regulations that limit the amount of a product people can own or require licensure or registration. Products on the list include fireworks, chickens, and uranium.

• Jeffrey wrote a blogpost explaining why he believes that strict regulation of AI development is plausible without disrupting progress in other areas of society.

Miscellany

• Zach wrote Cruxes for overhang, a blogpost identifying some of the crucial considerations for the possibility that slowing AI progress now could lead to faster progress later.

• Jeffrey compiled a list of possible risks from AI.

• Zach wrote a blogpost summarizing some takeaways about US public opinion on AI, based on collected survey data.

• Jeffrey compiled a list of examples of progress for a particular technology stopping, showing that a particular technology can have periods of both progress and stagnation.

Ongoing projects

• Katja, Rick, and Harlan worked with outside collaborators on the 2023 Expert Survey on Progress in AI. We are looking forward to sharing the details and results soon.

• Zach is working on a project on best practices for frontier AI labs to develop and deploy AI safely.

Recent references to AI Impacts research

• Data from the 2016 and 2022 Expert Surveys on Progress in AI was referenced in the UK government’s discussion paper Frontier AI: capabilities and risks, an open letter advocating for an AI treaty, an article from the Telegraph titled ‘This is his climate change’: The experts helping Rishi Sunak seal his legacy, and a paper published by the Centre for the Governance of AI titled Risk Assessment at AGI Companies: A Review of Popular Risk Assessment Techniques From Other Safety-Critical Industries

• Katja was quoted in a recent press release from California State Senator Scott Wiener, as well as an article in Vox titled AI is a “tragedy of the commons.” We’ve got solutions for that

Funding

Thank you to our recent funders! Including Jaan Tallinn, who gave us a $179k grant through the Survival and Flourishing Fund, Future of Life Institute, who gave us a $162k grant through the Survival and Flourishing Fund, and Open Philanthropy who gave us a $150k grant in support of the 2023 Expert Survey on Progress in AI.

We are still seeking funding for 2024. If you want to talk to us about why we should be funded or hear more details about our plans, please write to Rick or Katja at [firstname]@aiimpacts.org. If you'd like to donate to AI Impacts, you can do so here. (And we thank you!)

Image credit: DALL-E 3

Subscribe now

In this report, Rose looks into evidence for:

• Misalignment,1 where AI systems develop goals which are misaligned with human goals; and 

• Power-seeking,2 where misaligned AI systems seek power to achieve their goals.

Rose found the current state of this evidence for existential risk from misaligned power-seeking to be concerning but inconclusive:

• There is empirical evidence of AI systems developing misaligned goals (via specification gaming3 and via goal misgeneralization4), including in deployment (via specification gaming), but it's not clear to Rose whether these problems will scale far enough to pose an existential risk.

• Rose considers the conceptual arguments for power-seeking behavior from AI systems to be strong, but notes that she could not find any clear examples of power-seeking AI so far.

With these considerations, Rose thinks that it’s hard to be very confident either that misaligned power-seeking poses a large existential risk, or that it poses no existential risk. She finds this uncertainty to be concerning, given the severity of the potential risks in question. Rose also expressed that it would be good to have more reviews of evidence, including evidence for other claims about AI risks5 and evidence against AI risks.6

Subscribe now

Epistemic Status: My intuition strongly disagrees with other people here. I hope to explain my intuition, and provide enough historical evidence to make this intuition at least plausible.

Introduction: An Intuition Pump

Suppose you told someone in 1978 that no new nuclear power plants would be built in the US until 2023.1 This would probably be very surprising. Nuclear power was supposed to be the power of the future.2 The Nuclear Regulatory Commission had only been created 3 years earlier.

Given this information, someone in 1978 might predict that something terrible was about to happen. Maybe a nuclear war between the USA and USSR that destroys America’s industrial capacity. Maybe economic collapse due to overpopulation or global warming. Maybe an Orwellian police state in time for 1984, or a World Authority designed to regulate nuclear weapons that got out of hand.3

None of this happened. Instead, the Nuclear Regulatory Commission increased the regulatory ratchet4 until building new nuclear power plants became uneconomical. These regulations only applied to the USA, but they seem to have significantly impacted nuclear power research globally. Countries that are building new nuclear power plants are still using designs that were developed before 1970.5

Regulation on nuclear power probably did slow US economic growth over the next 45 years compared to the counterfactual.6 But the past 45 years have hardly been catastrophic. Economic growth and innovation did continue, driven by other industries.

Consequences of Stopping AGI?

Some people involved in the debate about slowing or pausing AI seem to think that successfully stopping AI progress over the long term would likely lead to death or dystopia:

Either we figure out how to make AGI go well or we wait for the asteroid to hit.

- Sam Altman7

If we don’t get AI, I think there’s a 50%+ chance in the next 100 years we end up dead or careening towards Venezuela.

- Scott Alexander8

I think we should be quite worried that the global government needed to enforce such a ban would greatly increase the risk of permanent tyranny, itself an existential catastrophe.

- Nora Belrose9

It seems likely that we would need to create a worldwide police state, as otherwise [an indefinite AI pause] would fail in the long run.

- Matthew Barnett10

It feels to me like this is the same sort of mistake that our hypothetical person from 1978 made. It might seem like AI will be an extremely important thing in the future, and so something dramatic would have to happen in order to prevent it. I think that we should put more probability on the boring future where regulation stifles this one field, while the rest of society continues as it had before.

This seems like an important disagreement. If you think that our descendants’ lives will be pretty good, and getting better if not unimaginably quickly, then stopping AI progress might be worth it for them. If you think that our descendants’ future will be “short and grim,”11 then they might be less of a consideration when deciding whether to take this risk now.

Specific Concerns

Scott Alexander mentions several specific concerns that cause him to be pessimistic about a future without AI progress. Each of them seems like a real problem that people now and in the future should be trying to solve. We should be improving biosecurity,12 promoting economic growth,13 spreading democracy & freedom,14 and giving people hope in future generations. But none of these things seem even close to a 50% chance of causing death or dystopia in the next 100 years. It would not be worth accepting existential risk from AI, which Scott Alexander estimates as having a ~20% chance of causing human extinction, to avoid these. 

Both Nora Belrose and Matthew Barnett are concerned that a global police state would be needed to enforce a long term ban on AI progress. This position does not seem uncommon in the AI safety community. The concerns are that research might shift to locations with fewer regulations, and that algorithmic progress will make AGI possible on a personal computer. The only way to avoid AGI then is a massive expansion of global government power. 

Other Historical Examples

I do not think that these concerns have been realized with other technologies. 

Regulations in one industry do not stop progress in all other industries. People in the Bay Area likely underestimate the importance of emerging technologies other than AI, or software more generally, because information technology is disproportionately important in the local economy.15 I would similarly expect that people living in Detroit in 1950 would underestimate the importance of emerging technologies other than cars. Lots of progress is still possible without AI. Two emerging technologies I am particularly excited about are fusion and space colonization.

Regulations in one country can stop progress in a single industry. Progress stopping in a particular industry is not that uncommon.16 Most innovation in a particular industry is done in one or a few cities. These clusters of innovation are difficult to build and maintain, so if one is crushed by regulation, it typically does not just move to another country. On a broader scale, some countries are much more innovative than others. In most industries, including heavily regulated ones, the USA is clearly more innovative than (most of) Europe or East Asia, which are much more innovative than the rest of the world. A lot has to go right: a high standard of living, an educated populace, the rule of law, the possibility of future profit, available capital, and a culture that encourages innovation. Countries which flaunt international regulations or norms typically do not attract innovation. Once a technology exists, it is much easier for other countries to copy it. The designs and skills needed already exist, and the benefits of the technology are clear. Regulation to prevent innovation is much easier than regulation to prevent proliferation.

I have previously investigated some Resisted Technological Temptations,17 or technologies where a long term pause has been achieved through our current institutions:

• Nuclear power, discussed above.18

• Geoengineering is not explicitly illegal, but opposition from scientists and activists has prevented even research from being done.19

• Vaccine development is heavily regulated in Western countries. During the recent pandemic, both Russia and China relaxed some of these restrictions and approved vaccines before the West. The resulting vaccines were less effective, because the best medical research is still located in the West. In particular, human challenge trials have been regulated into almost non-existence.20

• Nuclear weapons are sometimes mentioned as a technology where regulation has failed to prevent their spread. I think the evidence on this example is mixed. There are about 10 countries which have nuclear weapons (much fewer than the number which could), but only 2 developed them independently: the USA and France. Cutting edge research has not moved to countries which are not party to the Non-Proliferation Treaty. India, Pakistan, and North Korea seem to have similar capabilities as the USA or USSR in the 1940s and 50s.21 Testing bans have probably also helped keep nuclear weapons from becoming increasingly powerful: the most powerful bomb ever was detonated in 1961.

• Biological weapons have some of the strongest treaties and taboos against their development or use,22 and no country openly has a biological weapons program. A problem with this example is that the USSR signed treaties against developing biological weapons - and then continued developing them.

• Various nuclear technologies, like atomic gardening, using nuclear explosions in construction, or Project Orion, have been proposed but not developed.

• Cloning the most effective soldiers has never been done.

• Genetic modification of humans has been done by one researcher in China, before he was arrested.

• It is unclear whether colonialism counts as a technology. The Ming dynasty’s decision to stop their Treasure Fleets in the early 1400s delayed colonialism globally by about 50 years and may have impacted China’s developmental trajectory for centuries.

• Bell Labs invented or discovered the transistor, charged-coupled device, photovoltaic cell, information theory, Unix, C, and the cosmic microwave background radiation between 1945-1980.23 After the Bell System was broken up by antitrust laws in 1982, the research community there fragmented. It seems plausible that there are some technologies that were not invented because this unusually prolific center of innovation was destroyed.

Most technologies are not banned, nor have had their progress stifled by regulation. Most technologies are also not as scary as AI: I have a hard time imagining how solar panels or ballpoint pens could constitute an x-risk. Scary-sounding technologies, like weapons of mass destruction or some kinds of medical research, often do face bans or regulations that make their development no longer worth it, and the bans sometimes work.

I don’t want to say that effective bans on scary-sounding technologies happen by default. When they do work, they are the result of a concerted effort. But making a ban on a new, potentially dangerous technology seems very doable without disrupting the rest of society.

Maybe AI Will Be Different

While this post is mostly about historical precedents of other technologies being stopped, it seems worth saying a few words on AI in particular. There are several reasons why AI might be different from other technologies: 

1. AI research is easier to do remotely than other emerging technologies. 

2. Once an AI system is created, it can be transmitted easily, as software.

3. Simple economic models suggest that powerful AI would be extremely economically advantageous to whoever adopts it.

All technologies are different. Some differences make regulations easier or harder, but none of these feel so different that they make regulation impossible:

1. Law enforcement can enforce the law based on where the research is done or where the researchers live. The USA in particular has an expansive view of where its law can apply.24

2. Most proposed regulations focus on the hardware required to train powerful AI.

3. Policy makers do not know this. They know that someone is telling them this. They definitely do not know that they will get the economic promises of AGI on the timescales they care about, if they support this particular project. These promises are not that distinguishable from other technologies’ hype.25

There are also ways in which regulating AI is easier.

There are multiple stages of the supply chain where there are only one or a few companies in the world capable of cutting edge work. There are only a few actors who need to coordinate in order for regulation to be effective.

Current leading AI models require a lot of compute, which is capital intensive and easy to keep track of. This might change with enough improvements from algorithmic efficiency. But we should expect algorithmic progress to dramatically slow down in response to a long term pause on AI as capital and talent moves to other industries.

Lots of substances and items are regulated, and the details of this regulation vary widely based on what it is and what the government is trying to avoid.26 Regulating GPUs will have some unique challenges, but does not seem impossible under our current institutions.

How Long of a Pause?

Most of the historical evidence is for global pauses that have lasted about 50 years. This is useful evidence for discussing a 100 year pause. If “long term” means 1,000 years, then there is much less historical evidence. Matthew Barnett has argued that a regulatory ratchet within existing institutions might accomplish a 50 year pause in AI research, but something more dramatic would be needed for a 1,000 year pause.

I am skeptical that a global police state would be easier to maintain than more normal regulations for 1,000 years. My model for how to sustain institutions on this time scale is:

1. Build an institution that lasts for a generation.

2. Convince the rising generation that this institution is a good thing to maintain.

If you fail at (2), then it does not matter what institution was built. If not even the elite believe that the police state is a good thing, then it will not maintain itself.27 An institution which has less hard power, but is better at getting people to believe in it, is more likely to last 1,000 years.

Conclusion

Building AGI is an extremely uncertain endeavor. It might lead to Our Glorious Future. It might lead to human extinction. It might not even be possible. If we decide to not try to build AGI, the future seems much less uncertain. Society will continue to be clearly not optimal, but also far from dystopian. Making scientific, technological, economic, social, and political progress will continue to be hard, but people will continue to do it. We can continue to hope for at least marginal improvements for our children, and they for their children, long into the future.

It should not be surprising if a scary-sounding technology faces a regulatory ratchet that slows and then stops all progress in that field. This is not death or dystopia - it’s normal.

Thanks to Aaron Scher, Matthew Barnett, Rose Hadshar, Harlan Stewart, and Rick Korzekwa for useful discussion on this topic.

Preview image by Theen Moy: https://www.flickr.com/photos/theenmoy/8003177753.