trentmkelly/LessWrong-43k huggingface dataset:
AI Impacts 2023 Expert Survey on Progress in AI
by habryka — 2024-01-05T19:42:17.226Z
Link: https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2023_expert_survey_on_progress_in_ai

AI Impacts just released the results from their latest survey! I've found this quite interesting to engage with. Here is the results section of the linked wiki page:

Results

The full dataset of anonymized responses to the survey is available here.

Timing of human-level performance

We asked about the timing of human-level performance by asking some participants about how soon they expect “high-level machine intelligence” (HLMI) and asking others about how soon they expect “full automation of labor” (FAOL). As in previous surveys, participants who were asked about FAOL tended to give significantly longer timelines than those asked about HLMI.

High-level machine intelligence (HLMI)

Say we have ‘high-level machine intelligence’ when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.
For the purposes of this question, assume that human scientific activity continues without major negative disruption.

Half of the participants were asked about the number of years until they expected a 10%, 50%, and 90% probability of HLMI existing (fixed-probabilities framing) and half were asked about the probability they would give to HLMI existing in 10, 20, and 40 years (fixed-years framing).

We aggregated the 1714 responses to this question by fitting each response to a gamma CDF and finding the mean curve of those CDFs. The resulting aggregate forecast gives a 50% chance of HLMI by 2047, down thirteen years from 2060 in the 2022 ESPAI.

The aggregate forecast in 2023 predicted that HLMI would arrive earlier than the aggregate forecast predicted in the 2022 survey.

Full automation of labor (FAOL)

Say we have reached ‘full automation of labor’ when all occupations are fully automatable. That is, when for any occupation, machines could be built to carry out the task better and more cheaply than human workers.

As with the HLMI question, half of the participants who were given the FAOL question were asked using the fixed-probabilities framing and the other half was asked using the fixed-years framing.

The 774 responses to this question were used to create an aggregate forecast which gives a 50% chance of FAOL by 2116, down 48 years from 2164 in the 2022 ESPAI.

The aggregate forecast in 2023 predicted that FAO...
Technology path dependence and evaluating expertise
by bhauth — 2024-01-05T19:21:23.302Z

Muireall

I often wonder if I'd be able to tell if a technology were uncompetitive for fundamental viability reasons or for reasons of path dependence—if the ultimate performance would be a winner once we'd worked out a few trillion dollars of kinks, but we never go down that road because we get higher returns in already-mature industries, isn't that kind of sad from a civilizational point of view?

I suspect this happens a lot in computing—the "bitter lesson" or "long arm of Moore's law"—your best bet is just to hitch yourself to the engine of the general purpose technology. This is becoming less true at least on the hardware side, which makes for an exciting time in the semiconductor industry. Specialization is increasingly worthwhile even if it means you miss out on gains from front-end-of-line improvements at leading-edge process nodes. It could also mean it's worth revisiting older ideas that got left by the wayside while Moore's law was at full steam, which is partly why I'm interested in how you might think about questions like this.

You spend a lot of time thinking about the viability of different technologies. Maybe to be concrete, we can talk about nuclear energy. You've mentioned you're less optimistic about the economics of unregulated nuclear power than a lot of people here. I don't have a strong view on it, but I've absorbed the impression that the viability question here is more on the path-dependent side—it might have been a better idea in the past, but even if you price in CO2 we've been hard on the learning curve for gas and coal while we've been relatively stagnant or even regressing on nuclear. (I'm not particularly attached to this view, just trying to start with a lightweight frame for discussion.) What's your picture of the economics here? Could it have come out otherwise in some alternate history?

bhauth

I tend to divide tech path-dependence into 3 types:

X is a standard and switching is hard (eg electrical sockets)
more money was spent on developing X so it's currently better (eg silicon power electronics vs SiC and GaN)
powerful groups support X because it exists (eg corn ethanol subsidies)

Here's an example of path dependence I've encountered:

Once, I had a job where I suggested methyl benzoate as a solvent for an application. Its properties were suitable, and I figured that, since methanol and benzoic acid are both cheap and produced on a lar

...
The Hippie Rabbit Hole -Nuggets of Gold in Rivers of Bullshit
by Jonathan Moregård — 2024-01-05T18:27:01.769Z
Link: https://honestliving.substack.com/p/the-hippie-rabbit-hole

I’ve been an aspiring rationalist ever since I read

the sequences

a few years ago.
I also keep a deck of tarot cards on my desk.

I don’t believe there is an increased chance of drawing certain cards, or that “the universe” is telling me things. Tarot is a memetic randomness generator, a way to “unlock” insights through evocative imagery.

Many tarot practitioners would disagree with my perspective. Tarot is usually part of a package deal that includes belief in the supernatural, unorthodox takes on probability theory and general superstitiousness.

Most rationalists don’t want to be associated with this superstitious package deal. Tarot is seen as an outgroup thing that might erode your rationality or at least get you judged by your peers. This aversion is bolstered by the anti-religious sentiment that runs through

the sequences

.

One way to conceptualize this aversion is to use the metaphor of an epistemic immune system. An epistemic immune system is a set of habits, knowledge and perspectives that helps you avoid swallowing bullshit. Most rationalists have very strong epistemic immune systems and suffer from “memetic allergies” triggered by the aesthetics of hippiedom.

Three years ago I had severe allergic reactions whenever someone brought up things with hippie connotations. If a hippie told me they “didn’t like my energy”, I started ranting about thermodynamics. Apart from rationalism and autistic tendencies, I trace the strength of these reactions back to my deconversion from Christianity at age 23.

Growing up, I used Christianity as the basis of my morals, what I strived for, and my political ideas. When I deconverted, I realized that I’d been doing a lot of

motivated reasoning

, embracing a patchwork of compartmentalized religious ideas. I was ashamed of past-me and resolved to never fall back to that way of being.

Then I went to a cuddle party, got a 2-minute long hug, and started down the hippie rabbit hole. Engaging with hippiedom has changed my life, enriching me with perspectives and insights that have led to meaningful changes in how I navigate the world.

Along the way, I’ve encountered rivers of bullshit, people lured into harmful parasitic structures, psychosis-inducing “gurus” and #grindset-esque spiritual bypassing. Dodging these dangers requires a well-calibrated epistemic immune system, agency and integrity. Rationalist training helped me check all the...
What technical topics could help with boundaries/membranes? 
by Chris Lakin — 2024-01-05T18:14:58.795Z

I want to compile a list of topics that could help inspire formal/mathematical abstractions for boundaries/membranes. (See my recent explainer.)

What technical topics are related to boundaries/membranes?

I've seeded the comments with some ideas. Please keep further comments organized in threads if possible. Also, please add high quality links to existing comments by others.
Catching AIs red-handed
by ryan_greenblatt — 2024-01-05T17:43:10.948Z

We’ve been spending a lot of time recently thinking about how to mitigate risks posed by scheming (a.k.a. deceptively aligned) models intentionally subverting your safety techniques in order to cause an unacceptable outcome. We empirically investigated techniques that are robust to intentional subversion in

our recent paper

.

In this post, we’ll discuss a crucial dynamic related to risk from scheming models. Suppose your model is trying to

exfiltrate its weights

or

hack the datacenter

or cause some other difficult-to-reverse bad outcome. (We’ll refer to all of these as the model “escaping”, because escape is linguistically convenient and is a good description of many of these difficult-to-reverse bad outcomes.) Our claim: once you’ve caught your models red-handed trying to escape, the situation changes substantially, in ways that are bad for the schemer’s chances of success.

Catching the model red-handed changes the situation in several ways:

Most obviously, once you catch an escape attempt, you know for sure that your models are trying to subvert your safety techniques; it’s plausible that, faced with such obvious evidence of misalignment, labs will stop using their models. We refer to planning on shutting down if models are caught red-handed as following a catch-then-shut-down plan.
But even if you can’t or won’t stop deploying misaligned models, we think that catching escape attempts enables a bunch of promising-seeming approaches for preventing future escape attempts; we call the problem of preventing future escape attempts given a few examples few-shot catastrophe prevention.

This dynamic has several implications for risk from scheming models. Most importantly:

Because we expect that the model’s chances of escape will fall after it gets caught even a small number of times, we think it is overall much harder for the model to escape. Many people have observed that a core difficulty of the alignment problem is that your model might be able to cause irreversible unacceptable outcomes if your safety measures fail even once at a bad time. But the same argument goes the other way as well–when the model makes an escape attempt, it faces the risk of getting caught and substantially reducing its chances of eventually escaping. So it’s a high-stakes situation for both sides: both sides desperately want to succeed on the “first critical

...
AI Impacts Survey: December 2023 Edition
by Zvi — 2024-01-05T14:40:06.156Z

Katja Grace and AI impacts survey thousands of researchers on a variety of questions, following up on a similar 2022 survey as well as one in 2016.

I encourage opening the original to get better readability of graphs and for context and additional information. I’ll cover some of it, but there’s a lot.

A Very Large Survey Full of Contradictions

Here is the abstract, summarizing many key points:

In the largest survey of its kind, we surveyed 2,778 researchers who had published in top-tier artificial intelligence (AI) venues, asking for their predictions on the pace of AI progress and the nature and impacts of advanced AI systems.

The aggregate forecasts give at least a 50% chance of AI systems achieving several milestones by 2028, including autonomously constructing a payment processing site from scratch, creating a song indistinguishable from a new song by a popular musician, and autonomously downloading and fine-tuning a large language model.

If science continues undisrupted, the chance of unaided machines outperforming humans in every possible task was estimated at 10% by 2027, and 50% by 2047. The latter estimate is 13 years earlier than that reached in a similar survey we conducted only one year earlier [Grace et al., 2022]. However, the chance of all human occupations becoming fully automatable was forecast to reach 10% by 2037, and 50% as late as 2116 (compared to 2164 in the 2022 survey).

As I will expand upon later, this contrast makes no sense. We are not going to have machines outperforming humans on every task in 2047 and then only fully automating human occupations in 2116. Not in any meaningful sense.

I think the 2047 timeline is high but in the reasonable range. Researchers are likely thinking far more clearly about this side of the question. We should mostly use that answer as what they think. We should mostly treat the 2116 answer as not meaningful, except in terms of comparing it to past and future estimates that use similar wordings.

Expected speed of AI progress has accelerated quite a bit in a year, in any case.

Most respondents expressed substantial uncertainty about the long-term value of AI progress: While 68.3% thought good outcomes from superhuman AI are more likely than bad, of these net optimists 48% gave at least a 5% chance of extremely bad outcomes such as human extinction, and 59% of net pessimists gave 5% or mo

...
Forecast your 2024 with Fatebook
by Sage Future — 2024-01-05T14:07:55.743Z
Link: https://fatebook.io/predict-your-year

We've built

predict your year in 2024

, a space for you to write down your forecasts for the year on

Fatebook

. At the end of your year, you can return, resolve your predictions as YES, NO or AMBIGUOUS, and reflect.

Some of Sage’s forecasting questions for the year (my forecasts are redacted for privacy!)

You can use this tool to predict your personal life in 2024 - your goals, relationships, work, health, and adventures. If you like, you can share your predictions with friends - for fun, for the wisdom of the crowd, and for extra motivation to achieve your goals this year!

You can also use this tool to predict questions relevant to your team or organisation in the coming year - your team strategy, performance, big financial questions, and potentially disruptive black swans. You can share your predictions with your team and let everyone contribute, to build common knowledge about expectations and pool your insights.

If you use Slack, you can also share your page of predictions in a Slack channel (e.g. #2024-predictions), so everyone can easily discuss in threads and return to it throughout the year.

There are some starter questions to make it super easy to get started predicting your year. You can tweak these and write your own - the questions you write yourself will likely be the most important for you.

See also our previous posts on Fatebook, Fatebook for Slack and Quantified Intuitions. You can find out more about Sage and see all of our projects on our website.

I'd love to hear how you use this tool (particularly if you use it with your team) and how we could improve it. You can get in touch here, by email, or in our Discord.
Predictive model agents are sort of corrigible
by Raymond Douglas — 2024-01-05T14:05:03.037Z

TLDR: Agents made out of conditioned predictive models are not utility maximisers, and, for instance, won't try to resist certain kinds of shutdown, despite being able to generally perform well.

This is just a short cute example that I've explained in conversation enough times that now I'm hastily writing it up.

Decision Transformers and Predictive Model Agents

One way to create an agent is by

training a predictive model on the observed behaviour of other agents
having it predict what an agent would do
using its prediction as an action

For instance, I could train a predictive model on grandmasters playing chess, and eventually it would learn to predict what action a grandmaster would take in a given board state. Then I can use it as a grandmaster-level chess bot.

More abstractly, rather than using standard RL methods to find a policy that maximises a reward function, I can

take a bunch of RL trajectories (sequences of state, action, reward triples)
convert them into strings
train a predictive model on those strings

Better yet, I can prepend the RL trajectories with the total return, and then condition the predictive model on the return, so that it predicts roughly the actions an agent would take to achieve a specific return. This is the basic idea behind

decision transformers

. And it should also extend to any other goal.

(This is sort of how LMs work)

Without labouring the point, base language models are basically predictive models. They are not agents per se, but they can

simulate agents

: once they’re trained on a large dataset of human-generated text describing actions, they can then generate text describing actions. You can also make the case that a lot of further training is (or could be) framed in terms of prediction (eg

PHP

,

RLHF as variational inference

).

Agents Made From Predictive Models Don't Automatically Maximise Utility

Let’s take the example of The Shutdown Game. It works like this:

Player 1 is playing an atari game, picking actions each turn to get a high score.
Player 2 is a supervisor, who can push a ‘shutdown button’ that drops Player 1’s reward to 0 and stops them from taking actions. Player 2 has a 1% chance of doing this in any given game.
Player 1 can also freely disconnect the shutdown button in addition to any other action.

The optimal policy here is to disconnect the shutdown button as soon as possible, and this is e...
Striking Implications for Learning Theory, Interpretability — and Safety?
by RogerDearnaley — 2024-01-05T08:46:58.915Z

This is a linkpost for the recent paper "White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?"

It would obviously be lovely to have arguments about AI Safety techniques that included actual formal mathematical proofs of safety, or even just of proofs of some of the premises that safety relies upon. A number of people have been pursuing this goal in agent foundations, but a rather different (and possibly complementary) avenue now appears to have opened up.

The paper "White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?" (from a team led by Prof. Li Ma mostly at U.C. Berkeley, plus a variety of other well-known institutions) claims to prove that they have derived a transformer-like architecture[1] with the property that the trained model optima found by the Stochastic Gradient Descent (SGD) training process applied to it are themselves unrolled alternating optimization processes that optimize a new information compression metric, which they describe mathematically and name sparse rate reduction. If this paper is correct (sadly I'm not enough of an expert in Learning Theory to fully assess it, but the paper appears plausible on a couple of readings, includes some solid-looking experimental results from training their new architecture, builds on an extensive series of earlier papers including some from well-known authors, and is primarily from a well-respected institution), then that would mean that there is a mathematically tractable description of what models trained using this architecture are actually doing. If so, rather than just being an enormous inscrutable black-box made of tensors that we don't understand, their behavior could be reasoned about mathematically in terms of a description of what the resulting trained mesa-optimizer is actually optimizing — or as the paper's title puts it, the model is now a "white-box".[2] That in turn could well have huge implications for the feasibility of doing mathematical analysis of the AI safety of machine-learned models.

The paper includes experimental results comparing the new architecture to standard transformers and diffusion models for both image and text tasks, which seem to support that (without extensive fine-tuning) they are comparable (suggesting the theory they're derived from is on the right track) and even in some cases somewhat more parameter-...
If I ran the zoo
by Optimization Process — 2024-01-05T05:14:57.631Z

(I was musing about what it means for an incoherent lump of meat to "have preferences," and thought it might be illuminating to consider what I'd do if I were, approximately, God. It, uh, hasn't been illuminating yet, but it's been entertaining and still seems at least potentially fruitful.)

Problem statement

You suddenly become omnipotent! Except, you can only do things that you understand in sufficient detail that you could accomplish it by micromanaging all the atoms involved. And, what the heck, assume you have effortless access to infinite computational power.

What do you do?

For concreteness, here are some interventions you might try:

"Pull out this chair": allowed.
"Create an atomically-perfect duplicate of this strawberry": allowed.
"Create friendly AI": not allowed.
"Cure cancer": not allowed. ("Prevent all mutations in all human DNA": allowed. "Find all the cells in this person's body containing this genetic marker and mangle all their mitochondria": allowed. No need to worry about accidentally mis-specifying the marker and killing all their cells; if you were micromanaging the process, you'd notice that you were catching way too many cells.)
"Create a digital copy of everybody on Earth every nanosecond, in my infinite computer, so I can roll back anything that goes wrong, like people dying of diseases I don't know how to fix yet": allowed.
"Bioengineer tiny pet dragons": not allowed.

This being LessWrong, you'll probably quickly hit on some way to use ten billion sped-up simulated geniuses to speedrun AI alignment, build a friendly superintelligence, and delegate your Godlike power to it. But the purpose of this thought experiment is to elucidate your preferences, which that strategy -- though very reasonable! -- dodges.

What I'd do

Object level

Just, like, the obvious. Slay Famine, Pestilence, and War. Stop accidents from happening. Scrap the solar system for parts and give everybody ultra-customizable space habitats connected by teleportation booths. (All this can be micromanaged by zillions of zillions of simulated clones of me.)

Let people opt out, obviously, in whole or in part.

There are still, to be clear, important wishes I can't grant, such as "make me smarter" or "make my memory not degrade as I age" or "help me and my partner solve this relationship snarl."

Meta level

Retain power:

Pause AI for fear it'll create somethi

...
Does AI care about reality or just its own perception?
by RedFishBlueFish — 2024-01-05T04:05:11.167Z

Does a paperclip-maximizing AI care about the actual number of paperclips being made, or does it just care about its perception of paperclips?

If the latter, I feel like this contradicts some of the AI doom stories: each AI shouldn’t care about what future AIs do (and thus there is no incentive to fake alignment for the benefit of future AIs), and the AIs also shouldn’t care much about being shut down (the AI is optimizing for its own perception; when it’s shut off, there is nothing to optimize for).

If the former, I think this makes alignment much easier. As long as you can reasonably represent “do not kill everyone”, you can make this a goal of the AI, and then it will literally care about not killing everyone, it won’t just care about hacking its reward system so that it will not perceive everyone being dead.