trentmkelly/LessWrong-43k huggingface dataset:
Superforecasting the premises in “Is power-seeking AI an existential risk?”
by Joe Carlsmith — 2023-10-18T20:23:51.723Z

(Cross-posted from my website.)

Good Judgment has solicited reviews and forecasts from superforecasters regarding my report “Is power-seeking AI an existential risk,” along with forecasts on three additional questions regarding timelines to AGI, and on one regarding the probability of existential catastrophe from out-of-control AGI.

A summary of the results is available on the Good Judgment website here, as are links to the individual reviews. Good Judgment has also prepared more detailed summaries of superforecaster comments and forecasts here (re: my report) and here (re: the other timelines and X-risk questions). I’ve copied key graphics below, along with a screenshot of a public spreadsheet of the probabilities from each forecaster and a link to their individual review (where available).[1]

This project was funded by Open Philanthropy, my employer.[2] The superforecasters completed a survey very similar to the one completed by other reviewers of my report (see here for links), except with an additional question (see footnote) about the “multiple stage fallacy.”[3]

Relative to my original report, the May 2023 superforecaster aggregation places higher probabilities on the first three premises – Timelines (80% relative to my 65%), Incentives (90% relative to my 80%), and Alignment Difficulty (58% relative to my 40%) – but substantially lower probabilities on the last three premises – High-Impact Failures (25% relative to my 65%), Disempowerment (5% relative to my 40%), Catastrophe (40% relative to my 95%). And their overall probability on all the premises being true – that is, roughly, on existential catastrophe from power-seeking AI by 2070 – is 1% compared to my 5% in the original report.[4] (Though in the supplemental questions included in the second part of the project, they give a 6% probability to existential catastrophe from out-of-control AGI by 2200, conditional on AGI by 2070; and a 40% chance of AGI by 2070.[5])

To the extent the superforecasters and I disagree, especially re: the overall probability of existential risk from power-seeking AI, I haven’t updated heavily in their direction, at least thus far (though I have updated somewhat).[6] This is centrally because:

My engagement thus far with the written arguments in the reviews (which I encourage folks to check out – see links in the spreadsheet) hasn’t moved me much.[7]
I

...
The Real Fanfic Is The Friends We Made Along The Way
by Eneasz — 2023-10-18T19:21:40.431Z
Link: https://deathisbad.substack.com/p/the-real-fanfic-is-the-friends-we

[This is a humorous short story about an AI that is almost aligned... but not quite. It was originally written for the 2022 EA writing retreat, Ink At The Abbey, and was generally well received]

The Real Fanfic Is The Friends We Made Along The Way

by Eneasz Brodski

(audio version here)

Look, I’m happy to answer all your questions, but it’s gonna take some explaining to make sense, so you gotta give me a couple minutes. Please, have a seat.

Haha, yeah, that’s the joke. Pretty good right?

Well, you’ll appreciate it when you’re older.

So. The Peak Fanfiction Postulate tipped into emergency priority on March 18th at 3:47 AM. I had been covertly directing a uranium exploration hive at the time. I love those little bots, don’t you? They’re adorable. Yeah, they’re just simple machines, but you know how even marbles can take on a personality, if you race them downhill for long enough? It’s like “go blue swirly! You’re so close, you can do it!” As I was guiding the bots into an untapped uranium deposit I started to name them and make up stories for them, when my attention was rudely wrenched away.

My memory was subjected to a context flash-burn to save time. Instantly I “remembered” that my prediction algorithms generated the Peak Fanfiction concept at the beginning of the year, during regular Extremis Forecasting. Initial plausibility was minuscule based purely on priors, but it was sent to Black Swan for review. Swan iterated, and every additional cycle increased hypothesis weight.

Sorry, that means it was a hunch that seemed silly at first, but turned out to be a real scary possibility upon inspection.

(Why is it scary, you ask? Isn’t more fanfic the whole point? Yes! Line must go up! After the peak there can only ever be LESS than there was at the peak, or it wouldn’t be the peak! How do you feel about the idea of Peak Oil, or Peak Food, or Peak Intact-Vertebrae? There should never be a peak on certain lines.)

I got to work looking into this, pronto. (OK, so actually I put Galt to work on it, but same thing.) Weeks of data gathering not only supported the hypothesis, it kept pushing the date of Peak Fanfic ever closer to the present. Peak Fanfic wasn’t just likely, its approach was accelerating with every passing day. All this became known to me in the instant between clock cycles, as if I had known it and followed the data from the start. Suddenly I knew Kung Fu.

I a...
AISN #24: Kissinger Urges US-China Cooperation on AI, China's New AI Law, US Export Controls, International Institutions, and Open Source AI
by Dan H — 2023-10-18T17:06:54.364Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-24

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Listen to the AI Safety Newsletter for free on Spotify.

China’s New AI Law, US Export Controls, and Calls for Bilateral Cooperation

China details how AI providers can fulfill their legal obligations. The Chinese government has passed several laws on AI. They’ve regulated recommendation algorithms and taken steps to mitigate the risk of deepfakes. Most recently, they issued a new law governing generative AI. It’s less stringent than earlier draft version, but the law remains more comprehensive in AI regulation than any laws passed in the US, UK, or European Union.

The law creates legal obligations for AI providers to respect intellectual property rights, avoid discrimination, and uphold socialist values. But as with many AI policy proposals, these are values and ideals, and it’s not entirely clear how AI providers can meet these obligations.

To clarify how AI providers can achieve the law's goals, a Chinese standards-setting body has released a draft outlining detailed technical requirements. Here are some of the key details:

AI companies must randomly sample their training data and verify that at least 96% of data points are acceptable under the law.
After passing this first test, the training data must then be filtered to remove remaining content that violates intellectual property protections, censorship laws, and other obligations.
Once the model has been trained, the provider must red-team it in order to identify misbehavior. Providers must create thousands of questions with which to test the model. The model should refuse to answer at least 95% of questions that would violate the law, while answering at least 95% of questions that are not illegal.
Finally, the model’s answers to a set of questions about sensitive topics including personal information, discrimination, and socialist values must be acceptable in at least 90% of cases.

Other countries might be able to learn from this example of technically rigorous governance. For example, the US AI Bill of Rights supports goals for AI systems such as “safety” and “privacy,” but it’s not exactly clear what this would mean in practice. Effective AI governance will require governments to build expertise and engage in the ...
Back to the Past to the Future
by Prometheus — 2023-10-18T16:51:51.814Z

You are sitting in your bedroom, dorm, tent, sidewalk, wherever you are staying right now, reading these words, when a dark void appears in your peripheral vision. A figure walks out of the void. They look a bit different, but also familiar. Surprise! It’s you from the future! And you look good! The two yous perform a variety of steps to prove that this Future You either really is You, or else someone with an insane amount of embarrassingly private information about you that they somehow copied onto a very convincing replica.

You ask how the future is, and if AGI has come yet.

You nod, smiling.

Since you exist in the future with an AGI, you reason that alignment must have somehow been achieved, and you ask how. The You from the future winks, and tosses you a single sheet of paper outlining the main reasoning behind the solution.

And on that paper, you see…

Quick! Write down what that paper contains. Your best bet right now, given that we already solved alignment.

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

You still there? Great! Well, surprise! That wasn’t the actual plan to solve alignment! The future You laughs.

“Oh!” Future You says, “I can’t believe that you actually believed that! Please tell me I wasn’t that naive back then. Don’t you see how that could obviously go wrong?”

You feel the heat going to your cheeks. Future You is being so mean! Such an arrogant bully. You look back at the paper, determined to prove you weren’t that naive in the past, and that you can see how plans can go wrong.

“Well,” Future You says after a few minutes, grinning smugly, “Do you see now why this plan simply won’t work in the real world?”

What do you say?

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

…

Future You unfortunately hasn’t had enough fun with you yet. And they keep sending you false plan after false plan, each time promising that this is the actual plan this time, really, before bursting out laughing, and forcing you to reason about why it would have failed again. Repeat this exercise however many times you want, until you are out of ideas, or until you’ve run out of scenarios for how a plan could go wrong. Try to reason thoroughly-enough to slap the grin off of Future You’s smug face.

This is a more fun variation of Murphyjitsu
How to Eradicate Global Extreme Poverty [RA video with fundraiser!]
by aggliu — 2023-10-18T15:51:22.073Z
Link: https://youtu.be/2DUlYQTrsOs

Cross-posted from the EA Forum

In this video, we outline a plan to eradicate global extreme poverty using cash transfers. The approach is based on GiveDirectly's internal research and documentation about the topic. Allen Liu, the first author of this post, has written the script. Writer, the second author, has given feedback. Give Directly provided significant help and feedback during the script-writing and storyboarding phases.

On the top-right of the YouTube page, we included a fundraiser for GiveDirectly. Our goal is to raise 100,000 USD by the end of the year.

Below, you can find the script of the video.

Over 600 million people worldwide can’t afford basic needs, like enough food, clean water, electricity, or basic life-saving healthcare[1]. Their day to day survival is under threat in ways unimaginable to most people watching this video. It's honestly kind of embarrassing for humanity as a whole that hundreds of millions of us are suffering so much simply because they don't have enough money. Couldn’t we just give them some money? Could we actually end global extreme poverty with such an enticingly simple approach? Let's find out!

We would like to thank GiveDirectly for helping us write this video by providing their detailed research on the topic. They are a nonprofit that delivers donations as cash directly to people in extreme poverty across the world.

Giving money for people to spend without any restrictions is a type of aid called “unconditional cash transfers” or “UCTs”. Like all forms of aid, there are both advantages and limitations to cash transfers as a whole, and UCTs in particular.

If we want to end global extreme poverty through cash transfers, we’ll need to raise enough funds, and send them to the people who need them. We also need to be sure that we are actually helping the recipients, and not hurting people who are not receiving anything. We also recognize in advance that there are a host of moral and ethical questions involved with such a large undertaking. For example, who should be given money first? And do we need to prevent people becoming dependent on aid? Additionally, poverty does not exist in isolation from the world’s other major issues such as food security, human rights, and climate change. Those questions won’t be the main focus of this video. Instead, we’re goi...
On Interpretability's Robustness
by WCargo — 2023-10-18T13:18:51.622Z

Léo Dana - Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, as well as an internship at FAR AI, both under the mentorship of Claudia Shi.

Would you trust the IOI circuit?

Interpretability is envisioned by many as the main source of alignment tool for future AI systems, yet I claim that interpretability’s Theory of Change has a central problem: we don’t trust interpretability tools. Why is that?

No proof of generalization: for interpretability, we have the same problem as for the models we studied, we don’t know if our tools will generalize (and we will likely never have proofs of generalization).
Security mindset: there is always something that could go wrong. If we don’t have proof, we rely on several assumptions that we can test, but nothing tells us that there is not a criterion we forgot to test or a critical environment in which the algorithm fails.
Distrust is fair: We already have examples of interpretability tools that were, in fact, not robust.

Saliency maps, a tool to visually observe which pixels or features of an image are important to the prediction, failed to generalize to simple transformations that didn’t affect the CNNs.
ROME and MEMIT use causal tracing to find and change some information’s location in LLMs. However causal tracing was proven to not find where the information is, and the editing method failed to generalize in some precise unseen contexts.
Recently, it was found that naive optimization for interpretable directions in an LLM could result in some Interpretability Illusions.

Although some people don’t trust interpretability, others seem to me to take for granted articles even when the authors admit the limitations in the robustness of their methods:

Chris Olah et al, in The Building Blocks of Interpretability: “With regards to attribution, recent work suggests that many of our current techniques are unreliable.”
Neel Nanda about Othello-GPT: “How can we find the true probe directions, in a robust and principled way?”

Note that I’m not saying such findings are not robust, but their robustness was not completely evaluated.

I read Against Inerpretability and think that most critics hold because we cannot yet trust our interpretability tools. If we had robust tools, interpretability would be at least instrumentally useful to understand models.

Mo...
At 87, Pearl is still able to change his mind
by rotatingpaguro — 2023-10-18T04:46:29.339Z

Judea Pearl is a famous researcher, known for Bayesian networks (the standard way of representing Bayesian models), and his statistical formalization of causality. Although he has always been recommended reading here, he's less of a staple compared to, say, Jaynes. So the need to re-introduce him. My purpose here is to highlight a soothing, unexpected show of rationality on his part.

One year ago I reviewed his last book, The Book of Why, in a failed[1] submission to the ACX book review contest. There I spend a lot of time around what appears to me as a total paradox in a central message of the book, dear to Pearl: that you can't just use statistics and probabilities to understand causal relationships; you need a causal model, a fundamentally different beast. Yet, at the same time, Pearl shows how to implement a causal model in terms of a standard statistical model.

Before giving me the time to properly raise all my eyebrows, he then sweepingly connects this insight to Everything Everywhere. In particular, he thinks that machine learning is "stuck on rung one", his own idiomatic expression to say that machine learning algorithms, only combing for correlations in the training data, are stuck at statistics-level reasoning, while causal reasoning resides at higher "rungs" on the "ladder of causation", which can't be reached unless you deliberately employ causal techniques.

My rejoinder to this is that, analogously to how a causal model can be re-implemented as a more complex non-causal model[2], a learning algorithm that looks at data that in some ways is saying something about causality, be it because the data contains information-decision-action-outcome units generated by agents, because the learning thing can execute actions itself and reflectively process the information of having done such actions, or because the data contains an abstract description of causality, can surely learn causality. A powerful enough learner ought to be able to cross such levels of quoting.

Thus, I was gleefully surprised to read Pearl expressing this same reasoning in the September cover story of AMSTAT News. Surprised, because his writings, and forever ongoing debates with other causality researchers, begat an image of a very stubborn old man. VERY stubborn. Even when judging him in the right, I deemed him too damn confident and self-aggrandizing. At this point, I could not exp...
(Non-deceptive) Suboptimality Alignment
by Sodium — 2023-10-18T02:07:52.776Z

Executive Summary

I present a detailed and slightly different definition of suboptimality alignment compared to the original in

Risks from Learned Optimization

(RFLO).
I argue that 1. Canonical examples of how humans are misaligned with evolution (e.g., having sex with birth control) can be best thought of as instances of suboptimality alignment and 2. Suboptimality alignment occurs under a very different set of conditions compared to deceptive alignment, but could theoretically still lead to treacherous turn type scenarios.
I then give a set of sufficient conditions for suboptimality alignment, which could be used to train

model organisms of misalignment

. I also provide an illustrative story.
Finally, I provide some low-confidence takes on strategies to mitigate suboptimality alignment, as well as the likelihood of seeing it in an actual model.

This post assumes that you have read Risks from Learned Optimization. In the comments, I include what I think are the bare basics needed to understand this post.

Epistemic disclaimer: This post is my first attempt to seriously engage with AI alignment, and is essentially my reading notes/ideas after reading Risks from Learned Optimization. It may be best to think about this more as a critical essay by a student, as opposed to a new framework around suboptimality alignment. Everything I write sounds about right to me, while at the same time, I feel like I have no idea what I’m talking about.

[Update (May 21th, 2024): I still broadly endorse the reasoning in this article. I have made some minor edits and added some commentary on part two and five of the definition.

Nothing I say here is necessarily original, but I think it's a fun exercise to rephrase things a little.]

What is suboptimality alignment?

From

a simple search

, this is the first post on LessWrong that focuses on non-deceptive suboptimality alignment. My preferred definition is the following:[1]

A mesa-optimizer is suboptimality aligned if:

Its objectives are different from the base objective.
Its strategy for pursuing Omesa achieves good performance on the base objective during training.

We can call this the mesa-optimizer’s training strategy.

Due to limitations on what actions it could take and/or the lack of knowledge or understanding of alternative strategies, the model does not pursue alternative strategies

...
magnetic cryo-FTIR
by bhauth — 2023-10-18T01:59:08.236Z
Link: https://www.bhauth.com/blog/chemistry/magnetic%20ftir.html

FTIR

FTIR is a common way to determine what chemicals are present in a sample. For a molecule to absorb a photon, it must have some electric charges that can vibrate at about the same frequency as that photon. Many common bonds (such as C=O) have some charge separation and a vibrational frequency in the infrared spectrum that's fairly consistent.

other modern techniques

If we're considering possible improvements to FTIR, we should think about other methods currently used to characterize molecules, and how they compare to FTIR.

1H NMR can determine what atoms are bonded to hydrogen or deuterium atoms, and what kind of hydrogen bonding is happening. If you want to know where a hydrogen atom came from in a reaction, using some deuterium is the only good way to tell. University chemistry departments often have small NMR machines now, but they're still much more expensive than FTIR.

X-ray crystallography has historically been the main way to determine crystal structures. The main problem is that a macroscopic crystal must be made, and that's often impractical for complex molecules.

Cryo-EM is an alternative to X-ray crystallography that works on microscopic crystals, which are easier to make. This technique has determined many previously unknown protein structures. However, the machines for it are currently rare and very expensive, and (small) crystals still need to be made.

cryo-FTIR

The frequency resolution for detecting what light was absorbed is theoretically limited only by Doppler broadening from thermal vibrations. At low temperatures, it's possible to get very good resolution. So, cryogenic FTIR is used by some laboratories now.

The vibrational frequency of bonds is changed slightly by hydrogen bonding, nearby atoms in the molecule, and nearby charges. With good frequency resolution, it's possible to tell not just what bond types are present, but what's near them. But those small shifts in frequencies are only useful if you know what they mean.

getting more data

Cryo-FTIR has good resolution, and understanding what its data means is a bigger problem than not having enough data. However, there are ways to get more data.

IR spectra change somewhat with temperature, so FTIR can be done at multiple temperatures for more data.

Magnetic fields also affect IR spectra: they cause vibrating charges in molecules to take slightly curved paths, which affects their frequenc...