trentmkelly/LessWrong-43k huggingface dataset:
Consent across power differentials
by Ramana Kumar — 2024-07-09T11:42:03.177Z

I'd like to put forward another description of a basic issue that's been around for a while. I don't know if there's been significant progress on a solution, and would be happy to pointed to any such progress. I've opted to go for a relatively rough and quick post that doesn't dive too hard into the details, to avoid losing the thought at all. I may be up for exploring details further in comments or follow-ups.

The Question: How do you respect the wishes (or preferences) of a subject over whom you have a lot of control?

The core problem: any indicator/requirement/metric about respecting their wishes is one you can manipulate (even inadvertently).

For example, think about trying to respect the preferences of the child you're babysitting when you simply know from experience what they will notice, how they will feel, what they will say they want, and what they will do, when you put them in one environment versus another (where the environment could be as small as what you present to them in your behaviour). Is there any way to provide them a way to meaningfully choose what happens?

We could think about this in a one-shot case where there's a round of information gathering and coming to agreement on terms, and then an action is taken. But I think this is a simplification too far, since a lot of what goes into respecting the subject/beneficiary is giving them space for recourse, space to change their mind, space to realise things that were not apparent with the resources for anticipation they had available during the first phase.

So let's focus more on the case where there's an ongoing situation where one entity has a lot of power over another but nevertheless wants to secure their consent for whatever actually happens, in a meaningful sense.

Lots of cases where this happens in real life, mostly where the powerful entity has a lot of their own agenda and doesn't care a huge amount about the subject (they may care a lot, but maybe not as much as they do about their other goals):

rape (the perhaps central example invoked by "consent")
advertising
representative democracy
colonisation ("civilising" as doing what's good for them)

Our intuitions may be mostly shaped by that kind of situation, where there's a strong need to defend against self-interest, corruption, or intention to gain and abuse power.

But I think there's a hard core of a problem left even if we remove...
How bad would AI progress need to be for us to think general technological progress is also bad?
by Jim Buhler — 2024-07-09T10:43:45.506Z

It is widely believed in the EA community that AI progress is acutely harmful by substantially increasing X-risks. This has led to a growing priority on pushing against work advancing AI capabilities.[1]

On the other hand, economic growth, scientific advancements, and (non-AI) technological progress are generally viewed as highly beneficial, improving the quality of the future provided there are no existential catastrophes.[2]

But here’s the problem: contributing to this general civilizational progress that benefits humanity also substantially benefits AI researchers and their work.

My intuitive reaction here (and that of most, I assume) is maybe something like “yeah ok but surely this doesn’t balance out the benefits. We can’t tell the overwhelming majority of humans that we’re gonna slow down science, economic growth and improving their lives with these (and those of their descendants) until AI is safe just because these would also benefit a tiny minority that is making AI less safe”.

However, there has to be some threshold of harm (from AI development) beyond which we would think slowing down technological progress generally (and not only AI progress) would be worth it.

So what makes us believe that we’re not beyond this threshold?

^

For example, on his 80,000 hours podcast appearance, Zvi Mowshowitz claims that it is "the most destructive job per unit of effort that you could possibly have". See also the recent growth of the Pause AI movement.

^

For recent research and opinions that go in that direction, see Clancy 2023; Clancy and Rodriguez 2024.
How LLMs Learn: What We Know, What We Don't (Yet) Know, and What Comes Next
by Jonasb — 2024-07-09T09:58:20.502Z
Link: https://www.denominations.io/how-llms-learn/

Humans are amazing.

And–let's be honest–pretty weird.

I mean, why are so many of us all hyped up about Large Language Models (LLMs)? How did we collectively decide this kind of automated decision-making is "the next big thing"? It's not like a talking thesaurus can change the world, right?*

The thing most people seem to miss is that LLMs don't understand humans.

They can generate high-quality content, true, and some of them are already in the top 95th percentile when it comes to processing text, video, medical data etc. But they have no idea what a human "is".

Don't get me wrong, I think LLMs are an amazing technology–I've been working with language models since 2017–but I am also quite sceptical about the world-changing potential these models have.

So I thought it would be good to do a deep dive into how LLMs learn.

Let's dive right in.

Part one: Training Large Language Models

To start, like any other machine learning model, LLMs learn from examples.

These examples are selected by humans based on their ability to teach the model something about the task or tasks that need to be automated.

For example, if a machine learning researcher is training a model that needs to generate text, he or she will feed the model text examples.

Researchers have worked on different combinations of inputs and outputs based on the success of early LLMs. As a result we now have models that

... can generate images from text. They are shown examples of text as input, and examples of images as output.
... can generate translations. They are shown examples of text in one language as an input, and a (human-) translated version of that same text as output.
... can decipher proteins. They are shown images of protein structures as input, and mapped-out components of these structures as output.

You get the picture.

The sum total of the examples shown to a model is called its "training data".

People working on a model will tell it what to learn by configuring the prediction error rates that need to be reduced (in jargon: the "loss function").

Let's try to illustrate with an example.

Say you have a bakery you go to every day, and because you are a regular customer you know on a regular day croissants will run out by around 09:30 AM.

Then your "training data" is your earlier visits to the bakery, and the prediction problem is whether or not there will be croissants by the time you arrive at the bakery.

Through ea...
WTF is with the Infancy Gospel of Thomas?!? A deep dive into satire, philosophy, and more
by kromem — 2024-07-09T09:29:44.482Z

A few weeks ago, you may have seen sensationalist headlines like "A New Discovery Could Offer Some Clues About Jesus’ Childhood", discussing a find of a 4th-5th century manuscript fragment of the Infancy Gospel of Thomas, a strange apocryphal text that we've already had copies of for years. The discovery told us a little more about the development of the text, but certainly didn't promise any new revelations or insights as these headlines claimed. But it might be worth taking the opportunity of this news cycle to take a closer look at the text, as I think it hides a few surprises that have escaped most analyses to date.

Introduction

The Infancy Gospel of Thomas was the first apocrypha I ever read.

I went to a private school that was religiously affiliated, though much more concerned with getting its students into an Ivy League than into Heaven. And so while we needed to study religion in the course of our education, it wasn't done as a typical Sunday school indoctrination, but as classes under teachers with either an MA or PhD in Biblical Studies that focused on things like textual criticism, source analysis, sociology, etc.

These were a lot of fun, even for (or maybe especially for) a budding rationalist who was born and raised into an Agnostic household. For example, the course where we read the Infancy Gospel of Thomas - assigned to showcase the weirdest of the weird from the halls of heretical texts - was a semester focused on cults with half the class dedicated to analysis of the first few centuries of Christianity through the lens of a budding cult.

The field of Biblical Studies has very little in the way of testable predictions and empirical evidence for most of what's discussed. Insights and perspectives largely come down to rational arguments given available information. On top of this, there's a massive survivorship bias regarding both the texts and attitudes regarding them, compounded by the anchoring biases typical of academia and rampant in lieu of testable assertions.

In short, it's a ton of fun to take some time and look at, particularly if you don't have a significant inherent bias regarding the subject matter. And something you'll quickly find is that good analyses frequently come down to understanding everything you can about the context of what you are analyzing.

The context for The Infancy Gospel of Thomas

Most scholarship regarding Infancy Gospe...
Book Review: Safe Enough? A History of Nuclear Power and Accident Risk
by ErickBall — 2024-07-09T01:12:28.730Z

Epistemic status: This book covers a lot of topics related to nuclear energy; I have experience with some of them but not all, and some are inherently murky. I will try to make it clear what's fact and what's my opinion.

“If you’re not [pursuing safety] cost-effectively, you’re killing people.”

—David Okrent, former member of the Advisory Committee on Reactor Safeguards

1. Enlightenment Faith

Safe Enough? is the heartbreaking story of an industry that tried to move fast and break things, and in the process ended up completely broken. But it is also a cautiously hopeful tale of repeated attempts to resuscitate nuclear power through the arduous, and often controversial, process of Probabilistic Risk Assessment (PRA). While it doesn’t fully explain what went wrong with nuclear power, or whether it can be fixed, this book fills in some of the gaps, and avoids the ideological blinders that often define the popular debate.

Author Tom Wellock is the official historian of the U.S. Nuclear Regulatory Commission. His exhaustively researched history (the citations and bibliography run to over a hundred pages) sometimes feels like no more than a compendium of quotes from primary sources. But, if you read between the lines, it's a book with a message: The history of nuclear safety and its regulation is riddled with mistakes, but it has gradually improved, largely through better quantification of risk.

Safe Enough? shows how early nuclear regulators, and the industry itself, misunderstood the technology and its risks. The resulting screwups exacerbated public mistrust and political opposition, leading to excessive and chaotic regulation. Gradual improvements to PRA, it argues, have slowly made nuclear plants both safer and more efficient to operate.

This argument plays directly to my biases: PRA is sometimes called “the rationalist approach to regulatory development” [

ref

]. Still, I’m not sure how much I believe it. Can regulation really be both nuclear power’s albatross and its savior? Or is attributing so much power to a bunch of shaky numbers a case of high modernist hubris, a misplaced faith in logic and legibility? Improvements in efficient reactor operation over time followed accumulated experience and maturing technology, not just regulatory changes. And though the NRC’s oversight of existing reactors now leans heavily on risk information, licensing of ne...
Me, Myself, and AI: the Situational Awareness Dataset (SAD) for LLMs
by L Rudolf L — 2024-07-08T22:24:38.441Z

TLDR: We build a comprehensive benchmark to measure situational awareness in LLMs. It consists of 16 tasks, which we group into 7 categories and 3 aspects of situational awareness (self-knowledge, situational inferences, and taking actions).

We test 19 LLMs and find that all perform above chance, including the pretrained GPT-4-base (which was not subject to RLHF finetuning). However, the benchmark is still far from saturated, with the top-scoring model (Claude-3.5-Sonnet) scoring 54%, compared to a random chance of 27.4% and an estimated upper baseline of 90.7%.

This post has excerpts from our paper, as well as some results on new models that are not in the paper.

Links: Twitter thread, Website (latest results + code), Paper

The structure of our benchmark. We define situational awareness and break it down into three aspects.  We test these aspects across 7 categories of task.  Note: Some questions have been slightly simplified for illustration

Abstract

AI assistants such as ChatGPT are trained to respond to users by saying, “I am a large language model”. This raises questions. Do such models know that they are LLMs and reliably act on this knowledge? Are they aware of their current circumstances, such as being deployed to the public? We refer to a model's knowledge of itself and its circumstances as situational awareness. To quantify situational awareness in LLMs, we introduce a range of behavioral tests, based on question answering and instruction following. These tests form the Situational Awareness Dataset (SAD), a benchmark comprising 7 task categories and over 13,000 questions. The benchmark tests numerous abilities, including the capacity of LLMs to (i) recognize their own generated text, (ii) predict their own behavior, (iii) determine whether a prompt is from internal evaluation or real-world deployment, and (iv) follow instructions that depend on self-knowledge.

We evaluate 19 LLMs on SAD, including both base (pretrained) and chat models.
While all models perform better than chance, even the highest-scoring model (Claude 3 Opus) is far from a human baseline on certain tasks. We also observe that performance on SAD is only partially predicted by metrics of general knowledge (e.g. MMLU).
Chat models, which are finetuned to serve as AI assistants, outperform their corresponding base models on SAD but not on general knowledge tasks.

The purpose ...
Robin Hanson & Liron Shapira Debate AI X-Risk
by Liron — 2024-07-08T21:45:40.609Z
Link: https://www.youtube.com/watch?v=dTQb6N3_zu8

Robin and I just had an interesting 2-hour AI doom debate. We picked up where the Hanson-Yudkowsky Foom Debate left off in 2008, revisiting key arguments in the light of recent AI advances.

My position is similar to Eliezer's: P(doom) on the order of 50%.

Robin's position remains shockingly different: P(doom) < 1%.

I think we managed to illuminate some of our cruxes of disagreement, though by no means all. Let us know your thoughts and feedback!

Where To Watch/Listen/Read

For most casually-interested readers, I recommend consuming the debate via my debate highlights and analysis post, which has a clean transcript and its own video podcast to go with it.

Other links

YouTube video
Podcast audio
Raw Transcript

Topics

AI timelines
The "outside view" of economic growth trends
Future economic doubling times
The role of culture in human intelligence
Lessons from human evolution and brain size
Intelligence increase gradient near human level
Bostrom's Vulnerable World hypothesis
The optimization-power view
Feasibility of AI alignment
Will AI be "above the law" relative to humans

About Doom Debates

My podcast, Doom Debates, hosts high-quality debates between people who don't see eye-to-eye on the urgent issue of AI extinction risk.

All kinds of guests are welcome, from luminaries to curious randos. If you're interested to be part of an episode, DM me here or contact me via Twitter or email.

If you're interested in the content, please subscribe and share it to help grow its reach.
"The Singularity Is Nearer" by Ray Kurzweil - Review
by Lavender — 2024-07-08T21:32:27.307Z

Over the past week I read Ray Kurzweil's new book "The Singularity Is Nearer". This is my review.

First off, if you've read The Singularity Is Near, which was published 19 years ago in 2005, you should be aware that the sequel book is a lot less technical. The Singularity Is Near contained a couple dozen graphs explaining the exponential improvement in computers in computer engineering terminology. This was good at getting the point across but made some of the details of the book difficult to understand if you didn't know this terminology well. The Singularity Is Nearer is much lighter on this, which is probably because less of that is needed nowadays to get the point across. The book is more accessible to non-computer nerds. Which is likely due to the fact that nowadays it's easier to convince people who are not computer nerds that the singularity is near.

I think the most important prediction that Ray Kurzweil made in The Singularity Is Nearer is that we will have Artificial General Intelligence by 2029, meaning that by 2029 we will have machines capable of doing any cognitive task that an intelligent human can do. Ray explains that he first made this prediction in 1999, and everyone thought it was crazy at the time, but he has kept consistent with this prediction, and in recent years experts have started to agree with him, with the consensus on Metacalculus having reached that AGI will happen by 2026 in May of 2022, making Ray Kurzweil now in the conservative camp.

Ray goes onto argue that AI will be used to automate the production of food, housing, medicine, transportation, and clothing, making all of our necessities very cheap. And he argues that medicine will get qualitatively better, and that we will reach Longevity Escape Velocity by the 2030s (with the more diligent / wealthy / and well-connected among us reaching it by 2030 on the dot). He also argues that the technology needed to automate a lot of work is already here and that we're already in the cultural latency phase of waiting for technology that already exists to replace workers, that there will be an increasing shortage of jobs in the near future, and that Universal Basic Income is not just a good idea but is inevitable and that all developed countries will adopt it by the early 2030s and all developing countries will have it by the late 2030s. He argues fairly convincingly that UBI is mo...
Sample Prevalence vs Global Prevalence
by jefftk — 2024-07-08T21:00:03.809Z

Cross-posted from my NAO
Notebook. Thanks to Evan Fields and Mike McLaren for editorial
feedback on this post.

In Detecting
Genetically Engineered Viruses With Metagenomic Sequencing we have:

our best guess is that if this system were deployed at the scale of
approximately $1.5M/y it could detect something genetically engineered
that shed like SARS-CoV-2 before 0.2% of people in the monitored
sewersheds had been infected.

I want to focus on the last bit: "in the monitored sewersheds". The idea
is, if a system like this is tracking wastewater from New York City, its
ability to raise an alert for a new pandemic will depend on how far along
that pandemic is in that particular city. This is closely related to
another question: what fraction of the global population would have to be
infected before it could raise an alert?

There are two main considerations pushing in opposite directions, both based
on the observation that the pandemic will be farther along in some places
than others:

With so many places in the world where a pandemic might start, the
chance that it starts in NYC is quite low. To take the example of
COVID-19, when the first handful of people were sick they were all in
one city in China. Initially, prevalence in monitored sewersheds in
other parts of the world will be zero, while global prevalence will
be greater than zero. This effect should diminish as the pandemic
progresses, but at least in the <1% cumulative incidence
situations I'm most interested in it should remain a significant
factor. This pushes prevalence in your sample population to lag
prevalence in the global population.

NYC is a highly connected city: lots of people travel between there
and other parts of the world. Since pandemics spread as people move
around, places with many long-distance travelers will generally be
infected before places with few. While if you were monitoring an
isolated sewershed you'd expect this factor to cause an additional
lag in your sample prevalence, if you specifically choose places like
NYC we expect instead the high connectivity to redu

...
Advice to junior AI governance researchers
by Orpheus16 — 2024-07-08T19:19:07.316Z

This summer, I’m supervising some research fellows through

Cambridge’s ERA AI Fellowship

. The program started last week, and I’ve had conversations with about 6 fellows about their research projects & summer goals.

In this post, I’ll highlight a few pieces of advice I’ve found myself regularly giving to research fellows. This post reflects my own opinions and does not necessarily reflect the views of others at ERA.

Prioritize projects that have a clear target audience

Problem: One of the most common reasons why research products fail to add value is that they do not have a target audience. I think it can be easy to find a topic that is interesting/important, spend several months working on it, produce a 20-50 page paper, and then realize that you have no particular stakeholder(s) who find the work action-relevant.

Advice: Try to brainstorm what specific individuals you would want to have affected by your piece. This might be some folks in the AI safety community. This might be government officials at a relevant agency in the US or the UK. Prioritize projects that have a clear target audience and prioritize projects in which you have a way of actually getting your paper/product to that target audience. Ideally, see if you can talk to representative members of your target audience in advance to see if you have a good understanding of what they might find useful.

Caveat #1: Gaining expertise can be a valid reason to do research. Sometimes, the most important target audience is yourself. It may be worthwhile to take on a research project because you want to develop your expertise in a certain area. Even if the end product is not action-relevant for anyone, you might have reason to believe that your expertise will be valuable in the present or future.

Caveat #2: Consider target audiences in the future. Some pieces do not have a target audience in the present, but they could be important in the future. This is particularly relevant when considering Overton Window shifts. It’s quite plausible to me that we get at least one more major Overton Window shift in which governments become much more concerned about AI risks. There may even be critical periods (lasting only a few weeks or a few months) in which policymakers are trying to understand what to do. You probably won’t have time to come up wit...