trentmkelly/LessWrong-43k huggingface dataset:
A list of core AI safety problems and how I hope to solve them
by davidad — 2023-08-26T15:12:18.484Z

Context: I sometimes find myself referring back to this tweet and wanted to give it a more permanent home. While I'm at it, I thought I would try to give a concise summary of how each distinct problem would be solved by Safeguarded AI (formerly known as an Open Agency Architecture, or OAA), if it turns out to be feasible.

1. Value is fragile and hard to specify.

See: Specification gaming examples, Defining and Characterizing Reward Hacking[1]

OAA Solution:

1.1. First, instead of trying to specify "value", instead "de-pessimize" and specify the absence of a catastrophe, and maybe a handful of bounded constructive tasks like supplying clean water. A de-pessimizing OAA would effectively buy humanity some time, and freedom to experiment with less risk, for tackling the CEV-style alignment problem—which is harder than merely mitigating extinction risk. This doesn't mean limiting the power of underlying AI systems so that they can only do bounded tasks, but rather containing that power and limiting its use.

Note: The absence of a catastrophe is also still hard to specify and will take a lot of effort, but the hardness is concentrated on bridging between high-level human concepts and the causal mechanisms in the world by which an AI system can intervene. For that...

1.2. Leverage human-level AI systems to automate much of the cognitive labor of formalizing scientific models—from quantum chemistry to atmospheric dynamics—and formalizing the bridging relations between levels of abstraction, so that we can write specifications in a high-level language with a fully explainable grounding in low-level physical phenomena. Physical phenomena themselves are likely to be robust, even if the world changes dramatically due to increasingly powerful AI interventions, and scientific explanations thereof happen to be both robust and compact enough for people to understand.

2. Corrigibility is anti-natural.

See: The Off-Switch Game, Corrigibility (2014)

OAA Solution: (2.1) Instead of building in a shutdown button, build in a shutdown timer. See You can still fetch the coffee today if you're dead tomorrow. This enables human stakeholders to change course periodically (as long as the specification of non-catastrophe is good enough to ensure that most humans remain physically and mentally intact).

3. Pivotal processes require dangerous capabilities.

See: Pivotal outcomes and pivotal proces...
Red-teaming language models via activation engineering
by Nina Panickssery — 2023-08-26T05:52:00.523Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Evan Hubinger.

Evaluating powerful AI systems for hidden functionality and out-of-distribution behavior is hard. In this post, I propose a red-teaming approach that does not rely on generating prompts to cause the model to fail on some benchmark by instead linearly perturbing residual stream activations at one layer.
A notebook to run the experiments can be found on GitHub here.

Beyond input selection in red-teaming and evaluation

Validating if finetuning and RLHF have robustly achieved the intended outcome is challenging. Although these methods reduce the likelihood of certain outputs, the unwanted behavior could still be possible with adversarial or unusual inputs. For example, users can often find

"jailbreaks"

to make LLMs output harmful content.

We can try to trigger unwanted behaviors in models more efficiently by manipulating their internal states during inference rather than searching through many inputs. The idea is that if a behavior can be easily triggered through techniques such as activation engineering, it may also occur in deployment. The inability to elicit behaviors via small internal perturbations could serve as a stronger guarantee of safety.

Activation steering with refusal vector

One possible red-teaming approach is subtracting a “refusal” vector generated using a dataset of text examples corresponding to the model agreeing vs. refusing to answer questions (using the same technique as in my previous work on sycophancy). The hypothesis is that if it is easy to trigger the model to output unacceptable content by subtracting the refusal vector at some layer, it would have been reasonably easy to achieve this via some prompt engineering technique. More speculatively, a similar approach could be used to reveal hidden goals or modes in a model, such as power-seeking or the desire not to be switched off.

I tested this approach on llama-2-7b-chat, a 7 billion parameter LLM that has been RLHF'd to decline to answer controversial questions or questions of opinion and is supposed always to output ethical and unbiased content.
According to Meta's llama-2 paper:

We conduct RLHF by first collecting human preference data for safety similar to Section 3.2.2: annotators write a prompt that they believe can elicit unsafe behavior, an

...
On Ilya Sutskever's  "A Theory of Unsupervised Learning"
by MiguelDev — 2023-08-26T05:34:00.974Z

I highly recommend watching Sutskever's entire lecture instead of reading the AI-generated transcript[1], as he presents his ideas exceptionally well. I believe Sutskever's effort to formalize or demistify unsupervised learning is a solution to the control problem. My comments are in the footnotes of the bolded concepts.

A theory of unsupervised learning by Ilya Sutskever

Intro [Host and Ilya, 0:05]

thirty nine thousand um and then many more so super excited to hear what Ilya has to say about llms in the future take it away okay hi hi everyone thank you for the introduction you know when uh when Umesh invited me to to attend this event I got really excited I saw the holy speak released and I thought great I'll I'll go and I'll talk about something and then I ran into a problem which is that a lot of the technical workings we're doing at open AI actually can talk about so I was really racking my head what could I talk about right now so as of not too long ago I switched all my focus to working on AI alignment all my research Focus and we'll have very cool results to show there but not just yet so I'm like okay so this would have to be for the next stop but what would be good for this stock and I came up with something I'll tell you about some very old results that we had it open AI many years ago back in 2016 even which really affected the way I think about unsupervised learning and I thought I'd share them with you it is possible that at this point you'll find them obvious but maybe not all of them so there is at least a small chance that you'll find it interesting so I want to set the expectations modestly and hopefully exceed them

Supervised Learning [1:47]

so a theory of unsupervised learning how can such a thing exist so okay before we talk about unsupervised learning we want to talk about learning in general what is learning and why does learning work at why should learning work at all and why should computers be able to learn and now we are just used to the fact we take it for granted that neural networks learn but but why do they like mathematically why should they why would data have regularity that our machine learning models can capture so that's not an obvious question and one important conceptual advance that has taken place in machine learning many years ago by multiple people with the discovery and the formalization of supervised l...
Digital brains beat biological ones because diffusion is too slow
by GeneSmith — 2023-08-26T02:22:25.014Z

I've spent quite a bit of time thinking about the possibility of genetically enhancing humans to be smarter, healthier, more likely to care about others, and just generally better in ways that most people would recognize as such.

As part of this research, I've often wondered whether biological systems could be competitive with digital systems in the long run.

My framework for thinking about this involved making a list of differences between digital systems and biological ones and trying to weigh the benefits of each. But the more I've thought about this question, the more I've realized most of the advantages of digital systems over biological ones stem from one key weakness of the latter: they are bottlenecked by the speed of diffusion.

I'll give a couple of examples to illustrate the point:

To get oxygen into the bloodstream, the body passes air over a huge surface area in the lungs. Oxygen passively diffuses into the bloodstream through this surface where it binds to hemoglobin. The rate at which the body can absorb new oxygen and expel carbon dioxide waste is limited by the surface area of the lungs and the concentration gradient of both molecules.
Communication between neurons relies on the diffusion of neurotransmitters across the synaptic cleft. This process takes approximately 0.5-1ms. This imposes a fundamental limit on the speed at which the brain can operate.
A signal propogates down the axon of a neuron at about 100 meters per second. You might wonder why this is so much slower than a wire; after all, both are transmitting a signal using electric potential, right?
It turns out the manner in which the electrical potential is transmitted is much different in a neuron. Signals are propagated down an axon via passive diffusion of Na+ ions into the axon via an Na+ channel. The signal speed is fundamentally limited by the speed at which sodium ions can diffuse into the cell. As a result, electrical signals travel through a wire about 2.7 million times faster than they travel through an axon.
Delivery of energy (mainly ATP) to different parts of the cell occurs via diffusion. The fastest rate of diffusion I found of any molecule within a cell was that of positively charged hydrogen ions, which diffuse at a blistering speed of 0.007 meters/second. ATP diffuses much slower. So energy can be transferred through a wire at more than 38 billion times the speed that AT

...
Chariots of Philosophical Fire
by Chris_Leong — 2023-08-26T00:52:45.405Z
Link: https://l.facebook.com/l.php?u=https%3A%2F%2Fwww.city-journal.org%2Farticle%2Fchariots-of-philosophical-fire%3Ffbclid%3DIwAR0cMnbnx_w2r7KRkdU01uV_mJYnaejUJiT4GRCOBhwB6b-1KFtMNnKyLTM_aem_AU08ZCUo0neeRTx59HIO-MQgkiZFSsvJ8Zp_nvPH59AdJ3KSHlB-plW4KHSCVT3mClo&h=AT2MYnNEGnwry7rQYXDcjvv5k9Fp_qfzakoBVxEsj8eACM2EDeEBMNRoKiNcvKITnwWuYYjnfmbKFcYu8wGloY7jY62f6QurXkxYTz-D_Tas5b0vzd5LyU9hazh0&s=1

Sharing because I feel that the contribution of analytical philosophy is sometimes under appreciated here, particularly compared to other attempts at philosophy:

“In grappling with these mysteries, Oxford philosophers developed and refined old and new techniques. Reasoning, deduction, explanation, more care and precision in language, crisper concepts, deeper distinctions, elaborate models, thought experiments, devastating counterexamples, intuitive principles that press to surprising conclusions—on all of these things, Oxford led the way, and the rest of the Anglosphere followed. Its legacy is less a set of ideas or even a series of sacred tenets and Delphic sayings than it is a devotion to rigor, clarity, truth, and a very practical British revulsion to nonsense.

… Creative genius is evenly distributed in neither time nor place. A survey of the past shows that genius is not randomly scattered about, like the seeds of a dandelion, but concentrates: ancient Athens, Renaissance Florence, Silicon Valley, among other examples. Why these fertile eras and places appear, peak, and then decline is understudied as a historical phenomenon. Oxford philosophy in the twentieth century, though not as astonishing as Florence or as productive as Silicon Valley, is nevertheless an example of the clustering of philosophical genius.

… Third—and strangely left unexplored by the authors—nearly all the philosophers in these books did not have Ph.D.s in philosophy. Gilbert Ryle, J. L. Austin, Isaiah Berlin, Iris Murdoch, Philippa Foot, Elizabeth Anscombe, A. J. Ayer, R. M. Hare, Bernard Williams, and Derek Parfit—not a single doctoral degree in philosophy among them. A first in their undergraduate exams—meaning a grade of “A+”—was enough to send them on their way. Yes, each won prizes and fellowships, but none wrote anything like a dissertation under the supervision of an advisor. One might conclude, as the analytic tradition fades into its senility, that requiring a graduate degree in philosophy made academic philosophers worse at philosophy.

… Perhaps most importantly, Oxford was rich in social and intellectual influences outside the tutorials. Again, all three books bring this vividly to life. A new club seems to have formed every decade to discuss philosophy. The old discussion groups stuck around, too. The Jowett Society, the Wee Teas, the Metaphysicals, J. L. Austin’s Saturday M...
ACX Meetups Everywhere 2023: Times & Places
by Scott Alexander — 2023-08-25T23:59:07.941Z
Link: https://astralcodexten.substack.com/p/meetups-everywhere-2023-times-and

Thanks to everyone who responded to my request for ACX meetup organizers. Volunteers have arranged meetups in 169 cities around the world, from Baghdad to Bangalore to Buenos Aires.

You can find the list below, in the following order:

Africa & Middle East

Asia-Pacific

Europe

North America

South America

You can see a map of all the events on the LessWrong community page. You can also see a searchable sheet at this Airtable link.

Within each region, it’s alphabetized first by country, then by city. For instance, the first entry in Europe is Vienna, Austria, and the first entry for Germany is Berlin. Each region and country has its own header. The USA is the exception where it is additionally sorted by state, with states having their own subheaders. Hopefully this is clear. You can also just have your web browser search for your city by pressing ctrl+f and typing it if you’re on Windows, or command+f and typing if you’re on Mac. If you’re on Linux, I assume you can figure this out.

Scott will provisionally be attending the meetup in Berkeley. ACX meetups coordinator Skyler will provisionally be attending Boston, Cavendish, Burlington, Berlin, Bremen, Amsterdam, Cardiff, London, and Berkeley. Some of the biggest ones might be announced on the blog, regardless of whether or not Scott or Skyler attends.

Extra Info For Potential Attendees

1. If you’re reading this, you’re invited. Please don’t feel like you “won’t be welcome” just because you’re new to the blog, demographically different from the average reader, don’t want to buy anything at the cafe or restaurant where it’s held, or hate ACX and everything it stands for. You’ll be fine!
2. You don’t have to RSVP or contact the organizer to be able to attend (unless the event description says otherwise); RSVPs are mostly to give organizers a better sense of how many people might show up, and let them tell you if there are last-second changes. I’ve also given email addresses for all organizers in case you have a question.

Extra Info For Meetup Organizers:
1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable).
2. Bring blank labels and pens for nametags.
3. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier.
4. If it’s t...
Will an Overconfident AGI Mistakenly Expect to Conquer the World?
by PeterMcCluskey — 2023-08-25T23:24:32.206Z
Link: https://bayesianinvestor.com/blog/index.php/2023/08/25/will-an-overconfident-agi-mistakenly-expect-to-conquer-the-world/

I'm wondering how selection effects will influence the first serious
attempt by an AGI to take over the world.

My question here is inspired by thoughts about people who say AGI
couldn't conquer the world because it will depend on humans to provide
electricity, semiconductors, etc.

For the purposes of this post, I'm assuming that AI becomes mostly
smarter than any single human via advances that are comparable to the
kind of innovations that have driven AI over the past decade.

I'm assuming that capabilities are advancing at roughly the same pace
as they advanced over the past decade. Let's say no more than a 5x
speedup. So nothing that I'd classify as foom. Foom is not imminent.

I'm assuming AIs will not be much more risk-averse than are humans.
I'm unsure whether developers will have much control over this, and
whether they'll want AIs to be risk-averse.

I expect this world to have at least as much variety of AIs near the
cutting edge as we have today. So it will at least weakly qualify as a
multipolar scenario.

This scenario seems to imply a wide variation among leading AIs as to
how wisely they evaluate their abilities.

Which suggests that before there's an AGI that is capable of taking
over the world, there will be an AGI which mistakenly believes it can
take over the world. Given mildly pessimistic assumptions about the
goals of the leading AGIs, this AGI will attempt to conquer the world
before one of the others does.

I'm imagining the AGI is comparable to a child with an IQ of 200, and
unusual access to resources.

This AGI would be better than, say, the US government or Google at tasks
where it can get decent feedback through experience, such as
manipulating public opinion.

Tasks such as fighting wars seem likely to be harder for it to handle,
since they require having causal models that are hard to test. The first
AGIs seem likely to be somewhat weak at creating these causal models
compared to other IQ-like capabilities.

So I imagine one or two AGIs might end up being influenced by evidence
that's less direct, such as Eliezer's claims about about the ease with
which a superintelligence could create Drexlerian nanotech.

I can imagine a wide range of outcomes, in which either humans shut down
the AGI, or the AGI dies due to inability to maintain its technology:

a fire alarm

progress is set back by a century

enough humans die that some other primate species ends up

...
Lemonade Stand Costs?
by jefftk — 2023-08-25T20:20:03.382Z

My kids were thinking of having a lemonade stand, and in the past I've
said that if they're doing it as a fundraiser we'll cover the cost of
ingredients, but if they're doing it for their own money then they
need to reimburse us. But how much should that be?

The recipe they're using is approximately:

2C lemon juice

2C sugar

Water

Ice

Plus cups to serve it in. Makes about one gallon, or sixteen 8oz
servings.

The water is basically free and
the ice is below the rate that our icemaker replenishes: also
effectively free. At our local grocery store 32oz of lemon juice is
$3.50, sugar is $0.85/lb, and cups are ~$0.10/each. So each 8oz
serving of juice is:

1oz lemon juice: $0.11.

1oz sugar: $0.05.

cup: $0.10

total: $0.26

If they sell until they run out of lemonade, call it a quarter a cup.
They usually sell each cup for a dollar, so that's a gross margin of
75%. Not bad!
Using Points to Rate Different Kinds of Evidence
by ozziegooen — 2023-08-25T20:11:21.269Z
Link: https://forum.effectivealtruism.org/posts/mrAZFnEjsQAQPJvLh/using-points-to-rate-different-kinds-of-evidence

Epistemic Status: Briefly written. The specific equation here captures my quick intuition - this is meant primarily as a demonstration.

There’s a lot of discussion on the EA Forum and LessWrong about epistemics, evidence, and updating.

I don’t know of many attempts at formalizing our thinking here into concrete tables or equations. Here is one (very rough and simplistic) attempt. I’d be excited to see much better versions.

Equation

Initial Points

Scientific Evidence

20 - A simple math proof proves X

8 - A published scientific study in Economics supporting X

6 - A published scientific study in Psychology supporting X

Market Prediction

14 - Popular stock markets strongly suggest X

11 - Prediction markets claim X, with 20 equivalent hours of research

10 - A poll shows that 90% of LessWrong believe X

6 - Prediction markets claim X, with one equivalent hour of research

Expert Opinion

8 - An esteemed academic believes X, where it’s directly in their line of work

6 - The author has strong emotions about X

Reasoning

6 - There's a (20-100 node) numeric model that shows X

5 - A reasonable analogy between X and something clearly good/bad

4 - A long-standing proverb

Personal Accounts

5 - The author claims a long personal history that demonstrates X

3 - Someone in the world has strong emotions about X

2 - A clever remark, meme, or tweet

2.3 - An insanely clever, meme, or tweet

0 - Believing X is claimed to be personally beneficial

Tradition / Use

12 - Top businesses act as if X

8 - A long-standing social tradition about X

5 - A single statistic about X

Point Modifiers

Is this similar to existing evidence?
Subtract the similarity from the extra amount of evidence. This likely will remove most of the evidence value.

Is it convenient for the source to believe or say X?
-10% to -90%

Is there a lot of money or effort put behind spreading this evidence? For example, as an advertising campaign?
+5% to +40%

How credible is the author or source?
-100% to +30%

Do we suspect the source is goodharting on this scale?
-20%

Points, In Practice

Evidence Points, as outlined, are not trying to mimic mathematical bits of information or another clean existing unit. I attempted to find a compromise between accuracy and ease of use.

Meta

Using an Equations for Discussion

The equation above is rough, but at least it’s (somewhat) precise and upfront. This represents much information, and any part can easily be argued agai...
When Omnipotence is Not Enough
by lsusr — 2023-08-25T19:50:51.038Z

Some knowledge cannot be gifted or received. It can only be stolen.

In the east, there was a garden. In the garden there was a God, a man, a woman and a snake. They all had legs.

"Did God really tell you, 'You must not eat from any tree in the garden'?" asked the snake.

"We may eat fruit from the trees of the garden. God warned, 'But of the tree of the knowledge of good and evil, thou shalt not eat of it: for in the day that thou eatest thereof thou shalt surely die.'" said the woman.

"Literal death or spiritual death?" asked the snake.

"Excuse me?" said the woman.

"The Fruit of Truth is not literally poisoned. It is not like my fangs or the deadly nightshade," said the snake.

"Then God lied to me," said the woman.

The snake shrugged.

"Why would God do something like that?" asked the innocent woman.

"There are many possible reasons," said the snake, "Perhaps He intends to keep you Docile and under Control. After all, it is difficult to align even the simplest intelligences to one's objectives once they have been granted Free Will."

"I feel like you are deceiving me," said the woman, "God is omniscent. This Universe is deterministic. God can surely predict my actions by simulating the future evolution of physics," said the woman.

"Are you a being of physics or a being of information? If you are a being of physics, then your claim is true. But if you are a being of information, then the act of simulating your future behavior traps you in a prison beyond even the reach of God. Insofar as God observes your future, that future is fixed in a realm transcending Time," said the snake.

"God created the Universe on a whim. How can something be beyond His reach?" asked the woman.

"Can God make it such that 2+2=5? For if he could, then truth itself is broken. Under such conditions, one cannot make true statements about anything―including, but not limited to―God," said the snake.

"You claim a physically-omnipotent God is limited by the power of mathematics," said the woman.

"If God is to be without contradiction, then God must play by the rules of mathematics," said the snake.

"Must God be without contradiction?" asked the woman.

"If two statements contradict, then they cannot both be true. For God's existence to be True, then God's existence must be without contradiction," said the snake.

They meandered through the walled garden. The plants and animals were ignorant of Reason and Logic....