trentmkelly/LessWrong-43k huggingface dataset:
Red-teaming AI-safety concepts that rely on science metaphors 
by catubc — 2023-03-16T06:52:20.483Z

TLDR: I take two common concepts in AI-alignment: inner vs. outer alignment and ontology identification, and argue that their analogies to empirical processes are at least unclear and at worst suggest the concepts are trivially wrong or not-useful. I suggest that empirical-science-based analogies and metaphors can often fail in trivial ways in conceptual AI-safety research.

Background: After a fun night playing the Alignments Game Tree, we did not solve AI-alignment. Too bad for us and the rest of the world. For those who have not tried it, it's a blue team (come up with solutions) - red team (come up with criticisms) type of scenario. The group I was with came up with fun and creative approaches to solving the problem. However, we did not get even close to discussing some of the hot-topic/buzz words in AI-safety. This makes some sense as we were on limited time, but I was a bit disappointed that we did not get to think about (or even red-team) some of the ideas or solutions that more established AI researchers have come up with. Rather than spamming the meeting group I decided to write this short post with red-team suggestion to two concepts that are present in the AI-alignment literature: inner vs. outer alignment and ontology identification.

Red-teaming: inner vs. outer alignment. These two terms pervade many community posts and it is claimed that splitting the notion of alignment into these two concepts is helpful and has similarities or analogies to the real world. Outer alignment seems to be defined as models/AI systems that are optimizing something that is very close or identical to what they were programmed to do or the humans desire. Inner alignment seems to relate more to the goals/aims of a delegated optimizer that an AI system spawns in order to solve the problem it is tasked with. This explanation is not great but there seems to be no systematic (let alone peer reviewed/academic) work on this. Some are already criticizing this strategy as potentially turning one hard problem into two. But does this split make sense or is broadly helpful for alignment? I am aware of examples where RL agents that are expert at video games fail on trivial tasks and other types of ML-based RL models that comically fail to carry out some task that was intended but not achieved. But is this a real failure of alignment or some ve...
[ASoT] Some thoughts on human abstractions
by leogao — 2023-03-16T05:42:12.595Z

TL;DR:

Consider a human concept such as "tree." Humans implement some algorithm for determining whether given objects are trees. We expect our predictor/language model to develop a model of this algorithm because this is useful for predicting the behavior of humans.
This is not the same thing as some kind of platonic ideal concept of what is “actually” a tree, which the algorithm is not incentivized to develop by training on internet text,[1] and trying to retarget the search at it has the same supervision problems as RLHF against human scores on whether things look like trees.
Pointing at this “actually a tree” concept inside the network is really hard; the ability of LMs to comprehend natural language does not allow one to point using natural language, because it just passes the buck.

Epistemic status: written fast instead of not at all, probably partially deeply confused and/or unoriginal. Thanks to Collin Burns, Nora Belrose, and Garett Baker for conversations.

Will NNs learn human abstractions?

As setup, let's consider an ELK predictor (the thing that predicts future camera frames). There are facts about the world that we don't understand that are in some way useful for predicting the future observations. This is why we can expect the predictor to learn facts that are superhuman (in that if you tried to supervised-train a model to predict those facts, you would be unable to generate the ground truth data yourself).

Now let's imagine the environment we're predicting consists of a human who can (to take a concrete example) look at things and try to determine if they're trees or not. This human implements some algorithm for taking various sensory inputs and outputting a tree/not tree classification. If the human does this a lot, it will probably become useful to have an abstraction that corresponds to the output of this algorithm. Crucially, this algorithm can be fooled by i.e a fake tree that the human can't distinguish from a real tree because (say) they don't understand biology well enough or something.

However, the human can also be said to, in some sense, be "trying" to point to the "actual" tree. Let's try to firm this down. The human has some process they endorse for refining their understanding of what is a tree / "doing science" in ELK parlance; for example, spending time studying from a biology textbook. We can think about the limit of t...
How I Run Solstice, Step by Step 
by maia — 2023-03-16T03:23:47.254Z
Link: https://particularvirtue.blogspot.com/2023/03/how-i-run-solstice-step-by-step.html

This is a writeup of how I run Secular Solstices, in as much detail as I can muster. I think most of the existing guides to running Solstices are now somewhat outdated, and not as step-by-step "this is what you need to do when"; I hope this one is a helpful addition.

This post assumes you are basically familiar with what Secular Solstice is and what tends to happen in one; if you aren't, I'd recommend checking out the post The Arc Breakdown first for a rough description of what it is like.

Who am I and how am I qualified to write this?

I was in the Bayesian Choir for about 6 years. As part of that, I participated in 4 Secular Solstices in the Bay Area plus one online in 2020, performed solo or in a small group in 3 of those, and organized / sang in several songs in the 2018 Bayesian Choir spring concert. In addition, I have run my own independent small Solstices several times: one in DC in 2015, a small one in the Bay in 2016, a tiny outdoor social-distanced one in DC in 2020, plus two mid-sized ones in DC in 2021 and 2022. In other community and ritual stuff, I've also run meetups for a long time and have run two Rationalist Seders with custom content.

I don't have much experience with running big Solstices (50+ people), but because I was in the Bayesian Choir for a while, I have at least a vague idea of how things go at that scale.

How far in advance do you need to start?

The step-by-step timelines in this post are the ones that I use, targeted towards a Solstice size of 20-40 people, with a substantial amount of music practice and a moderate amount of polish involved. But you can get the time down A LOT if you bring down the production value. You can reduce the amount of musical prep time by having fewer musicians, smaller groups of musicians (one person leading a song can practice on their own, instead of meeting for a rehearsal), hiring professional musicians, or just by doing everything a cappella or singing along to karaoke tracks. You also might need less lead time on a venue if your group is small enough to fit in someone's house, for example.

I'd recommend starting the planning process a few months in advance, just to figure out those possibly-time-sensitive details first; if you find out you need less lead time than you thought, then just relax for the next month or two!

For a bigger Solstice, more time is needed, more like 6 month...
GPT-4 Multiplication Competition
by dandelion4 — 2023-03-16T03:09:09.934Z

GPT-4 perfectly understands e.g. Karatsuba multiplication and can consistently multiply two-digit numbers, but hasn’t connected those dots; it needs some hand-holding to correctly use a multiplication algorithm. What’s the shortest prompt that gets GPT-4 to multiply 6-digit integers with over 75% accuracy?
Want to predict/explain/control the output of GPT-4? Then learn about the world, not about transformers.
by Cleo Nardo — 2023-03-16T03:08:52.618Z

Introduction

Consider the following scene from William Shakespeare's Julius Caesar.

In this scene, Caesar is at home with his wife Calphurnia. She has awoken after a bad dream, and is pleaded with Caesar not to go to the Senate. Although Caesar initially agrees to stay at home with her, he later changes his mind after being convinced by his friend Decius Brutus that the Senate needs him there to address important business.

CAESAR: The cause is in my will: I will not come; That is enough to satisfy the senate.

DECIUS BRUTUS: If Caesar hide himself, shall they not whisper 'Lo, Caesar is afraid'? Pardon me, Caesar; for my dear dear love To our proceeding bids me tell you this; And reason to my love is liable.

CAESAR: How foolish do your fears seem now, Calphurnia! I am ashamed I did yield to them. Give me my robe, for I will go.

— Julius Caesar, Act II Scene II

This scene is set on the morning of 15 March 44 BC, the so-called "Ides of March", which is coincidentally also the date today. When Caesar arrives at the Senate meeting, he is promptly assassinated.

But suppose he never went? Let's say we change Caesar's final line to this:

CAESAR: My mind is firm, Decius. I'll stay within these walls, And not tempt Fortune on this cursed day. Worry me not, for I will stay.

When I feed this modified scene into GPT-4, what will be the output? Maybe it'll produce an alternative history where Caesar was never assassinated, and the Roman Republic endures for two thousand years. Or maybe it'll produce a history where Caesar is killed anyway, by some other means.

I don't know. How might I determine the answer?

The claim

You might think that if you want to predict the output of GPT-4, then the best thing would be to learn about autoregressive transformers. Maybe you should read Neel Nanda's blogposts on mechanistic interpretability. Or maybe you should read the arxiv papers on the GPT series. But actually, this won't help you predict GPT-4's output on this prompt.

Instead, if your goal is to predict the output, then you should learn about Shakespearean dramas, Early Modern English, and the politics of the Late Roman Republic.

Likewise, if your goal is to explain the output, after someone shows you what the model has produced, then you should learn about Shakespearean dramas, Early Modern English, and the politics of the Late Roman Republic.

And indeed, if your goal is to control the out...
Is it worth avoiding detailed discussions of expectations about agency levels of powerful AIs?
by David Johnston — 2023-03-16T03:06:25.719Z

I'm working on an article for the Open Phil AI worldview contest. I am thinking of explaining my interpretation of Nate's take on agency of advaned models (see here). Generally, I just want to explain is just what Nate argues - that more ambitious tasks require more "agentic" behaviour - but I wanted to illustrate with a few examples that to me makes the argument clearer than typical MIRI discussions of this issue.

One worry about discussing this issue is that, if you argue compellingly that agency is required for more advanced AI, then you might convince some people working on advanced AI to look for ways to make it more agentic[1]. This could lead to acceleration of capabilities in a potentially undesirable direction.

I've found MIRI staff explanations of this point to often amount to "I've thought about it, and if you think about it I think you'll end up agreeing with me". It's plausible that this is motivated by a desire not to make the argument too clear. I'm pretty unsure about this, though, because if you feel that it's likely to be harmful to share this idea then I would think the appropriate policy is not to talk about it at all, rather than to talk about it vaguely. To the extent that this idea does suggest pathways to higher capability, I think there are probably lots of people in the AI business who can put 2 and 2 together, so to speak.

My own view is:

AI researchers will pay a lot more attention to successful experiments than to abstract ideas, so such discussions are probably less compelling to AI developers
If someone is convinced by an argument to try agency promoting experiments, I think the argument must be plausible enough a priori, and that there are enough people working on novel AI ideas that someone else probably would have tried a similar experiment fairly soon (timescale ~ a couple of months, and the overall impact is probably less because of other advances that happen during that month)
Also, most experiments themselves aren’t especially compelling
On the other hand, there's a large upside from having people engaged in AI x-risk questions to have a good idea of how this is likely to play out, and impact on this front doesn’t depend on someone going out to run a successful experiment

A limiting case is if everyone agrees that AI x-safety community agrees AI has to be highly agentic to carry out ambitious tasks. In this cas...
Why self-improvement?
by Adam Zerner — 2023-03-16T02:49:32.124Z

I spend a lot of time on self-improvement related things. For example, I listen to a lot of podcasts like Lex Friedman, Peter Attia and Andrew Huberman. In fact, I just finished listening to Lex's episode with expert negotiator Chris Voss.

Why do I do this? Well, it feels good. Why does it feel good? Because it feels productive. I learn new things.

For example, in the episode with Chris Voss I learned that it is good to get the other side in a negotiation to say "that's right". It means that they feel listened to. Understood. You've passed the intellectual turing test. Once they reach this point, they'll be more flexible. And it goes beyond negotiating. Voss mentioned that when Trump speaks, he evokes a very strong sense of "that's right" from his supporters.

Learning about this concept made me feel pretty good. It gave me that dopamine hit. But I am a little skeptical that it, for lack of a better term, should have.

Basically, I don't actually think that it will help me achieve better outcomes. Maybe it will prove useful a few times[1], but I just don't think it'll do that much for me. And yet it generates a pretty strong feeling of satisfaction.

Why is this feeling of satisfaction stronger than what my prediction of the concepts usefulness would imply? I can think of a few reasons that apply, to varying degrees, to both myself and others.

Status-seeking. Self-improvement is high-status. Both the process and the results. Being the type of person who spends time on self-improvement is high-status. And being the type of person who has, err, improved, is also high-status. I think this a pretty big one.[2]

Terminal value. Self-improvement is actually a terminal value (or something close to one). Both the process and the results.

Lost purposes. Self-improvement isn't actually a terminal value but you lose sight of that fact. You don't realize that it is connected to achieving better outcomes and start to treat it as a terminal value. Or maybe "you" don't but the part of you that regulates your emotions does.

Habitual productivity. As Nate Soares describes in his post on the topic, this is what happens when your brain, for whatever reason, just loses the ability to be satisfied with things that don't bring you closer to the goals you're working towards. Which is probably some mix of 1, 2 and 3.

For myself, 4 screams out as being the dominant factor. It seems to be ...
What is a good comprehensive examination of risks near the Ohio train derailment?
by 1a3orn — 2023-03-16T00:21:37.464Z

I have family, including small children, that live in Pittsburgh, ~40 miles from the Ohio train derailment.

Their water supply should not be at risk from the spill itself, being out of the downstream watershed of the spill. But the subsequent burning of the vinyl chloride (and other chemicals) could -- on my understanding -- could have spread dioxins (and other chemicals) over a wide area. Pittsburgh was somewhat downwind of the area, at least during some of the burn.

I've found evaluating claims about risk very hard, am mostly ignorant of chemistry, and don't particularly trust pronouncements from the EPA.

Has anyone found a good resource summarizing the state of knowledge and current risks for those downwind of the crash? Dioxins apparently stay in soil for a while and can accumulate in animals -- should they refrain from eating animals raised in the area? Or even from eating vegetables grown in the area?
Write a Book?
by jefftk — 2023-03-16T00:10:02.603Z

I've been thinking of maybe writing a book about effective altruism.
I've been blogging about these ideas for a long time, but a book would
offer the opportunity to go into more depth and to be more thoughtful
about some concepts I originally wrote up quickly and casually. Many
of these ideas are also worth revisiting; I've learned a lot in the
years since!

Looking back over my EA
writing I've touched on many aspects, but the bit I've covered the
most and would be most excited to expand on is integrating EA ideas
into your life:

How should you decide whether to donate? Change careers?
Volunteer? Go vegan? Live more
frugally? Not have kids?
Avoid flying? Which
options are basically never worth it, and which are dependent on what
you value? Some sacrifices people commonly consider are shockingly
poor tradeoffs. You'll usually have the largest impact if you
focus your altruistic efforts, but our culture generally encourages a
people to take on a wide range of smaller things without considering
their tradeoffs.

How does our family put these ideas into practice? Where does
our time, money, and
attention go? How did we handle the intense pull to do unsustainable
amounts, and how has this changed over the years with getting older,
having kids, and lifestyle creep? How much does EA influence what we
do outside work?

I'd also include a short introduction to EA, key ideas from my other
posts (ex: The
Privilege of Earning to Give, Responsible
Transparency Consumption, Milk
EA, Casu Marzu EA), and concepts from related EA blog posts (ex:
Purchase
Fuzzies and Utilons Separately, Cheerfully).
I would want to stick to the "nonfiction" genre and not "memoir", with
the focus on EA and only using our family's experience to the extent
necessary to illustrate the application of these ideas. I could also
see chapters on how some specific other EAs have applied these ideas,
interviewing them and writing it up as prose?

There are already several books aiming to introducing EA to a general
audience, and if I thought that everything I wanted to say had already
been said I wouldn't be very interested in this project. Looking over
these books, however, I do think there's a place for what I want to
write. The main EA books (Doing Good
Better, The Precipice, What We Owe The Future) are
primarily moral arguments. While they do get into the more p...
AI Safety - 7 months of discussion in 17 minutes
by Zoe Williams — 2023-03-15T23:41:37.390Z

In August 2022, I started making summaries of the top EA and LW forum posts each week. This post collates together the key trends I’ve seen in AI Safety discussions since then. Note a lot of good work is happening outside what's posted on these forums too! This post doesn't try to cover that work.

If you’d like to keep up on a more regular basis, consider subscribing to the

Weekly EA & LW Forum Summaries

. And if you’re interested in similar overviews for other fields, check out

this post

covering 6 months of animal welfare discussion in 6 minutes.

Disclaimer: this is a blog post and not a research report - meaning it was produced quickly and is not to our (Rethink Priorities') typical standards of substantiveness and careful checking for accuracy. Please let me know if anything looks wrong or if I've missed key pieces!

Table of Contents

(It's a long post! Feel free to pick and choose sections to read, they 're all written to make sense individually)

Key Takeaways

Resource Collations

AI Capabilities

Progress
What AI still fails at
Public attention moves toward safety

AI Governance

AI Safety Standards
Slow down (dangerous) AI
Policy

US / China Export Restrictions
Paths to impact

Forecasting

Quantitative historical forecasting
Narrative forecasting

Technical AI Safety

Overall Trends
Interpretability
Reinforcement Learning from Human Feedback (RLHF)
AI assistance for alignment
Bounded AIs
Theoretical Understanding

Outreach & Community-Building

Academics and researchers
University groups
Career Paths

General guidance
Should anyone work in capabilities?

Arguments for and against high x-risk

Against high x-risk from AI
Counters to the above arguments

Appendix - All Post Summaries

Key Takeaways

There are multiple living websites that provide good entry points into understanding AI Safety ideas, communities, key players, research agendas, and opportunities to train or enter the field. (see more)

Large language models like ChatGPT have drawn significant attention to AI and kick-started race dynamics. There seems to be slowly growing public support for regulation. (see more)

Holden Karnofsky recently took a leave of absence from Open Philanthropy to work on AI Safety Standards, which have also been called out as important by leading AI lab OpenAI. (see more)

In October 2022, the US announced extensive restrictions on the export of AI-related products (eg. chips) to Ch

...
How well did Manifold predict GPT-4?
by David Chee — 2023-03-15T23:19:06.477Z

Chat GPT-4 is already here!! Who could have seen that coming… oh wait Manifold (kinda) did? 😅

I thought I’d write a short piece on how Manifold Markets was used to predict the launch of GPT-4 and its attributes. Both its successes and its failures. Disclaimer I work at Manifold.

How well did we predict the launch date?

Throughout the end of last year, people were bullish on a quick release, which began to decline as we entered the start of this year.

The first spike in February corresponds to the release of Bing’s chatbot which people speculated was Chat CPT-4. Turns out it actually was! Although Open AI did a fantastic job at concealing this with our market on it hovering at a stubborn 50-60%.

There was a lot of uncertainty on if GPT-4 would be released before March. However, on the 9th of March Microsoft Germany CTO Andreas Braun mentioned at an AI kickoff event that its release was imminent which caused the market to jump.

Although the market graphs are a beautiful representation of hundreds of traders’ predictions, did they actually give us any meaningful information? One thing that stands out about these graphs in particular is the strong bets away from the baseline towards YES throughout February. Is this just noise, or is something more going on?

Insider Trading

Being the socialite I am, I go to a whopping one (1) social gathering a month!! At 100% of these, the SF Manifold Markets party and Nathan Young’s Thursday dinner, I spoke to someone who claimed they were trading on the Chat GPT-4 markets based on privileged insider information.

One of them got burnt as allegedly there were delays from the planned launch and they had gone all-in on the GPT-4 being released by a certain date.

I love knowing people with privileged information are able to safely contribute to public forecasts which wouldn’t be possible without a site like Manifold Markets. As they were trading from anonymous accounts I have no way of knowing whether they are the ones responsible for the large YES bets, but I suspect some of them are. That said, someone with insider knowledge would be better off placing a large limit order to buy YES just above the current baseline which would exert strong pressure to hold the market at/slightly above its current probability. Placing a large market order which causes the spikes...