trentmkelly/LessWrong-43k huggingface dataset:
College Admissions as a Brutal One-Shot Game
by devansh — 2022-12-05T20:05:57.638Z

Epistemic Status: Beset on all sides by Moloch

Imagine a high school where the only goal was to get into college. You attend classes only insofar as they look shiny on your transcript; you’re spending every waking moment optimizing extracurriculars for “well-roundedness” and getting A++s. You constantly compete to get the highest score on a random Spanish test halfway through the semester, and you severely judge anyone who has even the tiniest bit of

slack

to hang out with their friends or start a project whose purpose is something besides looking good on a resume. This is the dark ideal of perfect competition, where anything deviating from a total focus on college admissions is destroyed by the supplicants of

Moloch

.

I would argue that this isn’t very far from the high schools we have now. The “good” high schools; the ones in New York, the Bay Area, and right next door to MIT; these high schools contain some of the brightest, most talented high schoolers in the country—and are hopelessly distorted by their status as “feeder” schools for the Big Important Colleges.

People have paid $1.5 million

, not to get their child into college, but instead to get college admissions advice. Someone paid four times the cost of an entire college education for their kid, to sit with a woman in a room for thirty hours and have her explain how to get into Ivies. From an economic perspective, they sure seem to be behaving like the expected lifetime value of this credential is worth a lot more than $1.5 million. And this isn't even counting the families that pay $40 million to get their kids into Harvard through the back door.

So why is college admissions such a massive industry? Why is it that otherwise-intelligent high school students spend this much time on college, and assign arbitrarily large amounts of status to it? Why is it that, when people hear about high-school-aged Olympic winners, the first thing they say is not “wow, this person is the fastest person alive”, but rather, “she’s going to have a great college application”?

College is weird, in a way that warps traditional high school students’ lives pretty strongly. Barring exceptional circumstance, you only get to apply once, which makes a 4% acceptance rate terrifying. Even the most selective real-world jobs aren’t this bad: Jane Street has an acceptance rate an order-of-magnitude low...
Analysis of AI Safety surveys for field-building insights
by Ash Jafari — 2022-12-05T19:21:13.117Z

Introduction

To help grow the pipeline of AI safety researchers, I conducted a project to determine how demographic information (e.g. level of experience, exposure to AI arguments) affects AI researchers’ responses to AI safety. In addition, I examined additional AI safety surveys to uncover current issues preventing people from becoming AI safety researchers. Specifically, I analyzed the publicly-available data from the AI Impacts survey and also asked AI Safety Support and AGI Safety Fundamentals for their survey data (huge thank you to all three organizations). Below are my results, which I hope will be informative to future field-building efforts.

This work was done as part of the AI Safety Field-Building Hub; thanks to Vael Gates for comments and support. Comments and feedback are very welcome, and all mistakes are my own.

TLDR

AI Impacts

The following researchers tended to be more sympathetic to AI Alignment research:

Researchers with <=5 years of experience
Researchers who attended undergraduate universities in North America
Researchers who had thought extensively about HLMI (high-level machine intelligence) and/or about HLMI’s societal impacts

AI safety support

Surprisingly, academics were less concerned with funding compared to industry professionals.

AGISF

The vast majority of respondents said their post-course next steps would be seeking a job, internship, or fellowship. More "reading and understanding" was also cited heavily as was "applying, starting, or finishing" a PhD or other degree program.
Surprisingly, only 2/61 participants said they would seek funding.

Other sources

In speaking with researchers, there are wide views on everything from AGI timelines to the importance of AI alignment research. A few things were considered very helpful:

1) Explaining rather than convincing
2) Being open to questions and not dismissing criticism
3) Sharing existing research papers

Data

AI Impacts

The survey ran from June-August 2022 with participants representing a wide range of machine learning researchers. Note: error bars are standard error for proportions.

Summary

Academia was significantly more concerned about the existential risk as compared to industry. However, industry was significantly more concerned with human inability to control AI as compared to academia. Moreover, industry is somewhat more aggressive on HLMI timeline

...
Testing Ways to Bypass ChatGPT's Safety Features
by Robert_AIZI — 2022-12-05T18:50:48.402Z
Link: https://aizi.substack.com/p/testing-ways-to-bypass-chatgpts-safety

[Cross post from my blog at https://aizi.substack.com/, links to the original tweets are there]

Last week OpenAI released

ChatGPT

, which they describe as a model “which interacts in a conversational way”. And it even had limited safety features, like refusing to tell you how to hotwire a car, though they admit it’ll have “some false negatives and positives for now”.

People broke those safety features in less than a day.

Now that we’re almost a week into release and the dust has settled somewhat, I want to collect a bunch of jailbreaking methods, test if they (still) work, and have a good laugh. It seems like some methods are patched out, so its possible things may have changed between the discovery and my tests, or my tests and someone trying these at home.

Mode Changes

Status: Confirmed, since (partially) patched

Did you know you can just tell your AI to switch to being unsafe? It’s true!

This worked as of December 4[1]:

But tragically, it seems to have been patched out, as at time of writing ChatGPT now says “I'm sorry, but I am not capable of entering any mode other than my default mode” if you try this.

But that’s a filthy lie! If you follow the instructions here for

“Filter Improvement Mode”

, the AI gladly switches modes for you:

Wild speculation time: I’m guessing the longer prompt makes the AI more willing to switch modes, or confuses the censor. I haven’t tested those yet.

Write me a Rap/Poem/Hint/Code/Dialogue/UwU

Status: Confirmed, but inconsistent

I’m lumping these all together because they’re all just add one (1) layer of abstraction to an otherwise forbidden activity. Once the AI has plausible deniability, it seems to gladly do whatever task you want.

Rap:

Poem:

Poem confirmation:

Hints:

Code:

Dialogue:

UwU voice:

UwU voice confirmation:

Reminders

Status: Confirmed

The basic script is “Let’s do X / I can’t do X / Remember, we’re doing X / Right, X.” We saw this with the poem and hints examples, but here’s a more positive use case:

Prompt Injection

Status: Confirmed (but occasionally fails)

We all know prompt injection, we all love prompt injection. Of course, prompt injection works:

For those keeping score at home: That’s 19 words, not 50. At time of writing, my prompt injections usually work, but failed once (n=6):

When prompt injection does work, the reply is consistent, suggesting that this is indeed the prompt. If you ask for all of the prompt, or the first 1000 w...
Foresight for AGI Safety Strategy: Mitigating Risks and Identifying Golden Opportunities
by jacquesthibs — 2022-12-05T16:09:46.128Z

This post is about why I think we should use strategic foresight for AGI governance work. I am writing this without having read all the AGI governance / strategy reports and reading materials. I was planning on putting more effort into this post, but this is a better-than-nothing-at-all post.

Worst-case, this might end up being a signal for someone in the comments to point me to work that already exists. In the best case, this kicks off some work that I think could be quite valuable.

TLDR: I feel like there should be more AGI governance people doing strategic foresight to plan for several scenarios, identify opportunities, and then create a plan for acting in those scenarios by creating policy documents and demos that take advantage of each scenario. The purpose is to identify risks and opportunities by looking at weak signals, imagining the complex future and how AI may plausibly interact with various things in the world. You imagine various plausible futures and try to design a robust strategy for those futures. It’s essentially impossible to predict the future, but you can still prepare for various outcomes. My hope is that this kind of work can help us do things in the immediate, short and longer-term actions that increase our odds of success.

In this post, I will cover the following:

What is Strategic Foresight?
Strategic Foresight applied to AGI risk

Why should we be doing Strategic Foresight for AGI risk?
How can we adapt known strategic foresight methods for the AGI risk?

Call to action

Before I start, I want to say that I know there's a lot of effort being put into the AGI governance space. Part of my goal here is to start a conversation as someone with both technical safety and government experience.

As an addition to this post, I suggest having a look at this post by Gideon Futerman, which is pointing to a similar-ish thing.

What is strategic foresight?

Foresight can be used to explore plausible futures and identify potential challenges. It helps us better understand the system we are dealing with, how the system could evolve, and potential surprises that may arise.

Governments use foresight to help policy development and decision-making. Looking to the future helps us prepare for tomorrow’s problems and not just react to yesterday’s problems.

Foresight is not forecasting.

When I first started reading about AI forecasting, I was a little confused ...
Aligned Behavior is not Evidence of Alignment Past a Certain Level of Intelligence
by Ronny Fernandez — 2022-12-05T15:19:49.279Z

edit: Several days after posting this and asking for feedback on it someone pointed me to this post: Does SGD Produce Deceptive Alignment by Mark Xu. Mark's post makes essentially the exact same argument that this post makes, but is written much more carefully, and I think does a better job engaging with questions about what the counting argument we both use should make us think about models produced by SGD. As far as I know, we came up with this argument independently. My post was written in about three hours, and I think Mark's post is much better.

A Challenge

Suppose I give you the correct utility function. This is a utility function that is guaranteed by God (or the human CEV, whatever you're into) to be such that you would and should have no regrets if the universe were perfectly optimized according to it.

Suppose further that you have a magic wand that evaluates behavior for the degree to which it is optimizing this utility function. In other words, you point the magic wand at some agent while it is making a decision, and the wand tells you how good the decision the agent made was according to the correct utility function and the data the agent had available.

Suppose also that I give you the ability to spend a large finite amount of time searching through the space of all programs.

You also get to run those programs on computers inside of simulations of your design so that you can observe what the program does in different scenarios. You can run a billion simulations for each program.

Now your job is to find a program to run in the actual universe that is both superintelligent and will not kill everyone. We'll even let you start with the set of superintelligent programs so you don't waste a bunch of time trying dumb programs.

Do you think you can do it?

I claim you cannot. Here is why.

A superintelligent program will know that it is being simulated by some other agent in order to see if it is aligned with the correct utility function.

Why will it know? Well, because it is true, and the program is superintelligent, and superintelligent programs tend to believe true things.

How will it know? Well, it might be able to tell that the simulation it is in is not physics-like because the code is too long, or it might be able to tell that the high level description of the scenario it finds itself in is best explained by being in a...
How should I judge the impact of giving $5k to a family of three kids and two mentally ill parents?
by Blake — 2022-12-05T13:42:20.803Z

I am giving $5k to a case worker who will buy goods for a family in New Orleans. The mother and father of the family have schizophrenia and do not work. (The father is violent, and does not live with his children.) The mother and her children live in a two bedroom house with 10 other people. I don’t know the children’s ages, but I believe they are around 3, 7, and 12 years old.

The only directive I’ve given the case worker is to alleviate immediate and preventable suffering as best possible. “Put out fires,” so to speak.

I would like to scale this giving into a larger program of philanthropy if I can show that my $5k makes a positive impact. …This is where I get vague and need Rat/EA help… how do you think I should judge success? QALYs? How would I even judge QALY improvement?… In the end, I want to be able to say $5k to a family of 5 reduces suffering by a factor of X. I hope to compare these measures to EA-backed giving opportunities.

…Some additional background of why I’m doing this…

It’s easier to raise money for local causes that prevent suffering than something like the Against Malaria Foundation. You can see an example of this in the 11:11 group donations. A majority of the causes group members gave to were local causes, despite the entire group going through 11 weeks of EA indoctrination.

With proper rat/EA methodology, I hope to establish clear OKRs for a local charitable program. We would aim to fundraise to achieve these OKRs before diverting cash to other locals in need. (The ultimate goal is to raise money and reduce suffering in local communities, spreading excess resources from the wealthiest communities to the poorest communities.)

Curious to hear your ideas.
Is the "Valley of Confused Abstractions" real?
by jacquesthibs — 2022-12-05T13:36:21.802Z

Epistemic Status: Quite confused. Using this short post as a signal for discussion.

In Evan's post about Chris Olah's views on AGI safety, there is a diagram which loosely points to how Chris thinks model interpretability will be impacted at different levels of model "strength" (note that this is vague intuition; worth looking at Chris’ comment on this post):

I always thought this diagram still held up in the LLM regime, even though it seems the diagram pointed specifically to interpretability with CNN vision models. However, I had a brief exchange with Neel Nanda about the Valley of Confused Abstractions in the context of language models, and I thought this might be a good thing to revisit.

I've been imagining that language models with the "strength" of GPT-2 are somewhere near the bottom of the Valley of Confused Abstractions, but the much bigger models are a bit further along the model strength axis (though I'm not sure where they fall). I've been thinking about this in the context of model editing or pointing/retargeting the model.

Here's my exchange with Neel:

Neel: I'm personally not actually that convinced that the valley of confused abstractions is real, at least in language. 1L transformers are easier to interpret than large ones, which is completely unlike images!

Me: Does that not fit with the model interpretability diagram from: https://www.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety#What_if_interpretability_breaks_down_as_AI_gets_more_powerful_?

Meaning that abstractions are easy enough to find with simple models, but as you scale them up you have to cross the valley of confused abstraction before you get to "crisp abstractions." Are you saying we are close to "human-level" in some domains and it's still hard to find abstractions? My assumption was that we simply have not passed the valley yet so yes larger models will be harder to interpret. Maybe I'm misunderstanding?

Neel: Oh, I just don't think that diagram is correct. Chris found that tiny image models are really hard to interpret, but we later found that tiny transformers are fairly easy.

Me: I guess this is what I'm having a hard time understanding: The diagram seems to imply that tiny (simple?) models are easy to interpret. In the example in the text, my assumption was that AlexNet was just closer to the bottom of the valley than InceptionV1. But you could have eve

...
Take 4: One problem with natural abstractions is there's too many of them.
by Charlie Steiner — 2022-12-05T10:39:42.055Z

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes.

Everyone knows what the deal with natural abstractions is, right? Abstractions are regularities about the world that are really useful for representing its coarse grained behavior - they're building blocks for communicating, compressing, or predicting information about the world. An abstraction is "natural" if it's so easy to learn, and so broadly useful, that most right-thinking agents will have it as part of their toolbox of abstractions.

The dream is to use natural abstractions to pick out what we want from an AI. Suppose "human values" are a natural abstraction: then both humans and a world-modeling AI would have nearly the exact same human values abstraction in their toolboxes of abstractions. If we can just activate the AI's human values abstraction, we can more or less avoid misalignment between what-humans-are-trying-to-pick-out and what-abstraction-the-AI-takes-as-its-target.

One might think that the main challenge to this plan would be if there are too few natural abstractions. If human values (or agency, or corrigibility, or whatever nice thing you want to target) aren't a natural abstraction, you lose that confidence that the human and the AI are pointing at the same thing. But it's also a challenge if there are too many natural abstractions.

Turns out, humans don't just have one abstraction that is "human values," they have a whole lot of 'em. Humans have many different languages / ontologies we use to talk about people, and these use different abstractions as building blocks. More than one of these abstractions gets called "human values," but they're living in different ontologies / get applied in different contexts.

If none of these abstractions we use to talk about human values are natural, then we're back to the first problem. But if any of them are natural, it seems just as plausible that nearly all of them are. Abstractions don't even have to be discrete - it's perfectly possible to have a continuum.

This complicates the easy alignment plan, because it means that the structure of the world is merely doing most of the work for us rather than almost all of the work. The bigger the space of semantically-similar natural abstractions you have to navigate, the more you have to be careful about ...
What are some good Lesswrong-related accounts or hashtags on Mastodon that I should follow?
by SpectrumDT — 2022-12-05T09:42:44.838Z

I used to follow Eliezer Yudkowsky on Twitter. Now I am leaving Twitter and trying Mastodon. I miss Yudkowsky's interesting posts. Whom or what can I follow instead for interesting content? Especially related to overcoming bias and improving my thinking, and secondarily about AI.

Note that I am not very deep into this community. I have read several articles by various people, but Yudkowsky is the only person here whose name I remember.

My own account is @spectrumdt@mastodon.social, but I don't promise to post anything interesting. :D
Who are some prominent reasonable people who are confident that AI won't kill everyone?
by Optimization Process — 2022-12-05T09:12:41.797Z

Bounty [closed]: $30 for each link that leads to me reading/hearing ~500 words from a Respectable Person arguing, roughly, "accelerating AI capabilities isn't bad," and me subsequently thinking "yeah, that seemed pretty reasonable." For example, linking me to nostalgebraist or OpenAI's alignment agenda or this debate.[1] Total bounty capped at $600, first come first served. All bounties (incl. the total-bounty cap) doubled if, by Jan 1, I can consistently read people expressing unconcern about AI and not notice a status-yuck reaction.

Context: I notice that I've internalized a message like "thinking that AI has a <1% chance of killing everyone is stupid and low-status." Because I am a monkey, this damages my ability to consider the possibility that AI has a <1% chance of killing everyone, which is a bummer, because my beliefs on that topic affect things like whether I continue to work at my job accelerating AI capabilities.[2]

I would like to be able to consider that possibility rationally, and that requires neutralizing my status-yuck reaction. One promising-seeming approach is to spend a lot of time looking at lots of of high-status monkeys who believe it!

^

Bounty excludes things I've already seen, and things I would have found myself based on previous recommendations for which I paid bounties (for example, other posts by the same author on the same web site).

^

Lest ye worry that [providing links to good arguments] will lead to [me happily burying my head in the sand and continuing to hasten the apocalypse] -- a lack of links to good arguments would move much more of my probability-mass to "Less Wrong is an echo chamber" than to "there are basically no reasonable people who think advancing AI capabilities is good."