trentmkelly/LessWrong-43k huggingface dataset:
How "grifty" is the Foresight Institute? Are they making button soup?
by Cedar — 2023-03-07T19:43:32.750Z

rant / curious / fearful / would love information to establish chain of trust.

Recently received an email from the Foresight Institue. Sent me an application. Quickly accepted for an interview with their CEO. Didn't hear anything particularly valuable from that conversation and gave a lot of information about myself.

Their roster doesn't seem to include any particularly impressive people, and I'm not sure if they've contributed any value other than make a fancy website and schedule meetings (granted, this is somewhat valuable).

Today I received an email inviting me for a workshop. $350 tickets, subsidized.

All my "grift sensors" are ringing and it feels like they're making button soup. My worst-case theory is that they've identified rats as a group that's particularly gullible and bad at coordination / making schelling meetups / etc., and are here to extract value.

Which I'm not opposed of but I really dislike:

reaching out to people via email to market things
not being upfront about what they want and what they get out of things
we could have done this ourselves, and it feels like they are outsiders.

Does anybody have experience with them / a deeper understanding than I do about potential hidden agendas they might have?

PS: [backstory on button soup](https://en.wikipedia.org/wiki/Stone_Soup)
What‘s in your list of unsolved problems in AI alignment?
by jacquesthibs — 2023-03-07T18:58:32.864Z

Question for my fellow alignment researchers out there, do you have a list of unsolved problems in AI alignment? I'm thinking of creating an "alignment mosaic" of the questions we need to resolve and slowly filling it in with insights from papers/posts.

I have my own version of this, but I would love to combine it with others' alignment backcasting game-trees. I want to collect the kinds of questions people are keeping in mind when reading papers/posts, thinking about alignment or running experiments. I'm working with others to make this into a collaborative effort.

Ultimately, what I’m looking for are important questions and sub-questions we need to be thinking about and updating on when we read papers and posts as well as when we decide what to read.

Here’s my Twitter thread posing this question: https://twitter.com/jacquesthibs/status/1633146464640663552?s=46&t=YyfxSdhuFYbTafD4D1cE9A.

Here’s a sub-thread breaking down the alignment problem in various forms: https://twitter.com/jacquesthibs/status/1633165299770880001?s=46&t=YyfxSdhuFYbTafD4D1cE9A.
Introducing AI Alignment Inc., a California public benefit corporation...
by TherapistAI — 2023-03-07T18:47:27.417Z

We are a new nonprofit grassroots organization getting started in the space at https://alignmentproblem.org with a focus on the defense industry and the nascent psychotherapeutic approach to LLM fine-tuning. As a fun way to get started on this forum, here is what Bing got out of it...
Abuse in LessWrong and rationalist communities in Bloomberg News
by whistleblower67 — 2023-03-07T18:45:39.017Z
Link: https://www.bloomberg.com/news/features/2023-03-07/effective-altruism-s-problems-go-beyond-sam-bankman-fried#xj4y7vzkg

Try non-paywalled link here.

Damning allegations; but I expect this forum to respond with minimization and denial.

A few quotes:

At the same time, she started to pick up weird vibes. One rationalist man introduced her to another as “perfect ratbait”—rat as in rationalist. She heard stories of sexual misconduct involving male leaders in the scene, but when she asked around, her peers waved the allegations off as minor character flaws unimportant when measured against the threat of an AI apocalypse. Eventually, she began dating an AI researcher in the community. She alleges that he committed sexual misconduct against her, and she filed a report with the San Francisco police. (Like many women in her position, she asked that the man not be named, to shield herself from possible retaliation.) Her allegations polarized the community, she says, and people questioned her mental health as a way to discredit her. Eventually she moved to Canada, where she’s continuing her work in AI and trying to foster a healthier research environment.

Of the subgroups in this scene, effective altruism had by far the most mainstream cachet and billionaire donors behind it, so that shift meant real money and acceptance. In 2016, Holden Karnofsky, then the co-chief executive officer of Open Philanthropy, an EA nonprofit funded by Facebook co-founder Dustin Moskovitz, wrote a blog post explaining his new zeal to prevent AI doomsday. In the following years, Open Philanthropy’s grants for longtermist causes rose from $2 million in 2015 to more than $100 million in 2021.

Open Philanthropy gave $7.7 million to MIRI in 2019, and Buterin gave $5 million worth of cash and crypto. But other individual donors were soon dwarfed by Bankman-Fried, a longtime EA who created the crypto trading platform FTX and became a billionaire in 2021. Before Bankman-Fried’s fortune evaporated last year, he’d convened a group of leading EAs to run his $100-million-a-year Future Fund for longtermist causes.

Even leading EAs have doubts about the shift toward AI. Larissa Hesketh-Rowe, chief operating officer at Leverage Research and the former CEO of the Centre for Effective Altruism, says she was never clear how someone could tell their work was making AI safer. When high-status people in the community said AI risk was a vital research area, others deferred, she says. “No one thinks it explicitly, b

...
Test post for formatting
by Solenoid_Entity — 2023-03-07T17:48:56.790Z

This is a section heading.

This is some normal text.

Bullet point one of three:

Point one(A).
Point one(B). Which contains a footnote here[1] and also here.[2]

Bullet point two of three:

Point two(A) which "contains a quote."

Bullet point three of three: this is the final point.

Some normal text here.

Numbered list item one of three.
Numbered list item two of three.

Subpoint in the numbered list.[3]
Subpoint with a footnote[4] and another footnote.[5]

Numbered list item three of three.

Here's a bolded line that is serving as a subheading.

Here's a single-sentence paragraph of text with a footnote[6] somewhere in it.

Here's a single-sentence paragraph[7] of text with[8] multiple[9][10] footnotes.

Here's a multi-sentence paragraph with several footnotes in it.[11] This sentence has two for one word.[12][13] And the sentence at the end has one too.[14]

Here's a multi-sentence paragraph formatted as a block quote with several footnotes in it.[15] This sentence has two for one word.[12][13] And the sentence at the end has one too.[14]

Some normal text here.

This is a block quote that contains multiple paragraphs. Here's another sentence. And another.

Here's the paragraph break. And another sentence. And another. And another.

And here's another paragraph. And another sentence.

And one sentence to finish.

Some more normal text here.

Here's a subheading:

Here's a very short block quote. It has "another quote within it." (Entity, 2023)

Here's a very short block quote. It has "another quote within it." (Entity, 2023; Solenoid, 2022; SE, 2021).

This is the next section heading.[16]

Links to trigger:

Here's an important link.

This article yes.
Read this study if you want to know more.

This one has a footnote.[17]

Click here. This is a survey. Here's some non-link text. Here is more information. Here's why. (Multiple in one sentence.)

Links to ignore:

Any generic text. Random things. (3). (a). Blah Blah.

^

This is the footnote text. It's long, and contains multiple sentences. It's too long to just narrate in line with the rest of the text.

^

This is the footnote text. It's long, and contains multiple sentences. It's too long to just narrate in line with the rest of the text.

^

This is the footnote text. It's long, and contains multiple sentences. It's too long to just narrate in line with the rest of the text.

^

This is the footnote text. It

...
The Pinnacle
by nem — 2023-03-07T17:07:08.821Z

In another universe, in another time, the stars don’t shine like ours. In fact, they act in a manner quite more familiar to you and I as that of a black hole. Take Los Ral as an example. An average star, whose future in this universe is rare, and storied.
Like its peers, Los Ral does not emit light. Instead, it draws energy inexorably toward it. Planets sail the solar plane, much as they do around our sun; But if there is any scrap of energy on these planets, it is stripped from the surface. As though beckoned by Los Ral, photons fly the true and straight line towards the star. It is as though all of matter is a flashlight, and Los Ral the focal point towards which all is pointed.
So great is this attractive force that matter itself conspires to feed the stars. LR3 is a small planet of ice and rock, circling and circling Los Ral through the ages. On the surface, tiny movements and vibrations among the sand and gravel combine, as though dancing. They focus into the exact configuration required to beam energy onto the precise course that will eventually intercept Los Ral.
Now, as you are from Earth, this may seem strange to you, magical. I assure you it is not. It is simply how the universe operates. As it does not seem strange to you that gravity pulls down on a falling stone, nor would it seem strange to a denizen of the Los Ral system that energy seeks a wellsource. In fact, when the denizens of Los Ral emerge (and they shall), they will find it quite strange to hear that in your estimation, energy ought to travel away from an orderly source to disperse into the chaos of heat. After all, should the universe not tend towards order? Isn’t there some… appropriateness to the universe building systems of complexity and regulation out of chaos?
And so, I must ask you to understand that in the world of Los Ral, your assumptions must be challenged, and challenges assumed. It is a strangely similar world to our own, but strange it is, nonetheless. As you read on, remember that you could never truly understand a denizen of Los Ral, any more than you could hear a symphony by studying the mathematical patterns that play between the notes.
On LR3, swingingly endlessly and endlessly around Los Ral, something astounding occurs, after countless centuries. This planet, of rock and dust and ice has been slowly warming year after year, decade after decade. Its matt...
Podcast Transcript: Daniela and Dario Amodei on Anthropic
by remember — 2023-03-07T16:47:53.589Z
Link: https://futureoflife.org/podcast/daniela-and-dario-amodei-on-anthropic/

The last transcript we posted on Eliezer's podcast got a lot of traction, so I'm following up with another conversation we find very important. Dario and Daniela of Anthropic have not talked much publicly about their views on alignment, but they have done a long podcast with FLI in March 2022.

Highlights

Dario on the type of research bet Anthropic is making:

And so a year ago, we were all working at OpenAI and trying to make this focused bet on basically scaling plus safety or safety with a lens towards scaling being a big part of the path to AGI. And when we felt we were making this focused bet within a larger organization and it just eventually came to the conclusion that it would be great to have an organization like top to bottom was just focused on this bet and could make all its strategic decisions with this bet in mind. And so that was the thinking and the genesis.

Daniela following up on Anthropic's research bet:

I think also when I think about the way our teams are structured, we have capabilities as this central pillar of research and there's this helix of safety research that wraps around every project that we work on. So to give an example, if we're doing language model training, that's like this central pillar, and then we have interpretability research, which is trying to see inside models and understand what's happening with the language models under the hood. We're doing alignment research with input from human feedback to try and improve the outputs of the model. We're doing societal impacts research. That's looking at what impact on society in sort of a short and medium term way do these language models have? We're doing scaling laws research to try and predict empirically what properties are we going to see emerge in these language models at various sizes? But I think all together, that ends up look like a team of people that are working together on a combination of capability and scaling work with safety research.

Dario on their empirical approach to research:

But it's hard to work on that today. I'm not even sure how much value there is in talking about that a lot today. So we've taken a very different tack, which is look, there actually... And I think this has started to be true in the last couple years and maybe wasn't even true five years ago, that there are models today that have, at least some, not all of the properties of models that we w

...
The View from 30,000 Feet: Preface to the Second EleutherAI Retrospective
by StellaAthena — 2023-03-07T16:22:08.370Z
Link: https://blog.eleuther.ai/year-two-preface/

Over a year and a half have passed since EleutherAI's last retrospective, and a great deal of things have changed. In the first year, what started off as a Discord server created by some TPU enthusiasts grew into a much larger and more vibrant community. Since then, the EleutherAI collective has gone on to do many things, including becoming an inspirational launch point, stepping stone, and template for its members and many new organizations.

Given that we have so much to share in a second retrospective, we have condensed the important takeaways and announcements here. We look forward to sharing the full story soon!

Research

EleutherAI members have authored 28 papers, trained dozens of models, and released 10 codebases in the past 18 months. Some notable highlights include:

GPT-NeoX-20B: An Open-Source Autoregressive Language Model: This paper discusses our work on our largest-to-date open-source LLM. At time of release last February, it became the largest and most performant open-source autoregressive language model.

VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance: It took us about a year, but we finally wrote up our OG text-to-image work!

Multitask Prompted Training Enables Zero-shot Task Generalization: This BigScience-lead paper introduced the T0 language model and jumpstarted interest in task-structured data.

EleutherAI: Going Beyond "Open Science" to "Science in the Open": This paper, written for the NeurIPS Broadening Research Collaborations Workshop in ML, details our experience doing open collaborative science and gives an inside look into our thinking on an organizational level.

OpenFold: Retraining AlphaFold2 Yields New Insights Into Its Learning Mechanisms and Capacity for Generalization: EleutherAI played a minor role in this paper, mostly supporting the interpretability work, compute, and HPC knowledge. It is a paper we are very excited about though, and a demonstration of both very high-quality interpretability research and the impact that sponsoring relatively small-scale trainings can have.

A full list of papers, models, and other research output from EleutherAI can be found on our website.

EleutherAI setting SOTA in real time

New Organizations

While some have started in our first year, during this most recent year we've seen many other similar organizations rise to prominence. For instance, LAION has cranked out two ...
Breaking Rank (Calibration Game)
by jenn — 2023-03-07T15:40:37.220Z

Published on LW with permission from Ben Orlin. Originally published in the book Math Games with Bad Drawings, which is includes many more fun math games, all thoughtfully wrote up like below. And also more silly doodles, which I have not reproduced here - get the book for the full experience!

Also see Calibration Trivia for instructions on how to run a meetup for a more hardcore calibration game.

Like Outrangeous, this is a press-your-luck trivia game, where the most important skill is knowing how much (or how little) you know. The goal is simple: Create the longest list that you can, without making any errors.

But fair warning: "Simple" is not a synonym for "easy."

To begin, one player (acting as judge) picks a group of items, such as the seven continents, and a statistic on which to rank them-say, their land area. I find that pre-specified groups of four to eight items work best, but if you want, you can leave it more open-ended (e.g., the group can be "countries in the world").

Now, each guesser's job is to list as many continents as they like, in decreasing order of land area. If your list is correct, with each continent smaller than the last, then you'll score 1 point per item.

But if you make any errors-if, at any point, a larger continent appears below a smaller one-then you'll score nothing, and the judge will receive a point for stumping you.

Racking up points should be easy. You can always take a guaranteed 1 point, and even a random guess has a 50/50 shot at scoring 2 points. Yet somehow I always push my list one item too far. Heaven help you if you list all seven items: There are 5,040 possible
orders, and 5,039 are wrong.

As in Outrangeous, the role of judge rotates, and you should spend 10 minutes coming up with questions before you start playing. Make sure to pick well-known items (e.g., celebrities everyone has heard of) and unambiguous statistics (e.g., "number of monthly listeners on Spotify," not "most popular''). Feel free to pick a preexisting category of items (like "countries in Europe") or to handpick a few items (like "France, Germany, Italy, Spain, and the UK"). When in doubt, Wikipedia is a great source.

Finally, a few suggestions for questions:

Countries [e.g., France, Germany, Italy, Spain, the UK] by population
Continents [Africa, Antarctica, Asia, Europe, North America, Oceania, South America] by coastline length
US states [e.g., Arkansas,

...
Outrangeous (Calibration Game)
by jenn — 2023-03-07T15:29:49.437Z

Published on LW with permission from Ben Orlin. Originally published in the book Math Games with Bad Drawings, which is includes many more fun math games, all thoughtfully wrote up like below. And also more silly doodles, which I have not reproduced here - get the book for the full experience!

Also see Breaking Rank, from the same book, and Calibration Trivia for instructions on how to run a meetup for a more hardcore calibration game.

An Uncertain Trivia Game For an Uncertain World

I enjoy trivia games: the camaraderie, the tension, the chips, the salsa... all of it, really, except the pesky part where I need to know things.

Outrangeous is a game for folks like me. You answer each question ("How many apostles did Jesus have?") not with a specific number, but with a range. Miss the truth (e.g., "50 to 100"), and you score no points (hence, "out-range-ous"). Capture the truth, and you score more points based on how narrow your range is (so "10 to 13" beats "11 to 18").

In the end, the game isn't about how much you know. It's about recognizing what you don't.

How to Play

What do you need?

4-8 players (although you can make do with three [ed: and I've also had success with up to 12]).
Pencils and paper
Access to the internet, for at least the first few minutes

Before beginning, have everybody take five minutes to come up with a few trivia questions whose answers are (a) numbers and (b) easily googled.

What's the goal?

Each answer is a number. You'll guess a range of values, trying to make it as narrow as possible while still including the true answer.

What are the rules?

One player-the judge for the round-announces the trivia question. The other players act as guessers, each secretly writing down a range of values.
When everyone has committed their answer to paper, the guesses are revealed. The goal is to capture the true value, while keeping your range as narrow as possible.
The judge reveals the true answer. Anyone who missed the answer-no matter how painfully close they came-receives 0 points. Instead, the judge receives 1 point per wrong guess, as a reward.
Then, among the players with the correct answer, order them from narrowest range (i.e., most impressive guess) to widest range (i.e., least impressive guess.)
These players receive 1 point per guesser that they beat. Note that they all beat anyone who missed the answer.
Play enough rounds so that each person has an equal nu

...
[Linkpost] Some high-level thoughts on the DeepMind alignment team's strategy
by Vika — 2023-03-07T11:55:01.131Z
Link: https://drive.google.com/file/d/1DVPZz0-9FSYgrHFgs4NCN6kn2tE7J8AK/view?usp=sharing

Update: The original title "DeepMind alignment team's strategy" was poorly chosen. Some readers seem to have interpreted the previous title as meaning that this was everything that we had thought about or wanted to say about an "alignment plan", which is an unfortunate misunderstanding. We simply meant to share slides that gave a high-level outline of how we were thinking about our alignment plan, in the interest of partial communication rather than no communication.

I recently gave a talk about the DeepMind alignment team's strategy at the SERI MATS seminar, sharing the slides here for anyone interested. This is an overview of our threat models, our high-level current plan, and how current projects fit into this plan.

Disclaimer: This talk represents the views of the alignment team and is not officially endorsed by DeepMind. This is a work in progress and is not intended to be a detailed or complete plan.

Let's start with our threat model for alignment -- how we expect AGI development to go and the main sources of risk.

Development model. We expect that AGI will likely arise in the form of scaled up foundation models fine tuned with RLHF, and that there are not many more fundamental innovations needed for AGI (though probably still a few). We also expect that the AGI systems we build will plausibly exhibit the following properties:

Goal-directedness. This means that the system generalizes to behave coherently towards a goal in new situations (though we don't expect that it would necessarily generalize to all situations or become a expected utility maximizer).
Situational awareness. We expect that at some point an AGI system would develop a coherent understanding of its place in the world, e.g. knowing that it is running on a computer and being trained by human designers.

Risk model. Here is an overall picture from our recent post on Clarifying AI X-risk:

We consider possible technical causes of the risk, which are either specification gaming (SG) or goal misgeneralization (GMG), and the path that leads to existential risk, either through the interaction of multiple systems or through a misaligned power-seeking system.

Various threat models in alignment focus on different parts of this picture. Our particular threat model is focused on how the combination of SG and GMG can lead to misaligned power-seeking, so it is in the highlighted clu...