trentmkelly/LessWrong-43k huggingface dataset:
The Pointer Resolution Problem
by Jozdien — 2024-02-16T21:25:57.374Z

Thanks to Garrett Baker and Evžen Wybitul for discussions and feedback on this post.

Imagine that you meet an 18th century altruist. They tell you “So, I’ve been thinking about whether or not to eat meat. Do you know whether animals have souls?” How would you answer, assuming you actually do want to be helpful?

One option is to spend a lot of time explaining why “soul” isn’t actually the thing in the territory they care about, and talk about

moral patienthood

and theories of

welfare and moral status

. If they haven’t walked away from you in the first thirty seconds this may even work, though I wouldn’t bet on it.

Another option is to just say “yes” or “no”, to try and answer what their question was pointing at. If they ask further questions, you can either dig in deeper and keep translating your real answers into their ontology or at some point try to retarget their questions’ pointers toward concepts that do exist in the territory.

Low-fidelity pointers

The problem you’re facing in the above situation is that the person you’re talking to is using an inaccurate ontology to understand reality. The things they actually care about correspond to quite different objects in the territory. Those objects currently don’t have very good pointers in their map. Trying to directly redirect their questions without first covering a fair amount of context and inferential distance over what these objects are probably wouldn’t work very well.

So, the reason this is relevant to alignment:

Representations of things within the environment are learned by systems up to the level of fidelity that’s required for the learning objective. This is true even if you assume a weak version of the natural abstraction hypothesis to be true; the general point isn’t that there wouldn’t be concepts corresponding to what we care about, but that they could be very fuzzy.

For example, let’s say that you try to

retarget an internal general-purpose search process

. That post describes the following approach:

Identify the AI’s internal concept corresponding to whatever alignment target we want to use (e.g. values/corrigibility/user intention/human mimicry/etc).
Identify the retargetable internal search process.
Retarget (i.e. directly rewire/set the input state of) the internal search process on the internal representation of our alignment target.

There are - very broadly, abstracting over a fair amount o...
Every "Every Bay Area House Party" Bay Area House Party
by Richard_Ngo — 2024-02-16T18:53:28.567Z

Inspired by Ricki Heicklen's house party inspired by

Scott Alexander

.

By the time you arrive in Berkeley, the party is already in full swing. You’ve come late because your reading of the polycule graph indicated that the first half would be inauspicious. But now you’ve finally made it to the social event of the season: the Every Bay Area House Party-themed house party.

The first order of the evening is to get a color-coded flirting wristband, so that you don’t incur any accidental

micromarriages

. You scan the menu of options near the door. There’s the wristband for people who aren’t interested in flirting; the wristband for those want to be flirted with, but will never flirt back; the wristband for those who only want to flirt with people who have different-colored wristbands; and of course the one for people who want to

glomarize

disclosure of their flirting preferences. Finally you reach down and grab the last one: the wristband for those who only flirt with those who don’t flirt with themselves. As you slip it over your wrist, you notice it’s fastened in a Mobius strip.

You scan around the living room, trying to figure out who to talk to first. The host is sitting on the sofa, with two boxes attached to the front of her shirt. One is filled with money, the other empty. A guy next to her is surreptitiously one-boxing, but she presciently slaps his hand away without even looking. You decide to leave them to it. On the other side of a room, there’s a lone postrationalist, surrounded by a flock of alignment researchers. You hear a snatch of their conversation: “–but what part of your model rules out FOOM? Surely–”. As they keep talking, the postrationalist looks increasingly uncomfortable, until eventually her interlocutor takes a breath and she seizes the opportunity to escape. You watch her flee down the street through the window labeled Outside View.

With the living room looking unpromising, you head into the kitchen to grab a drink. As you walk through the door, you hear a crunching sound from under your feet; glancing down, you see hundreds of paperclips scattered across the floor. On the table there are two big pitchers, carefully labeled. One says “For

contextualizers

”; the other says “For decouplers and homophobes”. You go straight for the former; it’s impossible to do any good countersignalling by decoupling these days....
"No-one in my org puts money in their pension"
by Tobes — 2024-02-16T18:33:28.996Z
Link: https://seekingtobejolly.substack.com/p/no-one-in-my-org-puts-money-in-their

Epistemic status: the stories here are all as true as possible from memory, but my memory is so so.

An AI made this

This is going to be big

It’s late Summer 2017. I am on a walk in the Mendip Hills. It’s warm and sunny and the air feels fresh. With me are around 20 other people from the Effective Altruism London community. We’ve travelled west for a retreat to discuss how to help others more effectively with our donations and careers. As we cross cow field after cow field, I get talking to one of the people from the group I don’t know yet. He seems smart, and cheerful. He tells me that he is an AI researcher at Google DeepMind. He explains how he is thinking about how to make sure that any powerful AI system actually does what we want it to. I ask him if we are going to build artificial intelligence that can do anything that a human can do. “Yes, and soon,” he says, “And it will be the most important thing that humanity has ever done.”

I find this surprising. It would be very weird if humanity was on the cusp of the most important world changing invention ever, and so few people were seriously talking about it. I don’t really believe him.

This is going to be bad

It is mid-Summer 2018 and I am cycling around Richmond Park in South West London. It’s very hot and I am a little concerned that I am sweating off all my sun cream.

After having many other surprising conversations about AI, like the one I had in the Mendips, I have decided to read more about it. I am listening to an audiobook of Superintelligence by Nick Bostrom. As I cycle in loops around the park, I listen to Bostrom describe a world in which we have created superintelligent AI. He seems to think the risk that this will go wrong is very high. He explains how scarily counterintuitive the power of an entity that is vastly more intelligent than a human is. He talks about the concept of “orthogonality”; the idea that there is no intrinsic reason that the intelligence of a system is related to its motivation to do things we want (e.g. not kill us). He talks about how power-seeking is useful for a very wide range of possible goals. He also talks through a long list of ways we might try to avoid it going very wrong. He then spends a lot of time describing why many of these ideas won’t work. I wonder if this is all true. It sounds like science fiction, so while I notice some vague discomfort with the ideas,...
Addressing Feature Suppression in SAEs
by Benjamin Wright — 2024-02-16T18:32:51.927Z

Produced as part of the ML Alignment Theory Scholars Program - Winter 2023-24 Cohort as part of Lee Sharkey's stream.

TL;DR

Sparse autoencoders are a method of resolving superposition by recovering linearly encoded “features” inside activations. Unfortunately, despite the great recent success of SAEs at extracting human interpretable features, they fail to perfectly reconstruct the activations. For instance, Cunningham et al. (2023) note that replacing the residual stream of layer 2 of Pythia-70m with the reconstructed output of an SAE increased the perplexity of the model on the Pile from 25 to 40. It is important for interpretability that the features we extract accurately represent what the model is doing.

In this post, I show how and why SAEs have a reconstruction gap due to ‘feature suppression’. Then, I look at a few ways to fix this while maintaining SAEs interpretability. By modifying and fine-tuning a pre-trained SAE, we achieve a 9% decrease in mean square error and a 24% reduction in the perplexity increase upon patching activations into the LLM.

Finally, I compare a theoretical example to the observed amounts of feature suppression in Pythia 70m, showing that features are suppressed based on both the strength of their activations and their frequency of activation.

Feature Suppression

The architecture of an SAE is:

f(x)=ReLU(Wex+be)

y=Wdf(x)+bd

The loss function usually combines a MSE reconstruction loss with a sparsity term, like L(x,f(x),y)=||y−x||2/d+c|f(x)|, where d is the dimension of x. When training the SAE on this loss, the decoder’s weight matrix is fixed to have unit norm for each feature (column).

The reason for feature suppression is simple: The training loss has two terms, only one of which is reconstruction. Therefore, reconstruction isn’t perfect. In particular, the loss function pushes for smaller f(x) values, leading to suppressed features and worse reconstruction.

An illustrative example of feature suppression

As an example, consider the trivial case where there is only one binary feature in one dimension. That is, x=1 with probability p and x=0 otherwise. Then, ideally the optimal SAE would extract feature activations of f(x)∈{0,1} and have a decoder with Wd=1.

However, if we were to train an SAE optimizing the loss function L(x,f(x),y)=|...
Retrospective: PIBBSS Fellowship 2023 
by DusanDNesic — 2024-02-16T17:48:32.151Z

Between June and September 2023, we (Nora and Dusan) ran the second iteration of the PIBBSS Summer Fellowship. In this post, we share some of our main reflections about how the program went, and what we learnt about running it.

We first provide some background information about (1) The theory of change behind the fellowship, and (2) A summary of key program design features. In the second part, we share our

reflections

on (3) how the 2023 program went, and (4) what we learned from running it.

This post builds on an extensive internal report we produced back in September. We focus on information we think is most likely to be relevant to third parties, in particular:

People interested in forming opinions about the impact of the PIBBSS fellowship, or similar fellowship programs more generally
People interested in running similar programs, looking to learn from mistakes that others made or best practices they converged to

Also see

our reflections on the 2022 fellowship program

. If you have thoughts on how we can improve, you can use

this name-optional feedback form

.

Background

Fellowship Theory of Change

Before focusing on the fellowship specifically, we will give some context on PIBBSS as an organization.

PIBBSS overall

PIBBSS

is a research initiative focused on leveraging insights and talent from fields that study intelligent behavior in natural systems to help make progress on questions in AI risk and safety. To this aim, we run several programs focusing on research, talent and field-building.

The focus of this post is our fellowship program - centrally a talent intervention. We ran the second iteration of the fellowship program in summer 2023, and are currently in the process of selecting fellows for the 2024 edition.

Since PIBBSS' inception, our guesses for what is most valuable to do have evolved. Since the latter half of 2023, we have started taking steps towards focusing on more concrete and more inside-view driven research directions. To this end, we started hosting several full-time research affiliates in January 2024. We are currently working on a more comprehensive update to our vision, strategy and plans, and will be sharing these developments in an upcoming post.

PIBBSS also pursues a range of other efforts aimed more broadly at field-building, including (co-)organizing a range of topic-specific AI safety...
Fatebook for Chrome: Make and embed forecasts anywhere on the web
by Adam B — 2024-02-16T16:08:15.917Z

Fatebook is the fastest way to track your predictions. Now we've made a Chrome extension that makes it even faster.

With

Fatebook for Chrome

, you can now create and embed forecasts inside Google Docs.

Or anywhere else on the web! Inside your to-do list...

Or even inside Google Meet!

To instantly create a forecast on any webpage, just press Ctrl-Shift-F, type your prediction, and hit enter:

Imagine you're writing a Google Doc – a report on the rate of AI progress. You want to write down a prediction: "The most capable LLM in 2026 will be made by OpenAI (80%)"

With Fatebook for Chrome, you can press Ctrl-Shift-F, write your prediction, and embed it right into your doc. Your forecast is recorded in Fatebook, so you won't lose track of it: you'll get a reminder to resolve it in 2026, and your accuracy counts towards your track record.

And for your colleagues reading your report, they'll see your prediction and can add their own, letting you harness the wisdom of the crowd, understand disagreements, and update your beliefs over time.

This is the next step on our mission to make Fatebook the fastest way to make and track predictions, and to make it super low friction to bring forecasting right into your team's workflows.

We're excited to hear how you use it – particularly if you're using it with your team, as we designed this tool with EA orgs and other high-impact teams in mind. Special thanks to @Jarred Filmer for his awesome engineering work to bring Fatebook into any site on the web!

You can

add Fatebook for Chrome to your browser

(whether you use Chrome, Edge, or Arc), and you can

see the extension in action here:

fatebook.io/extension
"Arctic Instincts? The universal principles of Arctic psychological adaptation and the origins of East Asian psychology" - Call for Reviewers (Seeds of Science)
by rogersbacon — 2024-02-16T15:02:52.135Z

Abstract

Modern East Asian populations plausibly retained extensive psychological adaptations in general personality traits and social dynamics to arctic environments, from ancestral Northeast Asian populations' late Pleistocene habitation of arctic and subarctic North Eurasia, prior to migrating southwards into East Asia in the Holocene. The first cross-cultural study between modern East Asian and Inuit populations is conducted, as an epistemically sound ethnographic analogy for paleolithic Arctic populations, and a shared set of arctic adapted personality traits is proposed- a higher than average tendency of emotional control/suppression, harmony/cohesion, indirectness, self and social consciousness, reserve/introversion, cautiousness, and perseverance/stoic endurance. The proposed candidate Arcticist traits are tested for environmental causation in modern polar personnel psychology, using decades of personality research on workers, expeditioners, and military personnel who adapt to Arctic and Antarctic environments. Consistently replicated results support causation of polar climates in necessitating the proposed Arcticist traits for adaptive success, and such traits are generalizable and predictive of adaptive success even across modern European populations that temporarily live in polar climates. A theoretical framework for the universal principles of arctic psychological adaptation is introduced, a novel methodology for testing causation and formation of psychological adaptation to post-OoA environments is proposed, and previous popular theories on the roots of Eastern psychology are reexamined in the light of Arcticism theory.

---

Seeds of Science is a scientific journal (funded through Scott Alexander's ACX grants program) publishing speculative or non-traditional research articles. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them). Comments that critique or extend the article (the "seed of science") in a useful manner are published in the final document following the main text.

We have just sent out a manuscript for review, "Arctic Instincts? The universal principles of Arctic psychological adaptation and the origins of East Asian psychology", that may be of interest to some in the LessWrong community so I wanted to see if anyone would be interested in joining us as a...
The Altman Technocracy
by PhilosophicalSoul — 2024-02-16T13:27:36.883Z

Imagine if the city of New York was inherited by newly sentient human beings a couple hundred years from now. These descendants of ours have no civil engineers or architects among them. They cannot even guess as to how these magnificent, glassy structures are made. Yet, every day they walk into these buildings. They climb "stairs" and use "elevators" to get to work. To this particular generation of human beings, "buildings" have always been there, and always will be. There's no need to understand the structure, or technical nature behind them.

One day, several of these buildings collapse--the death toll is in the thousands. In NY News, it's simply stated: 'God struck again!' or something to that effect. The ignorance of this generation is so deep and entrenched, that the notion of buildings collapsing is attributable to God.

Now, compare this analogy to the modern day understanding of OpenAI, algorithms, and potentially the atrophy of critical thinking. I know few people these days who aren't using ChatGPT and Midjourney in some small way. The more conservative ones will only use it for menial, automated tasks. But most (I suspect) are using it for almost everything, ousting their brain for a Machine.

What will the long-term effects on critical thinking be? My analogy argues that AI-assisted existence (AAIE?) will eventually give technical minds like Sam Altman a monopoly on knowledge; a technocracy on a scale we've never seen in human history. Mindless acceptance of spammed prompts that are built on hallucinations, and a Jenga tower of assumptions (apparently made for practical purposes?) is becoming an increasingly likely future.

I claim that very few people actually understand what they are using and what it effects it has on their mind. When our children inherit far more advanced iterations of ChatGPT, the 'buildings falling out of the sky' for reasons they don't understand will be mechanisms of control for technocrats.

Perhaps I'm being paranoid. But I do stand firmly on the idea that there will be 'invisible enemies' that manifest in strange ways in the future. So strange in fact that we won't be able to identify that it came from AI at first. Worst case scenario we'll be so pacified and neglectful of our intellectual faculties that we won't identify the source at all.

We need to simplify how we explain artificial intelligence. We need more...
Discord space for people with FTX clawbacks/claims request
by kotrfa — 2024-02-16T09:04:31.199Z
Link: https://forum.effectivealtruism.org/posts/99KphNoXj2bQBadBa/discord-space-for-people-with-ftx-clawbacks-claims-request

tl:dr; there is a support group for those dealing with FTX legal proceedings on https://discord.gg/X67NztVg

A while back, I asked this question looking for anyone navigating the FTX "clawbacks" and related matters. A few folks got in touch, but it's become clear there's no dedicated space for open discussion about FTX, where people can exchange experiences and advice. While I'm not a legal expert or directly involved in these proceedings, I see the value in having a place for affected individuals to coordinate, share their costs, experiences and strategies.

As the attorneys handling the FTX bankruptcy are now beginning to process smaller claims, including individual grants (leading to an increase in messages to me), I've decided to take the initiative and kick-start some default coordination. I've set up a Discord server for this purpose.

If you know anyone who might benefit from joining this community, please spread the word. They can contact me for more information or directly join the Discord server using this link: https://discord.gg/X67NztVg (note there's a simple verification process upon joining).

It's an experiment. I am not sure it will be useful or will survive, but I think it's worth the shot!
OpenAI's Sora is an agent
by Caleb Biddulph — 2024-02-16T07:35:52.171Z

If you haven't already, take a look at Sora, OpenAI's new text-to-video AI. Sora can create scarily-realistic videos of nearly any subject. Unlike previous state-of-the-art AIs, the videos are coherent across time scales as long as one minute, and they can be much more complex.

Looking through OpenAI's research report, this one section caught my attention:

For a moment, I was confused: "what does it mean, Sora can 'control the player in Minecraft with a basic policy?' It's generating footage of a video game, not actually playing it... right?"

It's true that in these particular demo videos, Sora is "controlling the player" in its own internal model, rather than interfacing with Minecraft itself. However, I believe OpenAI is hinting that Sora can open the door to a much broader set of applications than just generating video.

In this post, I'll sketch an outline of how Sora could be used as an agent that plays any video game. With a bit of "visual prompt engineering," I believe this would even be possible with zero modifications to the base model. You could easily improve the model's efficiency and reliability by fine-tuning it and adding extra types of tokens, but I'll refrain from writing about that here.

The capabilities I'm predicting here aren't totally novel - OpenAI itself actually trained an AI to do tasks in Minecraft, very similarly to what I'll describe here.

What interests me is that Sora will likely be able to do many general tasks without much or any specialized training. In much the same way that GPT-3 learned all kinds of unexpected emergent capabilities just by learning to "predict the next token," Sora's ability to accurately "predict the next frame" could let it perform many visual tasks that depend on long-term reasoning.

Sorry if this reads like an "advancing capabilities" kind of post. Based on some of the wording throughout their research report, I believe OpenAI is already well aware of this, and it would be better for people to understand the implications of Sora sooner rather than later.

How to play any video game by predicting the next frame

Recall from the OpenAI report that Sora can take any video clip as input and predict how it will continue. To start it off, let's give it a one-second clip from the real Minecraft video game, showing the player character shuffling around a bit. At the bottom of that video, we'll add a virtual keyboard and ...