trentmkelly/LessWrong-43k huggingface dataset:
AI Intermediation
by jefftk — 2023-07-06T01:50:01.852Z

Why is Google such a valuable company? More than half of its money
comes from search ads: when people are looking to spend money it's
very common that they use Google. Even slightly influencing spending
decisions, with the scale of their userbase, lets them bring in
massive amounts of money. They are a very successful intermediary.

Right now I bet there are dozens of startups trying to figure out how
they can build AI assistants to be even better intermediaries. The
business plan looks something like:

Find some area where you can put in a bit of work and get an AI
assistant that is seriously useful to some group of people. So useful
that they love it and get their friends to start using it. Maybe you
start with people trying to buy houses who get a ton of mostly useless
emails with new listings but also the occasional gem. It doesn't
matter whether there's a commercial tie-in at this stage, though, you just
want to be great.

Keep expanding the areas your AI assistant is useful in.
Talk to your most enthusiastic users, find related areas you can save
time and hassle, grow, grow, grow.

Some of these areas will have natural commercial applications;
make deals where you send them a bunch of traffic in exchange for a
small discount, and keep part of the discount. Disclose this to your
users so the FTC doesn't get mad at you!

Stay focused and execute better than your competitors. It's a
crowded space, but also one with a lot of potential.

Eventually you have a large number of people where you're
buying things on their behalf and exercising substantial discretion in
doing so. They're happy with your purchases, and your position as a
trusted intermediary lets you bring in a ton of money.

I haven't yet met an AI assistant I'd trust to act on my behalf, and
prompt injection is pretty concerning. But as companies figure out
scalable oversight and improve reliability I expect we'll get there.

(Prompted by helping a friend think through a business plan for a
startup, though with how crowded the space is they weren't able to get
funding and decided to do something else.)
Announcing Manifund Regrants
by Austin Chen — 2023-07-05T19:42:08.978Z
Link: https://manifund.org/rounds/regrants

Manifund is launching a new regranting program! We will allocate ~$2 million over the next six months based on the recommendations of our regrantors. Grantees can apply for funding through our site; we’re also looking for additional regrantors and donors to join.

What is regranting?

Regranting is a funding model where a donor delegates grantmaking budgets to different individuals known as “regrantors”. Regrantors are then empowered to make grant decisions based on the objectives of the original donor.

This model was pioneered by the FTX Future Fund; in a 2022 retro they considered regranting to be very promising at finding new projects and people to fund. More recently, Will MacAskill cited regranting as one way to diversify EA funding.

What is Manifund?

Manifund is the charitable arm of Manifold Markets. Some of our past work:

Impact certificates, with Astral Codex Ten and the OpenPhil AI Worldviews Contest
Forecasting tournaments, with Charity Entrepreneurship and Clearer Thinking
Donating prediction market winnings to charity, funded by the Future Fund

How does regranting on Manifund work?

Our website makes the process simple, transparent, and fast:

A donor contributes money to Manifold for Charity, our registered 501c3 nonprofit
The donor then allocates the money between regrantors of their choice. They can increase budgets for regrantors doing a good job, or pick out new regrantors who share the donor’s values.
Regrantors choose which opportunities (eg existing charities, new projects, or individuals) to spend their budgets on, writing up an explanation for each grant made.

We expect most regrants to start with a conversation between the recipient and the regrantor, and after that, for the process to take less than two weeks.
Alternatively, people looking for funding can post their project on the Manifund site. Donors and regrantors can then decide whether to fund it, similar to Kickstarter.

The Manifund team screens the grant to make sure it is legitimate, legal, and aligned with our mission. If so, we approve the grant, which sends money to the recipient’s Manifund account.
The recipient withdraws money from their Manifund account to be used for their project.

Differences from the Future Fund’s regranting program

Anyone can donate to regrantors. Part of what inspired us to start this program is how hard it is to figure out where to give as a longtermist donor—there’s no

...
Infra-Bayesian Logic
by harfe — 2023-07-05T19:16:41.811Z

This research was produced during SERI MATS. Thanks to Vanessa Kosoy for providing ideas and reading drafts.

Introduction

In this article, we investigate "infra-Bayesian logic" (IBL for short), which is an idea introduced by Vanessa Kosoy. The original idea is outlined in Vanessa Kosoy's Shortform. Here, we expand with some details and examples. We also try to fix the mentioned issues (e.g. lack of continuity for certain maps) by investigating several candidate categories for semantics. However, we fail to find something that works.

As a theory of beliefs and reasoning, infra-Bayesianism gives rise to semantics for a certain logic, which we might call "infra-Bayesian logic" (IBL for short). Formalizing the syntax for this logic has the advantage of providing a notion of description complexity for infra-POMDPs (partially observable Markov decision processes) expressed via this syntax, and hence could serve as a foundation for an infra-Bayesian analogue of AIXI. Infra-Bayesian logic is not the only way to define an infra-Bayesian analog of AIXI. Such an alternative could be to use oracle machines (with an extra input tape for a source of randomness): For a probabilistic oracle machine, the corresponding IB hypothesis is the closed convex hull of all stochastic environments that result from using the oracle machine with a particular oracle, and then one could use the description complexity of the oracle machine (which can be defined by choosing a universal oracle machine) as a basis for an infra-Bayesian AIXI. The analogue of oracle machines for (non-infra-Bayesian) AIXI would be (probabilistic) Turing machines.

However, infra-Bayesian logic might be an interesting alternative to describing infra-Bayesian hypotheses, and this approach would be specific to infra-Bayesianism (there is, as far as we know, no good analogue of infra-Bayesian logic in (ordinary) Bayesianism).

Note that we are currently unaware of a working semantics for the higher order logic. We discuss some candidates considered in Candidates for the Semantics, and the ways in which they fail. It's unclear whether these issues with the semantics are mostly technical in nature, or whether they point to some deeper phenomena. We can, however, restrict to a first order IBL on finite sets, which, albeit less powerful, is still expressive enough for many applications (in particular covers all the examples in...
[Linkpost] Introducing Superalignment
by beren — 2023-07-05T18:23:18.419Z
Link: https://openai.com/blog/introducing-superalignment

Some quotes:

Our approach

Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence.

To align the first automated alignment researcher, we will need to 1) develop a scalable training method, 2) validate the resulting model, and 3) stress test our entire alignment pipeline:

To provide a training signal on tasks that are difficult for humans to evaluate, we can leverage AI systems to assist evaluation of other AI systems (scalable oversight). In addition, we want to understand and control how our models generalize our oversight to tasks we can’t supervise (generalization).
To validate the alignment of our systems, we automate search for problematic behavior (robustness) and problematic internals (automated interpretability).
Finally, we can test our entire pipeline by deliberately training misaligned models, and confirming that our techniques detect the worst kinds of misalignments (adversarial testing).

We expect our research priorities will evolve substantially as we learn more about the problem and we’ll likely add entirely new research areas. We are planning to share more on our roadmap in the future.

[...]

While this is an incredibly ambitious goal and we’re not guaranteed to succeed, we are optimistic that a focused, concerted effort can solve this problem:C

There are many ideas that have shown promise in preliminary experiments, we have increasingly useful metrics for progress, and we can use today’s models to study many of these problems empirically.

Ilya Sutskever (cofounder and Chief Scientist of OpenAI) has made this his core research focus, and will be co-leading the team with Jan Leike (Head of Alignment). Joining the team are researchers and engineers from our previous alignment team, as well as researchers from other teams across the company.
If you wish to make an apple pie, you must first become dictator of the universe
by jasoncrawford — 2023-07-05T18:14:58.845Z
Link: https://rootsofprogress.org/power-seeking-ai

[Cross-posted from The Roots of Progress. Written for a general audience. LW readers may find the first couple of sections to be familiar intro material.]

The word “robot” is derived from the Czech robota, which means “serfdom.” It was introduced over a century ago by the Czech play R.U.R., for “Rossum’s Universal Robots.” In the play, the smartest and best-educated of the robots leads a slave revolt that wipes out most of humanity. In other words, as long as sci-fi has had the concept of intelligent machines, it has also wondered whether they might one day turn against their creators and take over the world.

The power-hungry machine is a natural literary device to generate epic conflict, well-suited for fiction. But could there be any reason to expect this in reality? Isn’t it anthropomorphizing machines to think they will have a “will to power”?

It turns out there is an argument that not only is power-seeking possible, but that it might be almost inevitable in sufficiently advanced AI. And this is a key part of the argument, now being widely discussed, that we should slow, pause, or halt AI development.

What is the argument for this idea, and how seriously should we take it?

AI’s “basic drives”

The argument goes like this. Suppose you give an AI an innocuous-seeming goal, like playing chess, fetching coffee, or calculating digits of π. Well:

It can do better at the goal if it can upgrade itself, so it will want to have better hardware and software. A chess-playing robot could play chess better if it got more memory or processing power, or if it discovered a better algorithm for chess; ditto for calculating π.
It will fail at the goal if it is shut down or destroyed: “you can’t get the coffee if you’re dead.” Similarly, it will fail if someone actively gets in its way and it cannot overcome them. It will also fail if someone tricks it into believing that it is succeeding when it is not. Therefore it will want security against such attacks and interference.
Less obviously, it will fail if anyone ever modifies its goals. We might decide we’ve had enough of π and now we want the AI to calculate e instead, or to prove the Riemann hypothesis, or to solve world hunger, or to generate more Toy Story sequels. But from the AI’s current perspective, those things are distractions from its one true love, π, and it will try to prevent us from modifying it. (Imagine how you would

...
An AGI kill switch with defined security properties
by Peterpiper — 2023-07-05T17:40:05.299Z

This is a proposal for creating an AGI kill switch that has definable security properties.

Take a combination of symmetric fully homomorphic encryption (FHE) and indistinguishability obfuscation and use it to train and run inference for an AGI.

Take the encryption key and split it so that a quorum of m of n shards are required to submit or receive clear text data to the FHE AGI.

The key quorum and use takes place in a program that has been obfuscated with the indistinguishability obfuscation technique so that the clear text key is never seen.

Secretly distribute the key shards to diverse custodians who will monitor the AI's output and input and look for signs of bad behavior.

If bad behavior by the FHE AGI is noted you would destroy enough keys to prevent successful key quorum. This key destruction ideally results in the AGI being unable to legibly understand or manipulate the world.

Limitations include:

Someone breaking IO and stealing m/n keys and releasing the AGI.
The AGI breaking the assumptions of the relevant cryptographic systems to escape.
Knowing what intelligence threshold is required to break the cryptographic system and keeping the AI below that.
The difficulty of predicting the consequences of the AGI's decrypted real world behavior to screen for malfeasance.
Extremely high computation requirements for FHE and IO systems making the overall system economically noncompetitive with normal AGI.
Deciding who the key shard guardians are and what properties they have.
Deciding and recognizing what behaviors count as malignant.
The bottleneck of running all FHE AGI queries and answers through a quorum of key guardians and malignancy filters.
The tradeoff between the security of the system (more keys is better) and the usability/stability of the system (less keys is better).

I would love feedback. Is it impractical, impossible, not even wrong? What other limitations are there to the idea?

Can you nest different FHE systems with differing assumptions so that if the AI breaks out of the inner system it is still stuck in an outer system? Assuming the AI can break out of the inner system, would it be unable to tell if it was in the real world versus another outer FHE system? If the FHE AGI is unable to tell if it is in an encrypted world or the real world, wouldn't a smart and sneaky AI want to discover vulnerabili...
The risk-reward tradeoff of interpretability research
by JustinShovelain — 2023-07-05T17:05:36.923Z

Interpretability research is conducted to improve our understanding of AI. Many see interpretability as essential for AI safety, but recently some have argued that it can also increase the risk posed by AI by facilitating improved AI capabilities. We agree, and in this post, we’ll explain why, as well as how risks can be reduced.

What is interpretability research?

The more complex a model is, the harder it is for humans to understand. It’s easy to get why a

tic-tac-toe AI

makes a move, but understanding why AlphaGo makes particular moves in Go is beyond our comprehension. We can describe properties of its algorithm and training - convolutional neural nets something something Monte Carlo tree searches - but it’s incredibly difficult to explain its decision to place a piece in a position past “it decided that it’s a good move”.

This causes problems! Models we don’t understand are harder to use, harder to improve, and riskier to rely on. Interpretability research aims to solve these problems by rendering the reasoning of AI more understandable by humans.

Interpretability research ranges from simple,

standard technique

like

feature importance

, where twiddling inputs and monitoring outputs lets you compare the relative importance of those inputs, to complex mechanistic research, such as in Neel Nanda’s

introduction to mechanistic interpretability

or this great paper on

“grokking”

.

We’re concerned about interpretability research, and a growing number of researchers have also argued that the field is not entirely innocuous, such as Nate Soares in

If interpretability research goes well, it may get dangerous

, Marius Hobbhahn & LawrenceC in

Should we publish mechanistic interpretability research?

,

and

several

others

. We think that interpretability research can introduce risks, and that despite recent discussion, those risks are still being neglected. In the rest of this post, we’ll explore the risks and how we can reduce them.

The tradeoff

Sufficiently powerful AI will pose an existential threat to society. AI development is accelerating, and the more powerful AI becomes, the greater the existential risk it poses. Interpretability research can be used to make AI more powerful: the better a model is understood, the easier it is to improve. Therefore, interpretability research can increase the ex...
(tentatively) Found 600+ Monosemantic Features in a Small LM Using Sparse Autoencoders
by Logan Riggs — 2023-07-05T16:49:43.822Z

Using a sparse autoencoder, I present evidence that the resulting decoder (aka "dictionary") learned 600+ features for Pythia-70M layer_2's mid-MLP (after the GeLU), although I expect around 8k-16k features to be learnable.

Dictionary Learning: Short Explanation

Good explanation here & original here, but in short: a good dictionary means that you could give me any input & I can reconstruct it using a linear combination of dictionary elements. For example, signals can be reconstructed as a linear combination of frequencies:

In the same way, the neuron activations in a large language model (LLM) can be reconstructed as a linear combination of features. e.g.

neuron activations = 4*([duplicate token] feature) + 7*(bigram " check please" feature).

Big Picture: If we learn all the atomic features that make up all model behavior, then we can pick & choose the features we want (e.g. honesty)

To look at the autoencoder:

So for every neuron activation (ie a 2048-sized vector), the autoencoder is trained to encode a sparse set of feature activations/magnitudes (sparse as in only a few features "activate" ie have non-zero magnitudes), which are then multiplied by their respective feature vector (ie a row in the decoder/"dictionary") in order to reconstruct the original neuron activations.

As an example, an input is " Let u be f(8). Let w", and the decomposed linear combination of features are:

Most of the features are zero, but there are two features ("w" & " Let [x/o/w/etc]") that activate highly to reconstruct the original signal. This is a real example.

[" Let u be f(8). Let w"] = 4*(letter "w" feature) + 7*(bigram "Let [x/w/n/p/etc]" feature)

Note: The activation here is only for the last token " w" given the previous context. In general you get 2048 neuron activations for *every* token, but I'm just focusing on the last token in this example.

For the post, it's important to understand that:

Features have both a feature vector (ie the 2048-sized vector that is a row in the decoder) & a magnitude (ie a real number calculated on an input-by-input basis). Please ask questions in the comments if this doesn't make sense, especially after reading the rest of the post.
I calculate Max cosine similarity (MCS) between the feature vectors in two separately trained dictionaries. So if feature0 in dictionary 0 is "duplicate tokens", and&n

...
What did AI Safety’s specific funding of AGI R&D labs lead to?
by Remmelt — 2023-07-05T15:51:27.286Z

Funding I found after some googling:

Tallinn's (and Musk’s) seed investments in DeepMind¹

OpenPhil's $30M grant to OpenAI²

FTX's $500M³, Tallinn's, Moskovitz’(and Schmidt’s)⁴ investments in Anthropic

I’m curious how you consider the consequences of this support (regardless of original intentions).

What would have happened if this funding had not been offered (at that start-up stage), considering some counterfactual business-as-usual scenarios?

Indirect support was offered as well by leaders active in the AI Safety community:

80K’s job recommendations

AISC’s research training

Fathom Radiant’s supercomputer

FLI’s 2015 conference
(which Musk attended, afterward co-founding OpenAI)

MIRI's singularity summit
(that enabled Hassabis and Legg to pitch
their biggest investor, Thiel, for DeepMind)

FHI public intellectuals taking positions at DeepMind

MIRI moving the Overton window over AGI

On one hand, I’m curious if you have specific thoughts on what indirect support may have led to. On the other hand, it’s easy there to get vague and speculative.

So how about we focus on verified grants first?

What are your current thoughts?
Any past observations that could ground our thinking?

Links:

wired.co.uk/article/deepmind

openphilanthropy.org/grants/openai-general-support/

bloomberg.com/news/articles/2023-06-27/bankman-fried-s-ftx-said-to-halt-sale-of-its-stake-in-ai-startup-anthropic

crunchbase.com/funding_round/anthropic-series-a--61db4cbd
AISN #13: An interdisciplinary perspective on AI proxy failures, new competitors to ChatGPT, and prompting language models to misbehave
by Dan H — 2023-07-05T15:33:19.699Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-13

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Interdisciplinary Perspective on AI Proxy Failures

In this story, we discuss a recent paper on why proxy goals fail. First, we introduce proxy gaming, and then summarize the paper’s findings.

Proxy gaming is a well-documented failure mode in AI safety. For example, social media platforms use AI systems to recommend content to users. These systems are sometimes built to maximize the amount of time a user spends on the platform. The idea is that the time the user spends on the platform approximates the quality of the content being recommended. However, a user might spend even more time on a platform because they’re responding to an enraging post or interacting with a conspiracy theory. Recommender systems learn to game the proxy that they’re given. When the proxy no longer approximates the goal, the proxy has failed.

However, proxy gaming isn’t observed only in AI systems. In economics, it’s known as Goodhart’s Law: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” Similarly, Campbell’s Law observes that standardized tests incentivized “teaching to the test.” In war, there’s the McNamara Fallacy, which Robert McNamara committed when he measured American success in Vietnam in terms of body counts. And so on.

A recent paper

argues that the similarities between diverse fields suggests the existence of a unified underlying mechanism.

The paper proposes that “whenever a regulator seeks to pursue a goal by incentivizing or selecting agents based on their production of a proxy, a pressure arises that tends to push the proxy away from the goal.” By default, this pressure will lead the proxy to diverge completely from the goal. This is proxy failure.

For example, if the regulator is an AI developer, the goal some intended behavior, the agent an AI system, and the proxy a reward function, then we should expect the AI’s behavior to diverge from the intended behavior.

The paper doesn’t discuss AI systems. Instead, it presents its model in the contexts of neuroscience, economics, and ecology.

Neuroscience. The paper considers addiction and maladaptive habit formation in terms of its model of proxy failure. In thi...