trentmkelly/LessWrong-43k huggingface dataset:
Introducing The Logical Foundation, an EA-Aligned Nonprofit with a Plan to End Poverty With Guaranteed Income
by Michael Simm — 2022-11-18T08:13:17.926Z

Hi EA Community, I’d like to introduce my high-impact startup and spark a conversation. I’m also seeking potential co-founders or funders to help get our idea off the ground. I introduced myself last week.

Summary - The Logical Foundation is a 501(c)(3) nonprofit startup founded in 2022. Its mission is to ensure a prosperous future for humanity by unleashing the massive anti-poverty potential of guaranteed income. This post uses an ITN (importance, tractability, neglectedness) framework to analyze our intervention, strategy, and cause area.
This post gives my understanding of Guaranteed Income’s potential to cost-effectively reduce poverty and homelessness at scale in developed nations, and why our Maximum Impact Pilot focuses on homelessness in Arizona. It’s based on my research on guaranteed income, homelessness, technological disruption, and some ideas from Doing Good Better.

I will discuss

The massive anti-poverty potential of guaranteed income in developed nations, and how we’re stuck in Pilotland.
Our Big Idea: Building a platform to unleash the massive anti-poverty potential of guaranteed income.
Our Theory of Change. How can we scale to the levels needed to meaningfully reduce poverty and end homelessness?
ITN Framework: Our grand plan to escape Pilotland and reduce poverty with guaranteed income.
The implications of guaranteed income for humanity and the Effective Altruism Movement.
Longtermist implications of guaranteed income.

Our Startup Plan: Maximum Impact in Developed Nations.

Experimental results support the cost-effectiveness of guaranteed income against homelessness.
Arizona is by far the deadliest place in the U.S. for homeless people.
QALYs (Quality-Adjusted Life Years) expected from Foundation's Maximum Impact Pilot

My recommendation? Effective Altruists should strongly consider helping us grow.

What help do we need to (hopefully) make it happen, and how EA people and organizations can increase our chance of success?

The massive anti-poverty potential of guaranteed income in developed nations, and how we’re stuck in Pilotland.

Guaranteed income

is a type of cash transfer program that provides continuous unconditional cash transfers to individuals or households.

Cash transfers are

the most widely researched intervention in the world

, and also the

most consistently pos

...
My Deontology Says Narrow-Mindedness is Always Wrong
by LVSN — 2022-11-18T06:11:25.115Z

The SBF controversy has not shifted my belief that you should always be sensitive to the relevant arguments pertaining to the specific situation, even when you think you've found good general rules, though of course those general rules may count as relevant arguments.

I don't even care about "utility." You know what my deontology says? Narrow-mindedness is always wrong.

I know you think you've found a rule that's important to follow absolutely and without regard for the reality of what the consequences will be, for the greater good. You haven't.

Having general rules that you consider when they are relevant is great. Having rules that were formed under circumstances wildly more lenient than what reality is going to trap you with, and pretending that reality will continue to be lenient in spite of the evidence so that you can keep following your rules, is stupid.

You solemnly argue that the human brain is too corrupt to sanely run utilitarianism; we must observe the consequences of that? Sure, just permit me to solemnly argue that the human brain is too corrupt to sanely run dogmatism, and we must observe the consequences of that too.
AI Ethics != Ai Safety
by Dentin — 2022-11-18T03:02:15.707Z

[shortform - understanding why we think certain things is hard, and if writing it down helps me, maybe posting it will help someone else too]

I've been hearing a lot about "AI ethics" from various people and companies in the tech industry for a while now. This has always annoyed me for some ill-defined reason; I think I know why now. Allow me an analogy:

AI Safety is kinda like the FBI. It needs to be big, comprehensive, have many arms, and capable of addressing real threats that can cause significant damage to things that matter. A lot of these threats are hypothetical or abstract, and require foresight, prediction, and planning to contain.

AI Ethics is kinda like a mall cop. It needs to be visible, help shoppers feel good, and discourage the occasional malcontent from making a fuss.

So far, so good; we need both of these, or at least both have a place.

The problem is that AI Ethics is the Hot New Thing, and everyone has Jumped On The BandWagon, and if you don't support AI Ethics there's a social cost to pay. Meanwhile, AI Safety is dull, boring, and "look I don't know why we even have those guys, it's not like anyone is going to fly a 747 into a skyscraper any time soon".

As a direct result of this, the tech industry appears to be allocating the FBI's budget to the mall cop, and the mall cop's budget to the FBI.

That's what has been bothering me about AI Ethics.
Don't design agents which exploit adversarial inputs
by TurnTrout — 2022-11-18T01:48:38.372Z

Summary. Consider two common alignment design patterns:

Optimizing for the output of a grader which evaluates plans, and
Fixing a utility function and then argmaxing over all possible plans.

These design patterns incentivize the agent to find adversarial inputs to the grader (e.g. "manipulate the simulated human grader into returning a high evaluation for this plan"). I'm pretty sure we won't find adversarially robust grading rules. Therefore, I think these alignment design patterns are doomed.

In this first essay, I explore the adversarial robustness obstacle. In the next essay, I'll point out how this is obstacle is an artifact of these design patterns, and not any intrinsic difficulty of alignment. Thanks to Erik Jenner, Johannes Treutlein, Quintin Pope, Charles Foster, Andrew Critch, randomwalks, and Ulisse Mini for feedback.

1: Optimizing for the output of a grader

One motif in some AI alignment proposals is:

An actor which proposes plans, and
A grader which evaluates them.

For simplicity, imagine we want the AI to find a plan where it makes an enormous number of diamonds. We train an actor to propose plans which the grading procedure predicts lead to lots of diamonds.

In this setting, here's one way of slicing up the problem:

Outer alignment: Find a sufficiently good grader.

Inner alignment: Train the actor to propose plans which the grader rates as highly possible (ideally argmaxing on grader output, but possibly just intent alignment with high grader output).[1]

This "grader optimization" paradigm ordains that the AI find plans which make the grader output good evaluations. An inner-aligned actor is singlemindedly motivated to find plans which are graded maximally well by the grader. Therefore, for any goal by which the grader may grade, an inner-aligned actor is positively searching for adversarial inputs which fool the grader into spitting out a high number!

In the diamond case, if the actor is inner-aligned to the grading procedure, then the actor isn't actually aligned towards diamond-production. The actor is aligned towards diamond-production as quoted via the grader's evaluations. In the end, the actor is aligned to the evaluations.

I think that there aren't clever ways around this issue. Under this motif, under this way of building an AI, you're not actually building an AI which cares about diamonds, and so you won't get a...
Engineering Monosemanticity in Toy Models

by Adam Jermyn — 2022-11-18T01:43:38.623Z
Link: https://arxiv.org/abs/2211.09169

Overview

In some neural networks, individual neurons correspond to natural "features" in the input. Such monosemantic neurons are much easier to interpret, because in a sense they only do one thing. By contrast, some neurons are polysemantic, meaning that they fire in response to multiple unrelated features in the input. Polysemantic neurons are much harder to characterize because they can serve multiple distinct functions in a network.

Recently, Elhage+22 and Scherlis+22 demonstrated that architectural choices can affect monosemanticity, raising the prospect that we might be able to engineer models to be more monosemantic. In this work we report preliminary attempts to engineer monosemanticity in toy models.

Toy Model

The simplest architecture that we could study is a one-layer model. However, a core question we wanted to answer is: how does the number of neurons (nonlinear units) affect the degree of monosemanticity? To that end, we use a two-layer architecture:

Features are generated as sparse vectors in a high-dimensional space. They are then run through a (fixed) random projection layer to produce the inputs into our model. We imagine this random projection process as an analogy to the way the world encodes features in our observations.

Within the model, the first layer is a linear transformation with a bias, followed by a nonlinearity. The second layer is a linear transformation with no bias.

Our toy model is most similar to that of Elhage+22, with a key difference being that the extra linear layer allows us to vary the number of neurons independent of the number of features or the input dimension.

We study this two model on three tasks. The first, a feature decoder, performs a compressed sensing reconstruction of features that were randomly and lossily projected into a low-dimensional space. The second, a random re-projector, reconstructs one fixed random projection of features from a different fixed random projection. The third, an absolute value calculator, performs the same compressed sensing task and then returns the absolute values of the recovered features. These tasks have the important property that we know which features are naturally useful, and so can easily measure the extent to which neurons are monosemantic or polysemantic.

Note that we primarily study the regime where there are more features than embedding dimensions (i.e. the sparse feature lay...
AGIs may value intrinsic rewards more than extrinsic ones
by catubc — 2022-11-17T21:49:30.719Z

TLDR: Although AI agent paradigms use explicit reward approaches, the psychology of human motivation suggests that humans value internally generated reward as much if not more than external reward. I suggest that AIs that begin to exhibit behaviors that appear to be "internally" rewarded may reflect signs of AGI. But that might require us to create AIs that are no longer pure tools for solving individual tasks, but more wholistic agents that exist in the world.

Introduction

AI research focuses on training artificial agents to be motivated by externally provided or generated rewards. The greater the reward, the less the error and therefore less to be learned. But human behavior is motivated by many different factors with extrinsic rewards (e.g. getting food, money etc) being just one component. Here I argue that AI agents that may develop preferences for internal rewards may be more akin to humans and thus closer to AGI.

Behaviorism is a theory of learning externally rewarded behaviors - not a complete theory of learning and behavior

Behaviorist principles were developed starting in the early 20th century (Watson 1913) but are most associated with Skinner (see Skinner 1953). Briefly, behaviorists abandoned previously popular tools for explaining behavior such instincts, drives, needs and the black boxes of psychological states and argued that only variables that could be observed directly and measured quantitatively were useful. A central tenet of the theory is that all behavior can be explained as a function of prior environmental stimuli; the environment makes the animal do it. There was no need for internal or hidden states.

Although very successful in explaining previously puzzling phenomena, behaviorism faced challenges. In particular it required that animals operate "spontaneously" on the environment and that animals could not engage in any active "seeking". This was a difficult sell; i.e. how would any animal behavior be initiated in the first place? The best explanation was that over time, and with sufficient reinforcement of reward-able actions - animals would begin to "emit" behaviors that were externally reinforced in the past. A very mechanical explanation of complex biological organisms.

Cognitive theories of motivation account for curiosity and latent learning.

Meanwhile, numerous experiments carried out in the middle of the 20th cent...
LLMs may capture key components of human agency
by catubc — 2022-11-17T20:14:33.450Z

TLDR: LLM confabulation may be a significant feature - not flaw - of how human memory works and how humans experience agency in the world. Namely through narratives that are largely self-consistent - but not guaranteed to match facts in the world.

Introduction

The recent explosion of Large Language Models (LLMs) models such as GPT2 and GPT3 has shown that language models can achieve state-of-the-art results on a variety of natural language processing tasks, while also providing insights into how humans construct meaning. One of the most interesting results is that LLMs often engage in "confabulation". Confabulation is a well known phenomenon (going back to Freud) and refers to the situations where people create false memories, or false explanations without the intent to deceive (also known as "honest lying" Moscovitch 1989).

There are several lines of criticisms on why GPT3 models are not as useful nor as good of models of human language as it is claimed (or even true artificial general intelligence (AGI)). Here I argue that GPT3 confabulation may not be a weakness but a feature. In particular, I argue that confabulation may be a feature of memory and the way the human mind works, and that even our sense of agency may be founded on this capacity to make up stories that are self-consistent.

Maybe human memory defaults to confabulation when information is missing or a self-consistent story needs to be told. Confabulation is often studied in subjects with memory disorders, dementia or psychiatric disorders and often the "vividness" with which subjects describe confabulates stories can be remarkable. Such stories can be internally consistent with one another and often contain details that seem impossible to know if they came from personal experience.

But confabulation is an extremely broad phenomenon occurring in both pathological and non-pathological conditions. And it may not be a flaw of reasoning or a way to escape social embarrassment of not knowing, it might be a fundamental way that human memory and language work. That is, memory is often imperfect—we forget things and most of us don't have photographic recollections - and our minds fill in the gaps with information from other sources. There is evidence showing that normal adults can also engage in confabulation. A study by Moscovitch (1992) showed that when asked questions about t...
Mastodon Replies as Comments
by jefftk — 2022-11-17T20:10:01.237Z

The comment section on most blogs is pretty minimal, with the real
discussion happening elsewhere, but people who come to the post later
won't see that discussion. One of the more unusual choices I've made
with this blog is that instead of hosting comments here, I pull in and
display comments people make on social media. While this started out
as laziness (who wants to handle accounts for users?) over the years
I've come around to thinking that this is how blog comments should
normally work.

The biggest problem with this approach, though, is that it's in
tension with the goals of the social media companies. Facebook,
Twitter, etc want to keep you on their platform, and aren't especially
interested in serving as the comment section of external blogs. On the
other hand, this is potentially a really good fit for federated social
media, and I've now made it so that ActivityPub replies to my blog
posts will show up as comments here:

For a live example, here's a post
from last week.

Integration was fast: if I fetch
https://[server]/api/v1/statuses/[id]/context" I get all
the publicly visible replies to that status that this server knows
about. Then it's just a matter of extracting the relevant
information, threading the responses, and fitting that into my
existing comment display infrastructure (code).

This is a bit of a hack on top of Mastodon, but for blogs to
participate directly in ActivityPub, where you could subscribe and
comment over the protocol, you wouldn't have to implement most of what
Mastodon does. That would make it much smaller, more efficient, and
more maintainable. It looks like maybe someone has made a WordPress plugin
for this? Has anyone tried it?

Comment via: mastodon
Announcing the Progress Forum
by jasoncrawford — 2022-11-17T19:26:29.584Z

I’d like to invite you to join the Progress Forum, the new online home for the progress community. It's a clone of this site, but with a focus on progress studies and the philosophy of progress.

This forum was pre-announced in January 2022, and quietly opened in April. Although anyone could sign up, we deliberately didn’t make any big announcement about it, aiming first for a small, high-quality community. Now that we have a lot of good content on the site, we’re announcing it more broadly.

The primary goal of this forum is to provide a place for long-form discussion of progress studies. It’s also, like LW, a place to find local clubs and meetups.

The broader goal is to share ideas, strengthen them through discussion and comment, and over the long term, to build up a body of thought that constitutes a new philosophy of progress for the 21st century (and beyond).

I invite you to post:

Essays (original, or cross-posted from your blog)
Drafts, half-baked ideas, and work-in-progress thinking, for feedback
Questions for brainstorming
Local events and community groups
Etc.

And please read and comment on what others have shared.

You can subscribe to Forum posts via email, RSS, or Twitter.

The Forum is sponsored by The Roots of Progress. Huge thanks to the people who worked to create and run it: Lawrence Kestleoot, Andrew Roberts, Sameer Ismail, David Smehlik, Alec Wilson, and Ross Graham. Thanks also to Kris Gulati for nudging this project along, and to Ruth Grace Wong for helpful conversations about community and moderation. Finally, thanks to the LessWrong team for creating this software platform, and especially to Oliver Habryka, Ruby Bloom, Raymond Arnold, JP Addison, James Babcock, and Ben Pace for answering questions and helping us customize this instance of it.

Go check it out.
What kind of bias is this?
by Daniel Samuel — 2022-11-17T18:44:21.495Z

What is a kind of bias in which nice things done by a person who is for example always lovely, have less value to us than nice things done by someone who is usually unpleasant? The reverse can also be true, of course.

If this is a specific type of bias do you know of quality research papers describing this phenomenon or this particular situation?
AI Forecasting Research Ideas
by Jsevillamol — 2022-11-17T17:37:40.815Z
Link: https://docs.google.com/document/d/1Vva6AW_udvfavtldWuGbqf8NbbzsgEH8mESEX6NlOvM

Overview

The linked document contains a collection of AI Forecasting research ideas, prepared by some

Epoch

employees in a personal capacity
We think that these are interesting and valuable projects that research interns or students could look into, though they may vary in difficulty (depending on your background/experience)
This is the result of a quick brainstorming and curation, rather than a thorough deliberative process. We encourage a critical outlook when reading them
You may also be interested in these other

forecasting research ideas

suggested by Jaime Sevilla
You can use

Epoch’s database

as a resource for finding notable machine learning papers with parameter, compute, and dataset sizes. On

Epoch website

you can also find

our past research

,

a tool for visualising the dataset

, and some other tools like

this compute calculator

.
Please feel free to

contact us

for clarification about these questions!
Results from the interpretability hackathon
by Esben Kran — 2022-11-17T14:51:44.568Z
Link: https://alignmentjam.com/post/results-from-the-interpretability-hackathon

We ran a

mechanistic interpretability

hackathon

(

original post

) with

25 projects submitted

by ~70 participants. Here we share the winning projects but many of the others were also incredibly interesting. In summary:

An algorithm to automatically make the activations of a neuron in a Transformer much more interpretable.
Backup name mover heads from “Interpretability in the Wild” have backup heads and all of these are robust to the ablation distribution.
The specificity benchmark in the ROME and MEMIT memory editing papers does not represent specificity well. A simple modulation shows that factual association editing bleeds into related texts, representing "loud facts".
TCAV used on an RL agent for a connect four game can have its neural activation compared to the provably best solution as a pilot for comparing learned activations more generally to human-made solutions.

Thank you to Sabrina Zaki, Fazl Barez, Thomas Steinthal, Joe Hardie, Erin Robertson, Richard Annilo, Itay Yona, other jam site organizers and all the participants for making it all possible.

Investigating Neuron Behaviour via Dataset Example Pruning and Local Search

By Alex Foote

Abstract: This report presents methods for pruning and diversifying dataset examples that strongly activate neurons in a language model, to facilitate research into understanding the behaviour of these neurons. The pruning algorithm takes a dataset example that strongly activates a specific neuron and extracts the core sentence before iteratively removing words, to find the shortest substring that preserves a similar pattern and magnitude of neuron activation.

This removes extraneous information, providing a much more concise input that is easier to reason about. The extracted substring, referred to as a Minimal Activating Example (MAE), is then used as a seed for local search in the input space. Using BERT, each word in the MAE is replaced by its most probable substitutes, and neuron activation is re-assessed. This creates positive and negative inputs that shed much more light on neuron behaviour than dataset examples alone.

In two case studies we identify neuron behaviours that were not obvious from the raw dataset examples using this combination of pruning and local search. These methods could facilitate and significantly speed up research into neuron behaviour in language models, which is a key asp...