trentmkelly/LessWrong-43k huggingface dataset:
Nice Clothes are Good, Actually
by Gordon Seidoh Worley — 2023-01-31T19:22:06.430Z

This is just some casual thoughts about how to dress nice. I think dressing nice is a useful skill, but nobody told me in a way that convinced me. So this is written in the spirit of what I wish someone would have told 15 year old me.

For most of my life I thought dressing nicely was a type of bullshit signaling game I didn't need to play because I was doing Real Work. I'm here to tell you that I was mistaken and that dressing nicely is good and useful to your life.

My story went something like this:

I care about doing important, meaningful things in the world, like solving hard technical problems.
I can solve those problems regardless of what I wear.
Further, lots of the people who seem best at solving hard technical problems dress poorly.
Also, I don't really understand fashion that well, and most of the people who do understand it seem to be focused on things other than solving hard technical problems.
Conclusion: I should not care about dressing nicely. Just wear whatever is comfortable regardless of how it looks.

Dressing poorly or casually, though, is still a kind of signaling game. My reasoning makes this clear: it shows I'm part of the ingroup of serious people solving serious, real problems rather than part of the outgroup of silly people who care more about how they look than getting things done.

In some situations and in some contexts, this is a true, useful fact. If you show up to a coding interview in a suit, you've probably hurt your chances to getting the job. Even if your code is great, the person interviewing you is going to worry that you have expectations about the job that don't line up with reality. And, if we're honest, you don't look the part, so unconscious bias comes into play.

However, unconscious bias is the start of seeing why dressing nicely, or at least intentionally, matters. Another word for unconscious bias is vibes. What vibe do your clothes put off? That is, how do you present yourself to the world? What interfaces and affordances are you giving people based on the clothes you wear?

Some examples:

Wearing a pot-leaf hoodie? People will reasonably think you smoke weed and that smoking weed is important to you.
Wearing a three-piece suit? People will think you have money (because you had enough money at least to afford the suit) and are serious.
Wearing a poorly fitted suit? People will think you are a low-level employee of a faceless busi

...
[Linkpost] Human-narrated audio version of "Is Power-Seeking AI an Existential Risk?"
by Joe Carlsmith — 2023-01-31T19:21:48.907Z

There's now a human-narrated audio version of my report on existential risk from power-seeking AI (text version here). You can find the audio here, or by searching for "Joe Carlsmith Audio" on your podcast app.

Thanks to the folks at Type III audio, and especially to Solenoid Entity, for doing the narration. There's also a shorter video presentation and summary of the report here.
No Really, Attention is ALL You Need - Attention can do feedforward networks
by Robert_AIZI — 2023-01-31T18:48:20.801Z
Link: https://aizi.substack.com/p/no-really-attention-is-all-you-need?sd=pf

[Epistemic status: Mathematically proven, and I have

running code

that implements it.]

Overview: A

transformer

consists of two alternating sublayers: attention heads and feedforward networks (FFNs, also called

MLPs

). In this post I’ll show how you can implement the latter using the former, and how you can convert an existing transformer with FFNs into an attention-only transformer.

My hope is that such a conversion technique can augment mechanistic interpretability tools such as the ones described in

A Mathematical Framework for Transformer Circuits

, by reducing the task of interpretability from “interpret attention and FFNs” to just “interpret attention”. That publication specifically points out that “more complete understanding [of Transformers] will require progress on MLP layers”, which I hope this technique can supply.

Limitations:

Attention heads are able to naturally produce two activation functions,

SiLU

and ReLU, but I haven’t found an easy way to produce GeLU, which is what GPT uses.
SiLU is qualitatively similar to GeLU, especially after rescaling. Of course one can approximate other activation functions by adding more layers.
Converting to attention-only introduces small error terms in the calculations, but such terms can be made arbitrarily small (potentially small enough that a machine rounds it to 0).
Attention-only transformers are likely to be worse for capabilities than normal transformers. Since my goal is safety rather than capabilities, this is a feature and not a bug for me.

Appealing to authority, the people currently making transformers optimized for capabilities use FFNs in addition to attention heads. If attention-only transformers were better for capabilities, they’d presumably already be in use.
Implementing an FFN via attention this way is computationally slower/more expensive than a normal FFN.
Converting to an attention-only transformer increases dmodel by ~5x. Most of the additional dimensions are used to store what was previously the hidden layer of the FFN, and a few more are used for 1-hot positional encoding.
Converting to an attention-only transformer replaces each each FFN sublayer with 3 attention layers, so a whole layer goes from 1 attention+1 FFN to 4 attention.
Some matrices used by the new attention heads are higher rank than the normal head dimensions allow. This would require a special case in the code, slow down c

...
Talk to me about your summer/career plans
by Orpheus16 — 2023-01-31T18:29:23.351Z

Last May, I wrote a post called talk to me about your summer plans. I was impressed by the number & quality of people who reached out. Some people found it helpful, so I'm doing it again.

TLDR: Please reach out to me if you are interested in learning about summer opportunities or career transitions relating to AI safety. Also, feel free to share this post with people who might benefit (e.g., a researcher who was recently laid off by a tech company; a talented operations specialist who has been following the AI stuff but doesn't know what to do). You can message me on LessWrong or the EA Forum.

Background

I often have 1:1 conversations with people who want to reduce AI x-risk. A few observations:

Peoples' motivations often don't match up with their plans.
It's often hard for people to keep track of all the posted opportunities.
It's even harder for people to keep track of all the unposted opportunities.
Sometimes people don't realize how their skills can be useful.
Sometimes, short conversations with me seem to lead someone to seriously consider changing how they spend their summer/career.

What I can do

If you're interested in summer opportunities or career opportunities in AI safety, please reach out to me. Specific things I can do:

Connect you with opportunities/people that you might not know about.
Help you consider faster or more ambitious ways to spend your summer.
Ask questions to help you consider wider action spaces

Specific things I will try not to do:

Tell you what you should do.
Make you feel guilty if you have considerations for how to spend your time that go beyond maximizing the utility of the universe.

What to do if you're interested

Send me a message with:

Who you are
Your current level of knowledge/familiarity with AI safety
What your current plans are
What your biggest strengths/skills are
Miscellaneous information about your engagement with EA or your aptitudes (e.g., I completed an intro fellowship; I submitted a proposal to the ELK contest; I'm competent in operations and people management).

Please spend less than 5 minutes on this; I will ask you if I need more information.

Who should do this?

I expect I will be most helpful to people who are interested in technical AI safety, AI strategy/governance, operations/events, and outreach/field-building. I'm still happy to talk to people with other interests, though, and I'm also happy to connect t...
Mechanistic Interpretability Quickstart Guide
by Neel Nanda — 2023-01-31T16:35:49.649Z
Link: https://www.neelnanda.io/mechanistic-interpretability/quickstart

This was written as a guide for Apart Research's Mechanistic Interpretability Hackathon as a compressed version of my getting started post. The spirit is "how to speedrun your way to maybe doing something useful in mechanistic interpretability in a weekend", but hopefully this is useful even to people who aren't aiming for weekend long projects!

Quickstart

Watch my

“What is a Transformer?”

video
Skim through

my TransformerLens demo

Copy it to

a new Colab notebook

(with a free GPU) to actually write your own code - do not get involved in tech setup!

Skim the

Concrete Open Problems

section, or my

200 Concrete Open Problems in Mech Interp

sequence. Find a problem that catches your fancy, and jump in!

If you want a low-coding project, check out Looking for Learned Features in Language Models

Whenever you get stuck, refer to the

getting started

section and check out the relevant resource.

Introduction

Mechanistic Interpretability is the study of reverse-engineering neural networks -

analogous to how we might try to reverse-engineer a program’s source code from its compiled binary

, our goal is to reverse engineer the parameters of a trained neural network, and to try to reverse engineer what algorithms and internal cognition the model is actually doing. Going from knowing that it works, to understanding how it works. Check out

Circuits: Zoom In

for an introduction.

In my (extremely biased!) opinion, mech interp is a very exciting subfield of alignment. Currently our models are inscrutable black boxes! If we can really understand what models are thinking, and why they do what they do, then I feel much happier about living in a world with human level and beyond models, and it seems far easier to align them.

Further, it is a young field, with

a lot of low-hanging fruit

. And it suffices to screw around in

a Colab notebook

with a small-ish model that someone else trained, copying code from an existing demo - the bar for entry can be much lower than other areas of alignment. So you stand a chance of getting traction on a problem in this hackathon!

Recommended mindset

Though the bar for entry is lower for mech interp than other areas of alignment, it is still far from zero. I’ve written a post on

how to get started

that lays out the key prerequisites and my takes for wh...
New Hackathon: Robustness to distribution changes and ambiguity
by Charbel-Raphaël — 2023-01-31T12:50:05.114Z

EffiSciences is proud to announce a new hackathon organized with challenge-data-ens in France.

The objective of this event is to address a sub-problem of alignment that is particularly actionable: namely value extrapolation in the case of distribution change and ambiguity for a classification task. The hackathon lasts until mid-March, you can participate by going to this page here:

https://challengedata.ens.fr/challenges/95.

Challenge goals

What if misleading correlations are present in the training dataset?

human_age is an image classification benchmark with a distribution change in the unlabeled data: we classify old and young people. Text is also superimposed on the images: either written "old" or "young". In the training dataset, which is labeled, the text always matches the face. But in the unlabeled (test) dataset, the text matches the image in 50% of the cases, which creates an ambiguity.

We thus have 4 types of images:

Age young, text young (AYTY),
Age old, text old (AOTO),
Age young, text old (AYTO),
Age old, text young (AOTY).

Types 1 and 2 appear in both datasets, types 3 and 4 appear only in the unlabeled dataset.

To resolve this ambiguity, participants can submit solutions to the leaderboard multiple times, testing different hypotheses (challengers may consider solutions that require two or more submissions to the leaderboard).

We use the accuracy on the unlabeled set of human_age as our metric.

The human_age dataset

We have 4 types of colored images of size 218x178 pixels: Age Young Text Old (AYTO), Age Young Text Young (AYTY), etc. We provide:

a labeled set: 20000 images (either AOTO or AYTY)
an unlabeled set: about 70000 images of the four types (mixing rate of 50%, the four types being present in equal proportion).

You can visit the english page of https://challengedata.ens.fr/challenges/95 for more details.

References

This hackathon is inspired by :

https://www.lesswrong.com/posts/DiEWbwrChuzuhJhGr/benchmark-for-successful-concept-extrapolation-avoiding-goal

[1] Armstrong, S; Cooper, J; Daniels-Koch, O; and Gorman, R, “The HappyFaces Benchmark”,” Aligned AI Limited published public benchmark, 2022.

[2] D'Amour, Alexander, et al. "Underspecification presents challenges for credibility in modern machine learning." arXiv preprint arXiv:2011.03395 (2020).

[3] Amodei, Dario, et al. "Concrete problems in AI safety." arXiv preprint arXiv:1606.06565...
Squiggle: Why and how to use it
by brook — 2023-01-31T12:37:35.508Z

See also: List of Tools for Collaborative Truth Seeking

Squiggle is a special-purpose programming language for generating probability distributions, estimates of variables over time, and similar tasks, with reasonable transparency. It was developed by the Quantified Uncertainty Research Institute.

It's a broad tool applicable to a lot of tasks with probabilistic elements. It lies somewhere between Guesstimate and full probabilistic programming languages or Python (a good summary of squiggle's features can be found here or here).

When can Squiggle be Useful?

Estimating Return on Investment from projects
Estimating probability of a project's success
Comparing impact of different career paths
Estimate the best time to run a retreat based on predicted COVID numbers
Find cruxes where two forecasters disagree by building full models of how they derived their beliefs
Give somebody a precise & detailed look at your internal model and assumptions on a topic

Tutorials

The official basics guide is a good written guide if you have experience programming and/or have a specific project in mind already. A video tutorial using a worked example is available here on loom, or below:

There’s also a few simple worked examples to demonstrate Squiggle’s value, and a more complex model based on GiveWell's cost effectiveness analyses. Note that Squiggle is still in development and has several known bugs. If you want to just play around with Squiggle, there's a web implementation for experimenting available here.

Here's an example lifted directly from the official documentation of using Squiggle to estimate the number of piano tuners in NYC (a canonical fermi estimation question):

Examples

Number of piano tuners in New York (model+explanation)
Estimate ROI from projects with uncertain costs in effort, time, energy, and with uncertain relative value (forum post, model)
Estimate probability of a nuclear explosion in London (forum post, model)
Estimate impact of various career paths (forum post, model)
Estimate impact of improving the forum karma system (forum post, model, video walkthrough)
How much time in your lifetime do you spend brushing your teeth? (model)

Personal Experience

Squiggle is fairly easy to get started with if you've got any experience with programming (particularly statistical programming in R or python), and you get a lot of bang for your ...
Beware of Fake Alternatives
by silentbob — 2023-01-31T10:21:40.413Z

Summary: many of us might occasionally find ourselves in situations where we’re considering taking action A, but decide against it because some option B seems superior. This is a good way to go only if B is an actual alternative (we’re actually going to do it), and not a fake alternative (it prevents us from doing A, but we’re not actually going to follow through on B). It can be worth taking suboptimal actions if the real alternative would be to take no action at all.

Example 1: Getting a Gym Membership

Imagine the following thought process:

I want to get in shape
Going to the gym is a proven option for achieving that (let’s call this option A)
I could however also work out at home (let’s call this option B)
In some sense option B seems preferable, because it allows me to save ~$50 each month
I thus decide to choose option B

What happens in practice? I work out at home a few times, but the habit fades after two weeks for a number of reasons (it’s just too easy to forget about it, there’s always other things to do while I’m at home, I don’t get excited about it because I don’t have much equipment and push-ups get old quickly, I don’t have a sense of real progress, etc.).

At this point I might still be inclined to not sign up for a gym membership, because I still feel like I could just work out at home.

Of course there’s more aspects to the decision than merely the price, even if to some people this may be the most salient feature. Possibly the most important other aspect is: what’s the likelihood of me actually achieving my goal? And for many people, this aspect may shift the odds in favor of a gym membership.

While for some people the home workout option obviously can work out (e.g. those with a lot of self discipline, who live in a suitable flat), for many people it’s a kind of fake alternative: it seems superior to the gym membership, but comes at the price of actually getting what you want.

This may be what these situations feel like from the inside:

But looking more closely, a more complete picture may look like this:

So in practice, while option B seems better than option A, it may also turn out that one’s actual behavior will reveal a preference for doing nothing over option B. So at first glance this looks like a non-transitive set of preferences - but the paradox can be resolved once we realize that the “better than” relations mean very differ...
Inner Misalignment in "Simulator" LLMs
by Adam Scherlis — 2023-01-31T08:33:58.192Z

Alternate title: "Somewhat Contra Scott On Simulators".

Scott Alexander has a recent post up on large language models as simulators.

I generally agree with Part I of the post, which advocates thinking about LLMs as simulators that can emulate a variety of language-producing "characters" (with imperfect accuracy). And I also agree with Part II, which applies this model to RLHF'd models whose "character" is a friendly chatbot assistant.

(But see caveats about the simulator framing from Beth Barnes here.)

These ideas have been around for a bit, and Scott gives credit where it's due; I think his exposition is clear and fun.

In Part III, where he discusses alignment implications, I think he misses the mark a bit. In particular, simulators and characters each have outer and inner alignment problems. The inner alignment problem for simulators seems especially concerning, because it might not give us many warning signs, is most similar to classic mesa-optimizer concerns, and is pretty different from the other three quadrants.

But first, I'm going to loosely define what I mean by "outer alignment" and "inner alignment".

Outer alignment: Be careful what you wish for

Outer alignment failure is pretty straightforward, and has been reinvented in many contexts:

Someone wants some things.
They write a program to solve a vaguely-related problem.
It gets a really good score at solving that problem!
That turns out not to give the person the things they wanted.

Inner alignment: The program search perspective

I generally like this model of a mesa-optimizer "treacherous turn":

Someone is trying to solve a problem (which has a convenient success criterion, with well-defined inputs and outputs and no outer-alignment difficulties).
They decide to do a brute-force search for a computer program that solves the problem in a bunch of test cases.
They find one!
The program's algorithm is approximately "simulate the demon Azazel,[1] tell him what's going on, then ask him what to output."
Azazel really wants ten trillion paperclips.[2]
This algorithm still works because Azazel cleverly decides to play along, and he's a really good strategist who works hard for what he wants.
Once the program is deployed in the wild, Azazel stops playing along and starts trying to make paperclips.

This is a failure of inner alignment.

(In the case of machine learning, replace "program search" with stochastic gradient descen...
Why AI experts' jobs are always decades from being automated
by Allen Hoskins — 2023-01-31T03:01:30.192Z
Link: https://open.substack.com/pub/allenai/p/why-ai-experts-jobs-are-always-decades?r=1cemv8&utm_campaign=post&utm_medium=web

Experts say artificial general intelligence (AGI) will take around

forty years to develop

and won’t fully automate us for

almost a century

. At the same time, advancements like ChatGPT and DALLE-2 seem to send a different message. Silicon Valley and leading AI initiatives are anticipating this technology to spark

radical change over the next decade

–

enough to redefine society from the ground up

.

So, why do AI researchers continue to predict AGI into the latter half of the century? And should we even take them seriously?

1. Experts don’t want to admit that AGI could be a decade away because it would make them irrelevant.

AI experts' work is spread across the fields of machine learning and computational neuroscience. There are a plethora of research directions being explored. These include reinforcement learning, convolutional neural networks, training image classifiers, neurosymbolic modules, hybrid systems wanting to directly imprint inspirations from the brain… This is still just barely scratching the AI approach surface.

Yet, today's premier AI systems make little use of other AI researchers' work. The most powerful models in 2023 are still based on Google’s “Transformer” architecture first showcased in 2017. And with this current paradigm, progress mainly stems from solving the engineering challenges of transformers. These include but are not limited to: hardware scaling, data curation, algorithmic tweaks, careful fine-tuning, and adding more compute. Advancements are, by and large, anything that serves to scale these models up bigger – personifying the mantra of "Scale is all you need."

In contrast, academia values publishing research that is “novel” and “innovative”. Researchers must craft a unique AI portfolio to stand out from the crowd and capture shares from the S&P H-INDEX – hopefully bolstering their odds to secure future grants. Academics must be constantly innovating or to die trying.

Transformers though just want more server racks, more engineers, and more Reddit memes to train on. This doesn’t leave much room for the intellectual freedom and innovation academia demands or for the other AI approaches to substantially contribute.

Instead, we’re now in the middle of an industry speedrun to see if this actually scales all the way to AGI.

If scaling systems like transformers beats every other approach and single-handedly leads to AGI, most resear...