trentmkelly/LessWrong-43k huggingface dataset:
Mech Interp Puzzle 2: Word2Vec Style Embeddings
by Neel Nanda — 2023-07-28T00:50:00.297Z

Code can be found here. No prior knowledge of mech interp or language models is required to engage with this.

Language model embeddings are basically a massive lookup table. The model "knows" a vocabulary of 50,000 tokens, and each one has a separate learned embedding vector.

But these embeddings turn out to contain a shocking amount of structure! Notably, it's often linear structure, aka word2vec style structure. Word2Vec is a famous result (in old school language models, back in 2013!), that `man - woman == king - queen`. Rather than being a black box lookup table, the embedded words were broken down into independent variables, "gender" and "royalty". Each variable gets its own direction, and the embedded word is seemingly the sum of its variables.

One of the more striking examples of this I've found is a "number of characters per token" direction - if you do a simple linear regression mapping each token to the number of characters in it, this can be very cleanly recovered! (If you filter out ridiculous tokens, like 19979: 512 spaces).

Notably, this is a numerical feature not a categorical feature - to go from 3 tokens to four, or four to five, you just add this direction! This is in contrast to the model just learning to cluster tokens of length 3, of length 4, etc.

Question 2.1: Why do you think the model cares about the "number of characters" feature? And why is it useful to store it as a single linear direction?

There's tons more features to be uncovered! There's all kinds of fundamental syntax-level binary features that are represented strongly, such as "begins with a space".

Question 2.2: Why is "begins with a space" an incredibly important feature for a language model to represent? (Playing around a tokenizer may be useful for building intuition here)

You can even find some real word2vec style relationships between pairs of tokens! This is hard to properly search for, because most interesting entities are multiple tokens. One nice example of meaningful single token entities is common countries and capitals (idea borrowed from Merullo et al). If you take the average embedding difference for single token countries and capitals, this explains 18.58% of the variance of unseen countries! (0.25% is what I get for a randomly chosen vector).

Caveats: This isn't quite the level we'd expect for real word2vec (which should be closer to 100%), an...
ETFE windows
by bhauth — 2023-07-28T00:46:55.556Z
Link: https://www.bhauth.com/blog/construction/ETFE.html

Modern buildings have a lot of windows. If you look at a recent skyscraper, its facade is likely to be mostly glass. Why use glass for that? Why not use a sheet of plastic instead? Glass sheets are heavy and brittle; plastic film would be much lighter and cheaper.

The main problem with plastic film is UV degradation. Most plastic degrades over time when exposed to sunlight and air. Is it possible to make clear plastic with good resistance to UV and water? Could that be used instead of glass for buildings? Yes, and people have done that.

When ethylene and tetrafluoroethylene are polymerized together, they naturally tend to alternate, producing ETFE. Each carbon is either fluorinated or next to a fluorinated carbon, which increases UV resistance.

That wikipedia page lists some buildings using it, notably including the Eden Project and the "water cube". It also notes:

ETFE's first large-scale use architecturally came in 2001 at the Eden Project where ETFE was selected as it can be printed and layered to control solar conditions and because it was found to have a low friction coefficient, which saves on maintenance as dust and dirt do not stick.

Current ETFE building facades typically have 3 layers clamped at the edges with aluminum frame, with some air pumped between the layers. So, current designs typically use some electricity. The frames are more expensive than the ETFE film, and my understanding is that they result in ETFE being slightly more expensive than glass. Cost reduction of such frames seems possible.

ETFE has some notable advantages over glass:

As thermal insulation, ETFE is slightly better than the same number of glass layers: 3 layers are slightly better than triple-glazed windows. That's good.

The frames are heavier than the ETFE film, but the overall system is still typically <0.5x the weight of windows. That's good. (That lower weight is advantageous when moving building sections on wheels.)

ETFE is slippery. Water will run off it at small slopes. Dust adheres poorly to it, so cleaning isn't usually needed. That's good.

Windows with irregular shapes are easier with ETFE than glass. That's good as long as you trust architects to use this capability wisely.

There are also some ways in which ETFE is different from glass but not exactly better or worse:

ETFE film is much lighter and less rigid than glass, so it transmits sound much more and re

...
A Short Memo on AI Interpretability Rainbows

by scasper — 2023-07-27T23:05:50.196Z

TL;DR: More work to study interactions and synergies between different AI interpretability tools seems important, tractable, and neglected.

Interpretable AI research tends to focus on one technique at a time

There is a lot of

interpretable

/

explainable

AI research. But one thing that almost all research papers seem to have in common is a focus on individual techniques, one at a time. From a practical perspective, this makes sense – this is usually the simplest way to straightforwardly make progress by introducing something new. Studying interactions between multiple things can also quickly get hard and complicated. Progress in ML tends to work this way.

Studying combinations of interpretability tools may be valuable

However, sometimes great work can be done by combining multiple tools to study networks, and they often find complementary interactions between them here are

a

few

random

examples

.

In engineering disciplines, progress is rarely made by the introduction of single methods that make a huge splash. Instead, game-changing innovations usually come from multiple techniques being applied at once. Progress on ImageNet is an excellent example of this. It took combinations of many new techniques – batch normalization, residual connections, inception modules, deeper architectures, hyperparameter engineering, etc – to improve the performance of CNNs so much so quickly. There was no silver bullet.

Progress on ImageNet has been driven by combining many techniques together over the years.

An example of “rainbowing” from RL

There is some precedent for valuable, influential papers that make no novel contributions other than combining old techniques. A somewhat-famous example of this comes from reinforcement learning.

Hessel et al. (2017)

improved greatly on the previous state-of-the-art on deep Q learning by simply combining a bunch of existing tricks. Because their technique combined many compatible old ones, they called it “Rainbow.”

In RL, the "Rainbow" paper set a new state-of-the-art with no new contributions at all.

It is particularly easy to combine intrinsic and post-hoc interpretability tools

In AI interpretability research, interesting combinations of methods are especially easy to come by. Interpretability tools can be divided into two kinds: intrinsic interpretability tools which are applied before or durin...
Pulling the Rope Sideways: Empirical Test Results
by Daniel Kokotajlo — 2023-07-27T22:18:01.072Z

Robin Hanson wrote this post a while back, which has since developed into a sort of inspiring slogan for how to achieve policy change: "Pull the Rope Sideways"

The policy world can thought of as consisting of a few Tug-O-War "ropes" set up in this high dimensional policy space. If you want to find a comfortable place in this world, where the people around you are reassured that you are "one of them," you need to continually and clearly telegraph your loyalty by treating each policy issue as another opportunity to find more supporting arguments for your side of the key dimensions. That is, pick a rope and pull on it.

If, however, you actually want to improve policy, if you have a secure enough position to say what you like, and if you can find a relevant audience, then prefer to pull policy ropes sideways. Few will bother to resist such pulls, and since few will have considered such moves, you have a much better chance of identifying a move that improves policy. On the few main dimensions, not only will you find it very hard to move the rope much, but you should have little confidence that you actually have superior information about which way the rope should be pulled.

I recently wrote in a casual slack channel:

Hanson introduced the term "pull the rope sideways" and I think it's a good concept. However the metaphor bugs me a bit. In a tug-o-war, is it actually the case that pulling sideways causes the rope to move much farther that if you pull the rope in one of the normie directions? Has anyone tested this? It's not obvious to me why this would be true.

Vigorous debate ensued. We bought some rope and did a quick test.

Contrary to the expectations of most people in the thread, pulling the rope sideways didn't seem to work -- the overall effect seemed to be about the same as pulling it in one of the standard directions.
Afterwards we joked about the implications for EA macrostrategy. "I guess we should just... pick a side." :D
A $10k retroactive grant for VaccinateCA
by Austin Chen — 2023-07-27T18:14:44.305Z
Link: https://manifund.org/projects/vaccinateca

I'm awarding a $10k retroactive grant to Karl Yang, for his role in starting VaccinateCA. In January 2021, the Covid vaccines were just starting to become available to the general public… in theory. In practice, there was a lot of confusion over how to get a dose. Demand for vaccines was overwhelming; information about them was scarce. Karl took a simple idea (“locate where the Covid vaccines are, and publish it”), and organized a volunteer effort that would go on to help millions of people get their shots.

Q&A with Karl

Adapted from a 30 min interview between Karl and Austin.

Q: What was Karl’s role in starting VaccinateCA?

A: Back in Jan 2021, I’d heard that the vaccines were coming out. I was paying attention as my family had at-risk members, but nobody knew what was happening. Then, Patrick tweeted out a project proposal: create a public list of hospitals in California.

I read that and thought: “this seems like something I can do”. And I was motivated to do something. Throughout the pandemic, I’d been waiting for some way to help. There’s always a vague “they” implicit in “they should do something”. But I was antsy. Why shouldn’t it be an “us”?

Usually, work like this feels outside of my skillset or capabilities. For example, I’d love to accelerate cancer research, but I have no relevant training. But Patrick packaged it in a way that I could act on. I can set up a spreadsheet, I can coordinate volunteers, I can populate a list of hospitals I can call. I can do this.

So I set up a Google doc, found some phone numbers, and planned to start calling them the next day. I also put out a call for help on Twitter; seeing this, a bunch of people DM’d me on Twitter.

I started a Discord server, inspired by the “eye mouth eye” thing (where a bunch of GenZ kids meme’d a fake startup into existence, and managed to get term sheets). Over Discord, I coordinated a campaign to call the next day. I was already working the night shift at the time, so I got up at 6am to start making phone calls.

Next, I set up a website to display what information we had. A bunch of volunteers quickly came in and made the website better; then other volunteers set up a backend on Airtable and got free credits from the Airtable team. By the next day, we had built a sophisticated volunteer dialer system. It would pull a phone number from the queue, and have a list of questions for the volunte...
Preference Aggregation as Bayesian Inference
by beren — 2023-07-27T17:59:36.270Z

Crossposted from my personal blog.

A fundamental problem in AI alignment, as well as in many social sciences is the problem of preference aggregation. Given a number of different actors who have specific preferences, what is a consistent way of making decisions that ensures that the outcome is fair and ideally that all of the agents are about as happy as they can be [1]. In AI alignment, this is essentially the outer alignment problem which, in layman’s terms is: assuming we have an AI superintelligence which makes all decisions according to a utility function, what should that utility function be?

Ideally, the utility function pursued by the aligned superintelligence, or by the democratic state in political science, in some sense closely and fairly represents the utility functions, interests, and values of the constituent stakeholders: in the case of a (direct) democratic state this is all citizens with voting rights; for an aligned superintelligence ideally this would be some form of humanity in general.

For the case of democracies we have a long history of political science which has coalesced around various forms of voting. Each stakeholder gets some number of votes and can either vote for their chosen policy or candidate or rank them. Then the highest ranking candidate or policy gets selected for implementation. Voting often works reasonably in practice and generates decisions that are usually seen as fair even if in practice direct voting is often significantly watered down, usually to prevent tyrannies of the majority by mechanisms such as constitutions which define democratically-inalienable rights, independent judiciaries which can strike down democratic laws, and most importantly representative democracy where voters do not vote directly on policies but instead for representatives who in theory should vote for their electorate’s desired policies but in practice often do other things. For more theoretical questions of alignment most discussion centers around the vaguely defined notion of coherent extrapolated volition (CEV).

The fundamental question is how do we somehow combine the preferences of the constituents into a coherent utility function or make an action which somehow respects their preferences?

My answer is that we already have an extremely well-developed mathematical machinery for dealing with questions which, I claim, are essentially iso...
AI #22: Into the Weeds
by Zvi — 2023-07-27T17:40:02.184Z

Events continue to come fast and furious. Custom instructions for ChatGPT. Another Congressional hearing, a call to understand what is before Congress and thus an analysis of the bills before Congress. An joint industry safety effort. A joint industry commitment at the White House. Catching up on Llama-2 and xAI. Oh, and potentially room temperature superconductors, although that is unlikely to replicate.

Is it going to be like this indefinitely? It might well be.

This does not cover Oppenheimer, for now I’ll say definitely see it if you haven’t, it is of course highly relevant, and also definitely see Barbie if you haven’t.

There’s a ton to get to. Here we go.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Ask for Patrick ‘patio11’ McKenzie.

Language Models Don’t Offer Mundane Utility. Not with that attitude.

Has GPT-4 Gotten Worse? Mostly no.

But Doctor, These Are the Previous Instructions. Be direct, or think step by step?

I Will Not Allocate Scarce Resources Via Price. This week’s example is GPUs.

Fun With Image Generation. An illustration contest.

Deepfaketown and Botpocalypse Soon. Who should own your likeness? You.

They Took Our Jobs. Go with it.

Get Involved. People are hiring. Comment to share more positive opportunities.

Can We X This AI? Elon Musk is not making the best decisions lately.

Introducing. Room temperature superconductors?!?! Well, maybe. Probably not.

In Other AI News. Some minor notes.

Quiet Speculations. A view of our possible future.

White House Secures Voluntary Safety Commitments. A foundation for the future.

The Ezra Klein Show. Oh, you want to go into the weeds of bills? Let’s go.

Another Congressional Hearing. Lots of good to go with the cringe.

The Frontier Model Forum. Could be what we need. Could be cheap talk.

The Week in Audio. Dario Amodei and Jan Leike.

Rhetorical Innovation. An ongoing process.

Defense in Depth. They’ll cut through you like Swiss cheese.

Aligning a Smarter Than Human Intelligence is Difficult. We opened the box.

People Are Worried About AI Killing Everyone. For good reason.

Other People Are Not As Worried About AI Killing Everyone. You monsters.

Other People Want AI To Kill Everyone. You actual monsters.

What is E/Acc? Why is this the thing to focus on accelerating?

The Lig

...
SSA rejects anthropic shadow, too
by jessicata — 2023-07-27T17:25:17.728Z
Link: https://unstableontology.com/2023/07/27/ssa-rejects-anthropic-shadow-too/

(or: "Please Do Anthropics with Actual Math")

The anthropic shadow argument states something like:

Anthropic principle! If the LHC had worked, it would have produced a black hole or strangelet or vacuum failure, and we wouldn't be here!

or:

You can't use "we survived the cold war without nuclear war" as evidence of anything. Because of the anthropic principle, we could have blown up the human race in the 1960's in 99% of all possible worlds and you'd still be born in one where we didn't.

This argument has already been criticized (here, here). In criticizing it myself, I first leaned on reasoning about large universes (e.g. ones where there are 100 worlds with low nuclear risk and 100 with high nuclear risk in the same universe) in a way that implies similar conclusions to SIA, thinking that SSA in a small, single-world universe would endorse anthropic shadow. I realized I was reasoning about SSA incorrectly, and actually both SSA and SIA agree in rejecting anthropic shadow, even in a single-world universe.

Recapping the Doomsday Argument

To explain SSA and SIA, I'll first recap the Doomsday Argument. Suppose, a priori, that it's equally likely that there will be 1 billion humans total, or 1 trillion; for simplicity, we'll only consider these two alternatives. We could number humans in order (numbering the humans 1, 2, ...), and assume for simplicity that each human knows their index (which is the same as knowing how many humans there have been in the past). Suppose you observe that you are one of the first 1 billion humans. How should you reason about the probability that there will be 1 billion or 1 trillion humans total?

SSA reasons as follows. To predict your observations, you should first sample a random non-empty universe (in proportion to its prior probability), then sample a random observer in that universe. Your observations will be that observer's observations, and, ontologically, you "are" that observer living in that universe.

Conditional on being in a billion-human universe, your probability of having an index between 1 and 1 billion is 1 in 1 billion, and your probability of having any other index is 0. Conditional on being in a trillion-human universe, your probability of having an index between 1 and 1 trillion is 1 in 1 trillion, and your probability of having any other index is 0.

You observe some particular index that does not exceed 1 bill...
AI-Plans.com 10-day Critique-a-Thon 
by Iknownothing — 2023-07-27T11:44:01.660Z
Link: https://manifund.org/projects/ai-planscom-critique-a-thon-300-prize-fund-proposal

Introduction

At AI-Plans.com, we are creating an accessible, centralized platform to discover and learn about the numerous alignment plans in existence. Our mission is to make the field of AI Alignment more transparent and accessible by providing a comprehensive compendium of alignment plans, along with criticisms against them.

As part of our effort to crowdsource knowledge and foster a critical, constructive discussion in the AI Alignment community, we are excited to announce the AI-Plans.com Critique-a-Thon, set to begin on the 1st of August and last for 10 days.

Proposal

We propose a prize fund of $500 to be awarded to the authors of the best critiques submitted during the Critique-a-Thon. Critiques will be judged based on the following criteria:

Accuracy: The criticism must correctly represent the alignment plan being critiqued. Misrepresentations would be deemed as unfair criticism.
Precision: Critiques must be precise and specific about the problem identified. Vague criticisms will be considered weaker than those providing an exact argument that could be assessed or refuted.
Communication: The criticism must clearly articulate its point. The critique should be formatted well, concise, and minimize the use of specialist language.
Evidence: Criticisms should be backed by logical reasoning and objective evidence, rather than speculative opinions.
Novelty: The criticism should raise new issues not already covered by others. Novel critiques will be considered more valuable than those rehashing known issues.

The judgments will be made by myself, members from our team(which includes an expert QA and Azai who has a strong background in mathematics), DragonGod (studying a Theoretical Computer Science Msc ), and a select group of alignment researchers.

Who is on your team and what's your track record on similar projects?

Our team includes Jonathan Ng, an independant alignment researcher, who will be taking a look at some critiques. Other members of the team include an expert QA with many years of experience and Azai, who has a strong background in mathematics. DragonGod, who is studying an Msc in Theoretical Computer Science will also be a judge for critiques. We also have a consultant who is CompTia certified, highly skilled in cybersecurity and red-teaming and a professor.
We have successfully launched AI-Plans.com in beta and have already added over 100 alignment plans to...