trentmkelly/LessWrong-43k huggingface dataset:
Worldly Positions archive, briefly with private drafts
by KatjaGrace — 2022-12-30T12:20:05.430Z

I realized it was hard to peruse past Worldly Positions posts without logging in to Tumblr, which seemed pretty bad. So I followed Substack’s instructions to import the archives into world spirit sock stack. And it worked pretty well, except that SUBSTACK ALSO PUBLISHED MY UNPUBLISHED WORLDLY POSITIONS DRAFTS! What on Earth? That’s so bad. Did I misunderstand what happened somehow in my rush to unpublish them? Maybe. But they definitely had ‘unpublish’ buttons, so that’s pretty incriminating.

This seems to have turned out alright for me, since it looks like I just never wrote any drafts that would be too embarrassing to anyone other than myself. And the most embarrassing to myself are probably at the level of bad and abortive poetry. Plus it turned up a few decent drafts to finish, and the adrenaline was a welcome pick-me-up in my current retreat-driven stimulation drought.

Some good bits of the archive (from pre-WSSP times) according to me:

Mine-craft: the composition of the ego in a procedurally generated sandbox game

The time I rented a robot baby

Why fiction is more horrifying than war photography

Home: up and down, colder and warmer: miscellanious thoughts on e.g. warmth and coldness, the sincerity of historic advertising, and why negativity is deep

How I learned to have fun on command though I rarely remember to do it

England: Attunement and borders, in which I get possible attunement, companionship, and a visa
Models Don't "Get Reward"
by Sam Ringer — 2022-12-30T10:37:11.798Z

In terms of content, this has a lot of overlap with Reward is not the optimization target. I'm basically rewriting a part of that post in language I personally find clearer, emphasising what I think is the core insight.

When thinking about deception and RLHF training, a simplified threat model is something like this:

A model takes some actions.
If a human approves of these actions, the human gives the model some reward.
Humans can be deceived into giving reward in situations where they would otherwise not if they had more knowledge.
Models will take advantage of this so they can get more reward.
Models will therefore become deceptive.

Before continuing, I would encourage you to really engage with the above. Does it make sense to you? Is it making any hidden assumptions? Is it missing any steps? Can you rewrite it to be more mechanistically correct?

I believe that when people use the above threat model, they are either using it as shorthand for something else or they misunderstand how reinforcement learning works. Most alignment researchers will be in the former category. However, I was in the latter.

I was missing an important insight into how reinforcement learning setups are actually implemented. This lack of understanding led to lots of muddled thinking and general sloppiness on my part. I see others making the exact same mistake so I thought I would try and motivate a more careful use of language!

How Vanilla Reinforcement Learning Works

If I were to explain RL to my parents, I might say something like this:

You want to train your dog to sit.
You say "sit" and give your dog a biscuit if it sits.
Your dog likes biscuits, and over time it will learn it can get more biscuits by sitting when told to do so.
Biscuits have let you incentivise the behaviour you want.
We do the same thing with a computer by giving the computer "reward" when it does things we like. Over time, the computer will do more of the behaviour we like so it can get more reward.

Do you agree with this? Is this analogy flawed in any way?

I claim this is actually NOT how vanilla reinforcement learning works.
The framing above views models as "wanting" reward, with reward being something models "receive" on taking certain actions. What actually happens is this:

The model takes a series of actions (which we collect across multiple "episodes").
After collecting these episodes, we determine how good the actions in ea

...
The hyperfinite timeline
by Alok Singh — 2022-12-30T09:30:06.483Z

Take an interval. Cut it into H pieces, where H is hyperfinite. This serves as the index set of a stochastic process, among many other uses. Imagine that for each of the H steps, you flip a coin to get -1 or +1. Then move an infinitesimal distance left or right based on the sign. This is Brownian motion. Each infinitesimal piece of the timeline is profitably thought of as a Planck time.

Discrete events, such as sudden hard shocks, can be modeled on this line. They are appreciable over an infinitesimal fraction of the line.
Things I carry almost every day, as of late December 2022
by DanielFilan — 2022-12-30T07:40:01.261Z

Here we see things I carry in my pockets regularly.

To the left is my phone case, with two bandaids in it. This is a stand-in for the phone (a Pixel 6) that the phone case usually encases (I was busy taking photos with it). I keep two bandaids between the case and the phone in case I need one - which does sometimes happen. I find my phone useful for the normal things one uses a phone for. As of this year, I’ve started using it in lieu of a credit card, which feels very cool and 21st century.

At the top right is a pocket constitution made by Legal Impact for Chickens. I received this at an Effective Altruism Global conference, during the career fair. What actually happened was that someone came up to the booth I was at holding the pocket constitution, I noted that it looked cool, and they were kind enough to offer it to me. Unfortunately, I have never knowingly met anybody from Legal Impact for Chickens. I have not actually used this pocket constitution, but I carry it anyway in my winter jacket’s inner breast pocket since (a) it fits very unobtrusively and (b) it seems cool to carry around a pocket constitution.

At the bottom right is my wallet, a Bellroy Slim Sleeve. I very much like the material it is made of: called “baida nylon”, it is pleasingly canvas-like. Unfortunately I do not know how to faithfully convey this to you via the internet, but you will have to take my word that this is one of my two favourite purchases in 2022, ranked by my tactile pleasure in interacting with it.

Here we see the wallet open. On the left is my student ID card (with my COVID vaccine card tucked behind it out of sight), with a collection of folded bills tucked behind it. On the right is my Alcor membership card. Behind the right card is a tab that can be pulled to reveal more cards.

As you can see, there are several cards stowed away: on top are emergency medical instructions if I am found dead, to prepare my head for cryogenic storage, and below that is:

my WeWork card

my health insurance card

my public transit pass

my state ID card, and

my credit card.

All in all, I like the way this wallet lets me store many cards with very little space. But this is not all that is in my wallet.

Inside the bills, I have hidden away two items. The first is a Purell hand sanitizing wipe, useful when I touch something gross and want to disinfect it (or when something gross gets on ...
More ways to spot abysses
by KatjaGrace — 2022-12-30T06:30:06.301Z

I liked Ben Kuhn’s ‘Staring into the abyss as a core life skill’.

I’d summarize as:

If you are making a major error—professionally, romantically, religiously, etc—it can be hard to look at that fact and correct.

However it’s super important. Evidence: successful people do this well.

This seems pretty plausible to me.

(He has a lot of concrete examples, which are probably pretty helpful for internalizing this.)

His suggestions for how to do better helped me a bit, but not that much, so I made up my own additional prompts for finding abysses I should consider staring into, which worked relatively well for me:

If you were currently making a big mistake, what would it be?

What are some things that would be hard to acknowledge, if they were true?

Looking back on this time from five years hence, what do you think you’ll wish you changed earlier?

If you were forced to quit something, what do you want it to be?

(Variant on 1:) If you were currently making a big mistake that would be gut-wrenching to learn was a mistake, what would it be?
Language models are nearly AGIs but we don't notice it because we keep shifting the bar
by philosophybear — 2022-12-30T05:15:15.625Z

I’m putting my existing work on AI on Less Wrong, and editing as I go, in preparation to publishing a collection of my works on AI in a free online volume. If this content interests you, you could always follow my Substack, it's free and also under the name Philosophy Bear.

Anyway, enjoy. Comments are appreciated as I will be rewriting parts of the essays before I put them out. A big thank you to user TAG who identified a major error in my previous post regarding the Chinese Room Thought experiment, which prompted its correction [in the addition that will go in the book] and a new corrections section for my Substack page.

Glossary:

GPT-3- a text-generating language model.

PaLM-540B- a stunningly powerful question-answering language model.

Great Palm- A hypothetical language model that combines the powers of GPT-3 and PaLM-540B. Probably buildable with current technology, a lot of money and a little elbow grease.

Great Palm with continuous learning (GPWCL)- A hypothetical language model that combines the capacities of GPT-3 and PaLM-540B, with an important additional capacity. Most language models work over a “window” of text, functioning as short-term memory. Their long-term memory is set by their training. Continuous learning is the capacity to keep adding to long-term memory as you go, and this would allow a language model to tackle much longer texts.

The argument

What I’ll be doing in this short essay is a bit cheeky, but I think we’ll make a few important points, viz:

Goals that seem very concrete can turn out to be vulnerable to bar-shifting- shifting which we may scarcely even notice.
AGI is such a goal.
We have gotten very good, much too good, at denying the progress we have made in AGI.
A focus on able-bodied humanity, and the tendency to forget disabled people exist when thinking about these topics, deceives us in these matters.

If I’m being a bit of a gadfly here, it’s not without a purpose.

Everything I say in this article in a sense maybe applies to GPT-3 alone, but for the avoidance of doubt, let me specify that I’m talking about a hypothetical language model that has the fluency of GPT-3 and the question-answering capabilities of PaLM-540B which we will call The Great Palm to make it clear that we’re not taking ourselves too seriously. In my view, The Great Palm is very close to being an AGI.

I think the Great Palm lacks only one thing, the capacity for...
Progress links and tweets, 2022-12-29
by jasoncrawford — 2022-12-30T04:54:51.905Z
Link: https://rootsofprogress.org/links-and-tweets-2022-12-29

Links

City size is determined by transportation technology (Tomas Pueyo)

Tweets

Did any sci-fi predict that when AI arrived, it would be unreliable, often illogical, and frequently bullshitting?
Sam Altman: in the next decade the marginal cost of intelligence and of energy will trend to zero
Where to, Mr. Chemist? “To a thousand untouched shores” (Du Pont, 1939)
Longevity escape velocity
Study by Philip Tetlock finds “a pervasive tendency… to see things as getting worse than they really are”
“Scientists may find themselves reporting only successful experiments”

Quotes

Sustainability is the disease, people are the cure
For the first 40 years of photography, handheld cameras were impossible
How communication and transportation networks help eliminate famine

Queries

Data on European pipeline gas and storage? (@Atomicrod)

Retweets

“Abundance is the only cure for scarcity… Everything else merely allocates scarcity” (@samhinkie, quoting @patio11)
Converting coal plants into nuclear with small modular reactors. But: Congress ordered the NRC to provide a pathway for new reactors—and they made the process more burdensome (@AlecStapp)
Photographer uses AI to generate image variations from his own photos (@jonst0kes)
If you want home values to go up, you don’t want homes to be affordable (@JerusalemDemsas)
The bureaucracy feedback loop, green card edition (@nabeelqu)
How Theodor Engelmann demonstrated that chloroplasts are the site of oxygen production in plants (@strandbergbio)
Announcing The Filan Cabinet
by DanielFilan — 2022-12-30T03:10:00.494Z

Happy holidays! Some months ago, I launched a new podcast called The Filan Cabinet, but forgot to announce the podcast on this blog. Today, I rectify that mistake.

In some ways, the podcast is similar to AXRP - the AI X-risk Research Podcast. On that show, I interview AI x-risk researchers about their work, and try to bring their underlying views about AI x-risk research into the open: why do they think what they’re doing matters, and which research avenues do they find more or less promising?

The main difference is that in The Filan Cabinet, I talk about whatever I want to talk about, while still maintaining the goal of helping my audience understand my guests’ perspectives. To give you some sense of the show’s range, the first four episodes are about:

Politics, pandemic preparedness, and semiconductor policy

A conservative Presbyterian view of God and Christianity

Cryptocurrency and stablecoins

The practicalities and politics of paid plasma donation

A secondary goal of the podcast is to give me practice at interviewing well, with the hope that this practice improves AXRP. With this in mind, I’ve optimized the production process for speed, meaning that I do less research before each interview, and that I do not release transcripts for episodes. With luck, this will enable me to release more frequently without sacrificing too much quality.

If you would like to listen to the show, you can search “The Filan Cabinet” on your podcast app of choice, or just click here to see it on Google Podcasts. You can also see announcements of new episodes on this Twitter account. You should see some new episodes being released in 2023.
But is it really in Rome? An investigation of the ROME model editing technique
by jacquesthibs — 2022-12-30T02:40:36.713Z

Thanks to Andrei Alexandru, Joe Collman, Michael Einhorn, Kyle McDonell, Daniel Paleka, and Neel Nanda for feedback on drafts and/or conversations which led to useful insights for this work. In addition, thank you to both William Saunders and Alex Gray for exceptional mentorship throughout this project.

The majority of this work was carried out this summer. Many people in the community were surprised when I mentioned some of the limitations of ROME (Rank-One Model Editing), so I figured it was worth it to write a post about it as well as other insights I gained from looking into the paper. Most tests were done with GPT-2, some were done with GPT-J.

The ROME paper (Locating and Editing Factual Associations in GPT) has been one of the most influential papers in the prosaic alignment community. It has several important insights. The main findings are:

Factual associations such as “The Eiffel Tower is in Paris” seem to be stored in the MLPs of the early-middle layers of a GPT model. As the Tower token passes through the network, the MLPs of the early-middle layers will write information (e.g. the Eiffel Tower’s location) into the residual so that the model can later read that information to generate a token about that fact (e.g. Paris).
Editing/updating the MLP of a single layer for a given (subject, relationship, object) association allows the model to generate text with the updated fact when using new prompts/sentences that include the subject tokens. For example, editing “The Eiffel Tower is in Paris Rome” results in a model that outputs “The Eiffel Tower is right across from St Peter’s Basilica in Rome, Italy. “

In this post, I show that the ROME edit has many limitations:

The ROME edit doesn’t generalize in the way you might expect. It’s true that if the subject tokens you use for the edit are found in the prompt, it will try to generalize from the updated fact. However, it doesn’t “generalize” in the following ways:

It is not direction-agnostic/bidirectional. For example, the ROME edit is only in the "Eiffel Tower is located in ____" direction, not in the "Rome has a tower called the ____" direction.
It’s mostly (?) the token association being edited, not the concept. “Cheese” and “Fromage” are separate things, you’d need to edit both.
I hoped that if you edit X (e.g. The Rock) and then tried to describe X without using the t

...
A Year of AI Increasing AI Progress
by TW123 — 2022-12-30T02:09:39.458Z

In July, I made a post about AI being used to increase AI progress, along with this spreadsheet that I've been updating throughout the year. Since then, I have run across more examples, and had others submit examples (some of which were published before the date I made my original post).

2022 has included a number of instances of AI increasing AI progress. Here is the list. In each entry I also credit the person who originally submitted the paper to my list.

A paper from Google Research used a robust supervised learning technique to architect hardware accelerators [March 17th, submitted by Zach Stein-Perlman]
A paper from Google Research and Stanford fine tuned a model on its own chain-of-thought outputs, to improve performance on reasoning tasks [March 28th, submitted by Nathaniel Li]
A paper from OpenAI used LLMs to help humans find flaws in other LLMs, thereby enabling them to more easily improve those models [June 12th, submitted by Dan Hendrycks]
A paper from Google used machine learning to optimize compilers. This is less obviously accelerating AI but an earlier version of the compiler is used in Pytorch so it may end up doing so. [July 6th, submitted by Oliver Zhang]
NVIDIA used deep reinforcement learning to generate nearly 13,000 circuits in their newest GPUs. [July 8th, submitted by me]
Google found that ML code completion improved the productivity of their engineers. Some of them are presumably working in AI. [July 27th, submitted by Aidan O'Gara]
A paper from Microsoft Research and MIT used language models to generate programming puzzle tasks for other language models. When finetuned on these tasks, the models were much better at solving the puzzles. [July 29th, submitted by Esben Kran]
A paper from Google and UIUC used outputs from a language model to fine tune a language model after a majority vote procedure was used to filter outputs. [September 30th, submitted by me]
A paper from DeepMind used reinforcement learning to discover more efficient matrix multiplication algorithms. [October 5th, submitted by me]
A paper from Anthropic used language models, rather than humans, for feedback to improve language models. [December 16th, submitted by me]
A paper from a number of universities used language models to generate examples of instruction following, which were then filtered and used to fine tune language models to follow instructions better. [December

...
Why not spend more time looking at human alignment?
by ajc586 — 2022-12-30T00:22:13.666Z

As we approach AGI, we also approach the risk of alignment failure: due to either mismatch between intended goals and specified goals ('outer misalignment'), or mismatch between specified goals and emergent goals ('inner misalignment'), we end up with a catastrophically powerful failure mode that puts us all in a very bad place.

Right now, we don't know what kind of model will lead to AGI. We can guess, but a few years back people didn't have a lot of hope for LLMs, and now look where we're at. A lot of people were deeply surprised. Solutions to alignment failure may be model dependent, and AGI may emerge from something entirely new, or something old which starts to behave in a surprising way when scaled. It's quite hard to know where to start when faced with unknown unknowns. Of course, if you're here, you already know all of this.

These challenges and uncertainties may be relatively novel to the academic research community (in which AI safety has only recently become a respectable area of study), but they are by no means novel to most parents, who also train models (albeit 'wet' ones) via RLHF and inevitably experience alignment issues. Much of the time, the biggies get mostly resolved by adulthood and we end up with humans that balance self-interest with respect for humanity as a whole, but not always. For example, assertiveness, persuasiveness, public speaking ability and self-confidence are common training goals but when alignment failure occurs, you get humans (who often manage to convince the masses to elect them democratically) that are just one button push away from the destruction of humanity, and of the pathological personality type to actually push it. We don't need AGI to see alignment failure resulting in the future of humanity on a knife edge. We're already there.

Since we expect AGI (even superhuman AGI) to exhibit a large number of human-type behaviours because their models will presumably be trained on human output, why do we not spend more time looking at alignment failure in humans, its determinants, its solutions and its warning signs, and see whether those things yield insights into AI alignment failure mechanisms? As any human parent (like me) will tell you, although probably not in quite these words, aligning biological general intelligences is hard, counterintuitive and also immensely important. But, unlike AI alignment, there is a vast...
Why and how to write things on the Internet
by benkuhn — 2022-12-29T22:40:04.636Z

Recently I noticed that most existing “why you should write a blog” articles (e.g.) have at least one of two shortcomings, according to me:

They mostly focus on counterarguments to not starting a blog, rather than positive arguments in favor of starting one—as if people’s natural state is to produce amazing blogs and the only thing holding them back is silly misconceptions. This might be true for extreme outliers like Scott Alexander, but personally, my natural state is to play lots of video games and I need to be convinced to do other stuff instead.

They don’t set the reader up to actually get the benefits of blogging, because they don’t give concrete enough advice on the mechanics. I’ve learned a lot of things about blogging which, if I’d known them earlier, would have helped me improve much faster (and therefore would have made me much more likely to keep blogging, since part of my motivation for writing is being good at it).

So here’s my own take on this genre of post, with positive reasons why you, yes you, should consider writing things on the Internet, and a guide that’s as concrete as I can manage for how to do a decent (imo) job at it.

To put my money where my mouth is, I’m offering to be a draft reader for anyone who is motivated by this post to start a blog; see Appendix: standing offer for details.

Why

More awesome friendships

In my opinion, the strongest reason for any random person to start a blog is that you will have more awesome friendships—both in the sense that you will meet new awesome people as a result of your blog, but also in the sense that writing will cause you to have more interesting ideas,1 which will make your existing friendships more awesome because you’ll have better stuff to talk about.

Most other important things in life, like job opportunities and romantic relationships, are downstream of the quality of your friends, so this is pretty great.

Examples from my own blog:

Often when I’m talking to friends I’ll try to explain to them the post that I’m in the process of writing, or just published, and this leads to an interesting conversation about their take on the topic. I’ve had a lot of great conversations about e.g. Searching for Outliers and You don’t need to work on hard problems.

I’d guess that for around 25% of my friends,2 I either first met them through one of our blogs or our friendship was significantly reinf

...