trentmkelly/LessWrong-43k huggingface dataset:
My summary of the alignment problem
by Peter Hroššo — 2022-08-11T19:42:25.358Z
Link: https://threadreaderapp.com/thread/1557597583077298176.html

I've been looking into the AI alignment problem last couple of days and came up with the following summary of what problems there are and why. Also, I'd prefer using the umbrella name of Human alignment problem, as AI alignment is just a subset of it.

The problem is that we don't know what we want.
And even if we individually knew, we couldn't agree with others. (opinion aggregation)
Even if we agreed with others what we want, it would be hard to implement it. (coordination)

Because of game theory, momentum, and because of disagreements about how to do it.

Maybe we can create something smarter than us that solves these problems.
But we don't know how to create something smarter than us.
Maybe we can create something that will start out dumber, but can learn and will eventually become smarter.
We are afraid that something like this could become very powerful very quickly, and it’s likely to kill us - either as a mere side-effect or because of conflicting goals. (AI alignment problem)
But we don't know how to describe what it should learn. (outer alignment)
So maybe we can just give examples of what we know we want it to learn. (ML training)

We could make it learn from humans, but data from humans are inconsistent.

But it's impossible to describe all the cases, so in practice the situations facing the ML model will be quite different. (distributional shift)
And if what we want the ML model to learn is very specific and complicated, it's quite likely that what the model learns will behave very differently outside of our examples than how we'd want it to. (inner alignment)
It will also be hard to distinguish the cases where it does and where it doesn't do what we want. (eliciting latent knowledge)

Generally, sufficiently capable ML models are hard to understand. (interpretability)

Especially if the model knows it can do more of the stuff from training when we are not looking. (deceptive mesa optimizers)
Also, if we realize it’s doing something else than what we wanted it to, it might be hard to change it, because you’d be interfering with its learned goals. (corrigibility)

This is just a summary of my current understanding of the problem landscape. I don't subscribe to the stated motivations and conclusions, but more about that some other time.

Please, let me know if I omitted or misrepresented some important aspect of the problem (given how simplified version it intends to be).
Language models seem to be much better than humans at next-token prediction
by Buck — 2022-08-11T17:45:41.294Z

[Thanks to a variety of people for comments and assistance (especially Paul Christiano, Nostalgebraist, and Rafe Kennedy), and to various people for playing the game. Buck wrote the top-1 prediction web app; Fabien wrote the code for the perplexity experiment and did most of the analysis and wrote up the math here, Lawrence did the research on previous measurements. Epistemic status: we're pretty confident of our work here, but haven't engaged in a super thorough review process of all of it--this was more like a side-project than a core research project.]

How good are modern language models compared to humans, at the task language models are trained on (next token prediction on internet text)? While there are language-based tasks that you can construct where humans can make a next-token prediction better than any language model, we aren't aware of any apples-to-apples comparisons on non-handcrafted datasets. To answer this question, we performed a few experiments comparing humans to language models on next-token prediction on OpenWebText.

Contrary to some previous claims, we found that humans seem to be consistently worse at next-token prediction (in terms of both top-1 accuracy and perplexity) than even small models like Fairseq-125M, a 12-layer transformer roughly the size and quality of GPT-1. That is, even small language models are "superhuman" at predicting the next token. That being said, it seems plausible that humans can consistently beat the smaller 2017-era models (though not modern models) with a few hours more practice and strategizing. We conclude by discussing some of our takeaways from this result.

We're not claiming that this result is completely novel or surprising. For example, FactorialCode makes a similar claim as an answer on

this LessWrong post about this question

. We've also heard from some NLP people that the superiority of LMs to humans for next-token prediction is widely acknowledged in NLP. However, we've seen incorrect claims to the contrary on the internet, and as far as we know there hasn't been a proper apples-to-apples comparison, so we believe there's some value to our results.

If you want to play with our website, it’s

here

; in our opinion playing this game for half an hour gives you some useful perspective on what it’s like to be a language model.

Previous claims and related work

One commonly cited source for...
Introducing Pastcasting: A tool for forecasting practice
by Sage Future — 2022-08-11T17:38:06.474Z

TL;DR: Visit

pastcasting.com

to forecast on already resolved questions you don’t have prior knowledge about to get quick feedback on how you’re doing.

We may try to coordinate multiplayer sessions in our Discord

Motivation

We want to make it easy for people to get better at forecasting. Existing tools are not well optimized for practicing relevant skills.

Forecasting platforms and betting markets[1] have slow feedback loops between predictions and question resolution. The questions with shorter time horizons often feel less important and may be systematically different than those with longer ones. Their scoring systems also incentivize constantly keeping predictions up to date and often heavily reward being the first to react to news
Calibration training[2] isolates the skill of intuiting probabilities and confidence intervals but doesn’t help with other aspects of forecasting (choosing reference classes & base rates, trend extrapolation, coming up with considerations, investigating different views, and determining the trustworthiness of news sources). Additionally, there is usually no reference point to compare your accuracy against.

Pastcasting

With pastcasting, you can:

Forecast on already resolved questions from a vantage point further in the past
Use our filtered search engine (“Vantage Search”) to look up relevant information without accidentally revealing the answer
Receive immediate feedback on your forecasts and get scored against the crowd
Host friendly multiplayer competitions where you and your friends can simultaneously pastcast on the same questions and see who does best

How it works

Question sources

Our questions are currently pulled from resolved

Metaculus

and

GJOpen

questions.

Scoring

We use relative

log scoring

against the original crowd forecast at that time. This means that you will receive zero points if you submit the same value as the crowd. The scoring rule is also strictly proper, meaning that your expected score is maximized if you report your true beliefs.

Vantage Search

Our preliminary results (in yellow) come from a search api with a restricted date range up to the vantage date. To further reduce information leakage (from the website changing its contents), we then pass the results through the internet archive api (in blue). This tends to be much slower, so some pastcasters opt to use the preliminary results direc...
Pendulums, Policy-Level Decisionmaking, Saving State
by CFAR!Duncan — 2022-08-11T16:47:58.298Z

Pendulums

"Remember back when we used to talk about pendulums too little?"

One excellent, quick-and-dirty models of social change is the mental image of a pendulum swinging back and forth around a central resting point.

Imagine that the pendulum is “stuck” at some point. For instance, in many parts of America in the early 1900s, marriages were viewed as essentially inviolable, and divorce was tantamount to social suicide.

Eventually, people began to realize that there was something bad about this—for instance, people being stuck forever in loveless marriages that they entered into with very little information when they were teenagers—and they agitated to "push" the pendulum to a new set point.

Generally speaking, that new point is better than the original one. It’s less distant from the ideal. It contains less total badness overall (or at least, we hope so).

But that new stuck point comes with its own problems. For instance, maybe we’ve traded “lots of people trapped in marriages that are net-negative” for “lots of people who never reap the benefits of what would have been strongly net-positive relationships, because they were implicitly encouraged to bail early on when they hit the first obstacle or stumbling block.”

The latter problem is clearly smaller, and is probably a better problem to have as an individual! But it’s nevertheless clear that the loosening of the absoluteness of marriage has negative effects in addition to the positive ones.

This is true for almost every sort of change. Very rarely is a total and costless improvement possible (though you should still check, à la Goal Factoring). Most of the time, the best we can hope to do is to exchange one set of problems for a less serious set of problems, and then do the same thing again later.

(Caveat: don’t let the simplicity of the model obscure the complexities of real life … in reality, the “pendulum” isn’t just swinging back and forth, or even around in spirals, but on a whole bunch of different dimensions all at once. It's usually not just “this one thing got better, but meanwhile this other thing got worse” or “this one problem got solved, but meanwhile this new problem got created.” There are usually many ways in which a given shift is better, and many ways in which it’s still bad or newly problematic. For instance, the changes in marriage norms over the past eight...
Covid 8/11/22: The End Is Never The End
by Zvi — 2022-08-11T16:20:01.654Z

On Tuesday morning as I was grabbing croissants, the man in front of me turned around and told me to back away. ‘Because of the pandemic.’ He was the only one in the bakery wearing a mask, yet in every way he gave off the clear vibe that I, not he, was the one being unreasonable. There was a time he would have been right. There was even a time I was the person being crowded and upset about it.

Now the pandemic, for this kind of purpose, is over. Now I had to tell myself to let the Wookie win. There was space behind me, in no way was it Worth It to do anything but put my hands up and walk away.

I remember it because it is in such sharp contrast to the everyone and everything else. Covid over. People sick sometimes, sure, and yes I missed a 1-on-1 meeting because of someone’s Covid exposure. Eric Topol continues to warn us how bad things are and not to get numb, despite it being way too late for that. Still.

I was going to say that this week marks the end of the Covid posts being majority Covid content. Then it turned out that this week’s is still largely Covid content, as it snuck up on us with a lot of content this week (none of which is alarming). So the point of transition isn’t quite there, but we will get there soon. I predict slash hope there will only be a handful more of these posts with this high a ratio of Covid to non-Covid news.

The biggest development this week in things-relevant-to-my-interests was that PredictIt’s no action letter was revoked, effectively forcing it to shut down its markets in February 2023 unless something changes by then. The most credible story is that this is new prediction market website Kalshi performing a hit job to take out the competition, but I have not had the time to look into it yet. Whether or not that was the reason, the shut-down is quite bad, speaks quite badly of everyone who let this happen, and is something worth fighting against if there is a practical way to do that. PredictIt was far from perfect, but it was a dim light in a dark world.

Executive Summary

Reports from people on their quest to get Paxlovid.
PredictIt was shut down, will cover that at some point.
Monkeypox growth slowing worldwide but not in America.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 750k cases (+3%) and 3,000 deaths (-2%)

Results: 690k cases (-6%) and 3,208 deaths (+5%).

Prediction for n...
Thoughts on the good regulator theorem
by JonasMoss — 2022-08-11T12:08:28.897Z

I just became aware of the good regulator theorem through John Wentworth's post.
Then I tried to read the paper he talked about, Every Good Regulator Of A System Must Be A Model Of That System. This is among the worst papers I've ever skimmed. It's barely comprehensible. Reading this paper is not like reading a regular text. I had to try to read their intentions instead.

This post is my attempt to make sense of the setup and the results of the paper. Summed up, my conclusions are:

Their main theorem is correct but trivial and not very interesting. (It can be strengthened too, see the proposition below.) Their proof is a mess I haven't bothered looking at, it's way too complex.

Their interesting conclusion, that an optimal action can only be made if you have a model of the system (i.e., everything outside your control), is false.

Some thoughts about terminology:

It's definition of "most simple" probably doesn't make sense.

The definition of "isomorphisms" at least makes sense, but shouldn't have been called "isomorphism".

Setup of the paper

I don't follow the notation or terminology in the paper as it can be replaced with more familiar notation from decision theory.

Define the sets (the name in the paper in parenthesis)
Ω(D):the domain of inputs, can be thought of as a probability space,X(S):everything outside our control,A(R):the set of actions we can take.Y(Z):the final output, combining X and a, defined below.
Define the functions
X(ϕ):Ω→X,a random variable we don't control,a(ρ):Ω→A,our action, choosen by us,ψ(ψ):X×A→Y,the outcome space.
For each uncontrollable outcome x we have a non-empty set G(x) describing the good outcomes. Two examples are:

Goal sets. Let G⊆Y be a goal set, a set of outcomes that are good, independently of the value of x. Assume that for all x there is an action a so that ψ(x,a)∈G. Then G(x)={(x,a)∣ψ(x,a)∈G} is non-empty for all x. This is what they do in the beginning of their paper.

Utility maximization. Let u be a utility function and assume that mina∈Au(ψ(a,x)) exists for all x. Define Gu(x)={(x,a)∣a∈maxa∈Au(ψ(x,a))}. In the paper they talk about minimizing entropy specifically, i.e., G(x)={(x,a)∣a∈maxa∈A−Entropy(ψ(x,a))}.

Note: The authors don't use G(x); they only use the entropy.

Our goal is to find an optimal action function a that satisfies ψ(X(ω),a(ω))∈G(X(ω)),for all ω. In other words, we want to ma...
How and why to turn everything into audio
by KatWoods — 2022-08-11T08:55:55.871Z

If you love podcasts and audiobooks and find yourself occasionally facing that particular nerd-torture of discovering that an obscure book isn’t available on Audible, read on.

I’m kind of obsessed with listening to content (hence building

the Nonlinear Library

), and there are easy ways to turn pretty much all reading materials into audio, including most books and even blog posts, like LessWrong.

In this post I’ll share my system to turn everything into audio and my rationale for people who haven’t yet discovered the joys of reading with your ears.

If you’re already sold on listening to everything, skip to the section “Apps I use and recommend” for the practical nitty-gritty of how to turn everything into audio.

Read while doing other things

Have you ever reluctantly dragged yourself away from a really engaging EA post so that you could make dinner or drive to work? Have you ever procrastinated on doing the household chores because what you’re reading is so much more exciting and probably higher impact too, now that you think about it? If you have an audio version, you don’t have to choose between reading and chores: you can keep reading as you cook or travel. It’s a way to actually

productively multi-task.

Generally, if you convert your books to audio, you can read at times when your mind is not occupied, but holding a physical book would be inconvenient. For example, you can read while you are:

Commuting to work
Exercising
Cleaning your house
Running errands
Traveling
Cooking
Brushing your teeth
Showering (yes, I do do this. And yes, it is awesome)

With audiobooks, you can consume content almost all the time, if you want. This is great if you’re a bibliophile whose bookshelves and internet browsers are overflowing with enticing unread material: you can get through more content and

make the more mundane parts of your day more interesting

.

Listen and read at the same time

A little-known life hack: you can read a book with your eyes and listen to the audio version at the same time. It’s called immersive reading or two-channel reading. I find that this requires much less energy than reading on its own. It’s more like a movie, where if your attention flags, it’ll draw your mind back in. There’s also something about engaging more of your senses. When I do this, I can stay focused for longer and reading feels more relaxing. So, if you’d like to read but are feeling tired or are

st

...
Shard Theory: An Overview
by David Udell — 2022-08-11T05:44:52.852Z

Generated as part of SERI MATS, Team Shard's research, under John Wentworth.

Many thanks to Quintin Pope, Alex Turner, Charles Foster, Steve Byrnes, and Logan Smith for feedback, and to everyone else I've discussed this with recently! All mistakes are my own.

Team Shard, courtesy of Garrett Baker and DALL-E 2

Introduction

Shard theory is a research program aimed at explaining the systematic relationships between the reinforcement schedules and learned values of reinforcement-learning agents. It consists of a basic ontology of reinforcement learners, their internal computations, and their relationship to their environment. It makes several predictions about a range of RL systems, both RL models and humans. Indeed, shard theory can be thought of as simply applying the modern ML lens to the question of value learning under reinforcement in artificial and natural neural networks!

Some of shard theory's confident predictions can be tested immediately in modern RL agents. Less confident predictions about i.i.d.-trained language models can also be tested now. Shard theory also has numerous retrodictions about human psychological phenomena that are otherwise mysterious from only the viewpoint of EU maximization, with no further substantive mechanistic account of human learned values. Finally, shard theory fails some retrodictions in humans; on further inspection, these lingering confusions might well falsify the theory.

If shard theory captures the essential dynamic relating reinforcement schedules and learned values, then we'll be able to carry out a steady stream of further experiments yielding a lot of information about how to reliably instill more of the values we want in our RL agents and fewer of those we don't. Shard theory's framework implies that alignment success is substantially continuous, and that even very limited alignment successes can still mean enormous quantities of value preserved for future humanity's ends. If shard theory is true, then further shard science will progressively yield better and better alignment results.

The remainder of this post will be an overview of the basic claims of shard theory. Future posts will detail experiments and preregister predictions, and look at the balance of existing evidence for and against shard theory from humans.

Reinforcement Strengthens Select Computations

A reinforcement learner is an ML model trained via a reinf...
Do advancements in Decision Theory point towards moral absolutism?
by Nathan1123 — 2022-08-11T00:59:07.247Z

From my understanding reading through the site, each successive decision theory builds off of previous models in an upward progression: CDT/EDT is surpassed by the "winning" paradigm of TDT, which in turn is surpassed by the updateless paradigm of UDT/FDT. And the goal that these decision theories move towards is one that enables an agent to "understand, agrees with, and deeply believe in human morality" (from Superintelligence FAQ). In other words, creating a mathematical model that correctly reproduces the same impetus of human morality.

With that in mind, I start to notice certain trends across the advancements in Decision Theory that reflects what kind of ideology a truly moral AGI would use. CDT, the most straightforward and naïve decision theory, is unaligned in a way that appears to be highly amoral. In Parfit's Hitchhiker, for example, our intuition says that rewarding someone who has gone out of their way to help you is the morally right thing to do. But in CDT, as soon as the agent obtains what it wants (getting home) then it has no incentive to pay.

Any kind of a precommitment like a contract or a constitution would be meaningless to a CDT agent, because as soon as the agent is in a position to achieve the highest utility by violating the contract then it will do so (assuming it isn't under immediate threat of punishment). CDT essentially seeks its own immediate gain (maximize utility of its actions) with little regard towards others.

But precommitting is the hallmark of timelessness, so TDT has the special property of being able to keep a promise, because fulfilling its promises in the future will be able to achieve the best result in the present. This aligns a lot closer with a human concept of morality, as a TDT agent will faithfully hold to its word even in situations that doesn't give it an immediate benefit, because it is factoring in the long-term benefit of cooperation (thinking back to Parfit's Hitchhiker).

In a way, TDT also aligns with our concept moral codes and punishment. Even though punishing an agent doesn't casually affect the actions it has already done, it does deter them from committing those actions in the future, and knowing about the punishment a priori would deter the agent from considering the action in the first place (akin to Acasual Blackmail).

Adding in updatelessness takes another step closer towards human morality. I foun...