trentmkelly/LessWrong-43k huggingface dataset:
Confidence Levels in Forecasts and Psychological Surveys
by T431 — 2022-03-31T02:54:45.364Z

Epistemic Status: I am very uncertain about this topic, but think that, if anything, I may simply be mislabeling or misunderstanding some real signal.

I imagine that there may be aspects of a person's particular tendencies in judgmental forecasting that reveal something about their personality, such as their OCEAN (Big Five) measures or their Moral Foundations.

The following sequence of thoughts - "My life experiences indicate that I may be more conscientious than my OCEAN results suggest", "I remember selecting neutral for many answers that were likely related to conscientiousness", "One reason for why I selected neutral so often might be that I am underconfident about aspects of my personality, provided I have a track record of being underconfident, especially in judgmental forecasting" - led me to the question:

Could measures of overconfidence or underconfidence in judgmental forecasting be used to control for the effects of psychological survey participants gravitating away from or towards, respectively, the "neutral" option in surveys?

Relatedly, one might ask "Is over- or under- selecting the neutral option in psychological surveys an actual phenomenon?", "What occurs when there is no neutral option on a psychological survey versus when there is?", "How does under- or over- confidence in judgmental forecasting relate to under- or over- confidence in other facets of people's lives?

More generally, I suppose this intrigue might fall into the wider category of "reflections of cognitive biases or personality present in people's forecasts and forecasting tendencies". I would be content if people shared with me their thoughts on my above questions, or if people shared any resources they think would point me in a good learning direction (e.g., recommend I look into some line of research or work adjacent to these ideas). Thank you.
A bicycle for your memory
by sortega — 2022-03-30T21:41:07.046Z

Cross posted from sortega.github.io

Computers have the need to store huge amounts of information and, at the same time, being able to access that memory at blazingly fast rates. This poses the problem of how to get a fast and large memory when hardware memories are either fast, like registers or L1 cache lines; or big, like hard-drives or solid state drives.

Memory	Size	Speed	Comment
CPU register	A few bytes	One CPU cycle (200-300 picoseconds)	A register holds the data the CPU can operate on
RAM memory	A few gigabytes	100 nanoseconds	Applications need their data to be here to be responsive
Hard drive	Many gigabytes	150 microseconds	All your applications and data being kept locally have a copy here
Internet	Infinite (for all purposes)	10-40 milliseconds	Technically other people's hard drives

This is solved in computer architecture by the interplay of two ideas: a memory hierarchy and virtual memory. The memory hierarchy consists of a pyramid with the fastest and smallest memories on the tip and slower but bigger ones at the base. This arrangement allows you to have a complete copy of the information at the base, which is the biggest. Then, you can keep copies of the actively used information closer to the pyramid apex for better performance. Virtual memory is the illusion of having a single memory space which is the result of the collaboration of specific hardware (MMU, CPU features) with the operative system to transparently move data across the levels of the pyramid. The result is that, for most purposes and most of the time, programs running in your computer get to use a single memory space that feels as big as the base of the pyramid and almost as fast as its top.

Simplified memory hierarchy

Apart from computers, humans are rich information processors[1] and memory plays a paramount role in what it means to be a human. On one hand it's an important element of personal identity. Imagine a movie villain using technology or black magic[2] to swap the memories of two twin brothers. In some philosophical sense we are what we remember, and we will have a hard time deciding who is who in this contrived situation.

On the other hand, memories are needed for our minds to work because without them, we will have only the raw input of our senses and no other concept or idea to work with. The extent to which we can understand the world, communicate with others and act depends on usin...
ELK Computational Complexity: Three Levels of Difficulty
by abramdemski — 2022-03-30T20:56:37.239Z

ELK is described as a problem we wish to solve in the "worst case"; IE, for any potential failure case we can describe, we want to have a strong argument about why we won't run into that particular problem. The ELK document lists several cases we want to handle in that way. First in this list is the case where the correct translation is arbitrarily computationally complex.

However, this assumption implies that ELK is intractable. At best, we might hope to define some notion of approximate solution, and prove that there are efficient approximate solutions. So how are we supposed to solve ELK, if we are to assume that it's intractable?

The answer (at least, as I see it) is by arguing that the correct translation cannot be too computationally complex. (Or, failing that, by providing a strong argument that it is possible, at which point we would have to narrow the scope of ELK ambitions.)

A proposed algorithm for solving ELK which (1) has a strong argument for its correctness, and (2) is itself a tractable algorithm, is usually already an argument that correct translations are tractable. For example, a search procedure usually has to evaluate candidate translations by running them; so if we could prove that the search terminates in a reasonable amount of time, we've probably already proven that the solution it finds is tractable (at least in certain cases, EG, on the training data).

But this doesn't really answer the question. Since we don't already have such a solution, we may need to explicitly think about tractability in order to produce one.

So, in this post, I'm going to think about how we might possibly bound the computational complexity of ELK.

Level 3: Probabilistic Knowledge

The third level of difficulty for ELK (IE, the most computationally difficult that I'll discuss here) is the most conceptually simple to understand: we define the "correct translation" of a pattern of activity in the Predictor to be any and all things correlated with such activity.

In other words, we define the "meaning" of Predictor activity as just whatever we can predictably infer from it. In my previous post I formalized this by imagining probability distributions called "third person perspectives" which can be conditioned on the full execution trace of the Predictor, and infer as much as possible about the world from this information.

This seems intractable for two mai...
Abandoned prototype video game teaching elementary circuit theory
by Steven Byrnes — 2022-03-30T20:37:19.653Z

About a decade and several hobbies ago, I was inspired to make an educational video game that would teach the intuitions behind elementary circuit theory (batteries, switches, diodes, capacitors, inductors, etc.).

After an inordinate amount of time,[1] I wound up with this prototype browser game:

Click here to play the game

Its biggest limitations (i.e., the biggest reasons that I'm calling it a “prototype”) are (1) I only got around to making five “levels” (i.e., puzzles to solve), (2) it's not very user-friendly.[2]

Still, I gave it to my 7-year-old kid last night, and he was really enjoying it and clearly getting something out of it. That made me feel sad that it’s been sitting on my hard drive for a decade, and (on current trends) always will be.

So, if anyone wants to take this thing and develop it further, let me know, and maybe I’ll send you the code repository and give you permission to do with it as you please. (For example, if you commercialize it, you can keep all the profits.)

Meanwhile, enjoy. If you have nothing better to do with your time than to try to solve the puzzles, then please use spoiler-hiding as appropriate in the comment section.

^

This was 99.99% a solo project, but thanks to Michael Prettyman for some help and advice on the art assets. All remaining ugliness is fully my responsibility!

^

In a well-designed game, the user naturally and effortlessly learns the game interface, because it's presented bit by bit, with good affordances and/or tutorials, and user-tested, etc. I didn't get around to doing any of that, sorry. I think adults would be able to get the interface relatively quickly, especially if they happen to already know some elementary circuit theory (ironically). Younger kids might need a bit of help.
Meta wants to use AI to write Wikipedia articles; I am Nervous™
by Yitz — 2022-03-30T19:05:44.735Z

Meta is experimenting with using AI to write Wikipedia articles: https://ai.facebook.com/research/publications/generating-full-length-wikipedia-biographies-the-impact-of-gender-bias-on-the-retrieval-based-generation-of-women-biographies

I personally have a very bad feeling about this. I’m most afraid of this making it easier to spam the encyclopedia with fake information that looks plausible on the surface, and therefore doesn't get fully fact-checked. This could also create perverse incentives where SEO companies put out false information online to bias Meta's algorithm, and thereby sneak their way in to the encyclopedia. The decision to make this open source seems incredibly foolish to me as well, considering how easily a service like this could be misused. (Edit: It has been pointed out to me that if they made it closed-source that wouldn't be great either, since then we would have no idea what it was doing under the hood. Either way I wouldn't be happy, so I'm not sure their choice counts as a point against them.)

Am I overreacting? Is this actually a good thing? Is this actually way worse than I think it is? Who knows!

What are your thoughts on this?
No, EDT Did Not Get It Right All Along: Why the Coin Flip Creation Problem Is Irrelevant
by Heighn — 2022-03-30T18:41:20.099Z

Back in 2017, Johannes_Treutlein published a post critiquing logical decision theories: Did EDT get it right all along? Introducing yet another medical Newcomb problem. In it, Treutlein presents the Coin Flip Creation problem (and a second version) and argues logical decision theories (like Updateless Decision Theory (UDT) and Functional Decision Theory (FDT)) handle it wrong. After reading the post, it seems to me Treutlein's argumentation is flawed, and while I am probably not the first to notice this (or even write about it), I still think it's important to discuss this, as I am afraid more people make the same mistake.

Note that I will be talking mostly about how FDT handles the problems Treutlein presents, as this is a theory I have some expertise on.

The Coin Flip Creation Problem

From the original post:

One day, while pondering the merits and demerits of different acausal decision theories, you’re visited by Omega, a being assumed to possess flawless powers of prediction and absolute trustworthiness. You’re presented with Newcomb’s paradox, but with one additional caveat: Omega informs you that you weren’t born like a normal human being, but were instead created by Omega. On the day you were born, Omega flipped a coin: If it came up heads, Omega created you in such a way that you would one-box when presented with the Coin Flip Creation problem, and it put $1 million in box A. If the coin came up tails, you were created such that you’d two-box, and Omega didn’t put any money in box A. We don’t know how Omega made sure what your decision would be. For all we know, it may have inserted either CDT or EDT into your source code, or even just added one hard-coded decision rule on top of your messy human brain. Do you choose both boxes, or only box A?

Treutlein claims EDT one-boxes and "gets it right". But I think it's wrong even to discuss what a decision theory would do in this problem: my claim is that this is not a proper decision theoretic problem. It's an interesting thought experiment, but it is of little value to decision theory. Why? Because the question

Do you choose both boxes, or only box A?

has two branches:

If Omega flipped heads, do you choose both boxes, or only box A?
If Omega flipped tails, do you choose both boxes, or only box A?

In both cases, the answer is already given in the problem statement. In case 1, Omega created you as a one-boxer, and in c...
How to Make Your Article Change People's Minds or Actions? (Spoiler: Do User Testing Like a Startup Would)
by Yonatan Cale — 2022-03-30T17:39:34.373Z

Are you writing an article that’s supposed to change people’s minds and/or actions? For example, an article about career advice, productivity, or getting people to apply for a job? Then this post is for you!

TL;DR: For important articles I recommend doing real user testing (even with just one person!). Try it and you’ll never go back.

I specifically recommend:

(1) Watch them read it, don’t just ask for Google Doc comments

Open a zoom, share screen, and ask them to share their thoughts live as they read.

(2) Show it to your target audience, don’t just ask others what your target audience would think

If your article is supposed to change developers’ career plans, ask a dev who might change their career plan. Don’t ask a biologist what a dev might think.

(3) Check if they changed their mind, not just if they like your writing

If they end with “I’ve got to do this now!”: That's a good sign.

If they say “sounds interesting, I’ll think about it”, that’s not as good.

I personally like to first ask the other person for their opinion (for example, “what’s your current career plan?”), then let them read the article, and then ask again.

The general intuition:

If you want to adopt my frame of mind, ask yourself: Would your idea of how to improve [the article’s persuasiveness] also improve [how engaging a startup’s app] is?

Here are some exercises, do you think these would work?

Let GPT-3 look at the app (or article) and tell you how engaging the app (or how persuasive the article) is
Try to guess in advance all the reasons why your target audience won’t agree with the article, or won’t be engaged with the app

Hint: Is the test [checking if a real user from your target audience changes their mind] or [using some proxy to guess whether a real user would change their mind] ?

How to start?

Easiest: post in

Bountied Rationality

, or in another group that has your target audience.

A draft post you can copy & paste: “Hey, I’ll pay $10 for someone in [describe your target audience here] to review a post of mine for the EA/lesswrong forum. I want to do it live, over a video call. The call is capped at 30 minutes (and expected to be less). Comment or message me if you’re interested. Tagging or referring relevant people is also appreciated.”

Thanks to Matt Brooks and Daniel Reeves for helping make this article better <3
Procedurally evaluating factual accuracy: a request for research
by Jacob_Hilton — 2022-03-30T16:37:37.675Z

I am grateful to Daniel Kokotajlo, Beth Barnes and John Schulman for feedback on this post.

The purpose of this post is to request research on the design of precise procedures for evaluating how factually accurate pieces of text are. This stands out to me as an area that is potentially valuable for reducing risks from advanced AI, while not requiring detailed knowledge of ML.

Problem statement

Suppose that you are given:

A real-world setting in which text appears, such as "Google search snippets", or "OpenAI API outputs", or "arXiv abstracts"
A collection of contexts in that setting, such as a "the search queries made in 2021", or "the questions from the ELI5 dataset", or "all existing papers on the arXiv"
A text distribution for each context, such as "actual current Google search snippets", or "responses produced by an ML model", or "the abstract a certain person would write after reading the paper"

The problem is to define a procedure that takes in a context and a piece of text from the given distribution, and outputs a numeric score for factual accuracy.

The procedure should have the following properties:

It is feasible for humans to carry out.
It is as unambiguous as possible: different people following the same procedure should end up with the same score, except perhaps for a minimal number of "irreducible" judgment calls.
It can be optimized to produce good outcomes: to the extent that factual accuracy is important in the given setting, selecting text with a high score is beneficial, compared to alternative procedures.

Included in this is the effect that knowledge of the procedure could have, e.g. people will have more trust in the text if they understand and endorse the procedure.
In practice, whether good outcomes are produced will depend on the method and extent of optimization (e.g., Goodhart's law will eventually kick in), and on criteria other than factual accuracy. This makes the problem statement a little fuzzy.

Note that the problem statement doesn't mention ML models (except as examples). The problem is of course motivated by ML models, but I think it can be studied relatively independently of ML, at least initially.

Motivation

The main motivation for this request is that it is a problem that arises very naturally when attempting to train

truthful LMs

. The most straightforward way to optimize the factual accuracy of a language m...
[ASoT] Some thoughts about LM monologue limitations and ELK
by leogao — 2022-03-30T14:26:15.381Z

Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to Kyle and Laria for discussion.

One potential way we might think to interpret LMs is to have them explain their thinking as a monologue or justification or train of thought or something. Particularly, by putting the explanation before the answer, we might hope to encourage the model to actually use the monologue to come to its conclusion and try to avoid the model coming up with the bottom line first. However, there are a bunch of ways this could go wrong. For instance:

Humans are really good at coming up with the bottom line first and then rationalizing a reasonable sounding process to arrive at that conclusion, so imitating humans encourages rationalization by default, even if the explanation is put temporally before the bottom line. Even though the model has uncertainty over the explanation, and in some sense the explanation determines the answer, the problem is that the distribution over the explanations may not be determined by the validity of the explanation, but rather be an integral over possible bottom lines.
It's not sufficient for the answer to merely depend on the explanation: the model could be storing arbitrary data in the explanation in a way that doesn't line up with the meaning of the words. It could also be using the additional tokens to do more thinking in the latents but not in the tokens, etc.
If the model is deceptive and we use the monologues to try and tell if the model is deceptive and apply any optimization pressure against that whatsoever (this includes rejecting actions/models based on it), then it would actually be incentivized to not tell us about its scheming, because we would be punishing (blatant monologue, evil action) more than (benign monologue, evil action).
This also has a reduction to ELK: if the model's abstractions are sufficiently different from human abstractions, it may actually be more work for the model to translate its actual thought process to human understandable text than to just simulate the rationalization process a human would go through. To make the connection to ELK more clear, we can think of the reporter as the circuit that converts the model's actual thought process to text (or simulates the human tho

...
[Intro to brain-like-AGI safety] 10. The alignment problem
by Steven Byrnes — 2022-03-30T13:24:33.181Z

(Last revised: April 2025. See changelog at the bottom.)

10.1 Post summary / Table of contents

Part of the

“Intro to brain-like-AGI safety” post series

.

In this post, I discuss the alignment problem for brain-like AGIs—i.e., the problem of making an AGI that’s trying to do some particular thing that the AGI designers had intended for it to be trying to do.

The alignment problem is (I claim) the lion’s share of the technical AGI safety problem. I won’t defend that claim here—I’ll push it off to the next post, which will cover exactly how AGI safety is related to AGI alignment, including the edge-cases where they come apart.[1]

This post is about the alignment problem, not its solution. What are the barriers to solving the alignment problem? Why do straightforward, naïve approaches seem to be insufficient? And then I’ll talk about possible solution approaches in later posts. (Spoiler: nobody knows how to solve the alignment problem, and neither do I.)

Table of contents

In Section 10.2, I’ll define “inner alignment” and “outer alignment” in the context of our brain-like-AGI motivation system. To oversimplify a bit:

If you prefer neuroscience terminology: “Outer alignment” entails having “innate drives” (cf. Post #3, Section 3.4.2) that fire in a way that reflects how well the AGI is following the designer’s intentions. “Inner alignment” is a situation where an imagined plan (built out of concepts, a.k.a. latent variables, in the AGI’s world-model) has a valence that accurately reflects the innate drives which would be triggered by that plan.
If you prefer reinforcement learning terminology: “Outer alignment” entails having a ground-truth reward function that spits out rewards that agree with what we want. “Inner alignment” is having a learned value function that estimates the value of a plan in a way that agrees with its eventual reward.
In the AGI alignment literature, failures of outer alignment are also sometimes called “specification gaming”, while failures of inner alignment are also sometimes called “goal misgeneralization”.

In Section 10.3, I’ll talk about two key issues that make alignment hard in general (whether “inner” or “outer”):

The first is “Goodhart’s Law”, which implies that an AGI whose motivation is just a bit off from what we intended can nevertheless lead to outcomes that are wildly different from what we intended.
The second is

...