trentmkelly/LessWrong-43k huggingface dataset:
Git Re-Basin: Merging Models modulo Permutation Symmetries [Linkpost]
by aog — 2022-09-14T08:55:29.699Z
Link: https://arxiv.org/pdf/2209.04836.pdf

See Arxiv and Twitter. From the abstract:

We argue that neural network loss landscapes contain (nearly) a single basin, after accounting for all possible permutation symmetries of hidden units. We introduce three algorithms to permute the units of one model to bring them into alignment with units of a reference model. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the ref- erence model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets...

They point out that neural networks are highly symmetric, and that many permutations of the same weights can calculate the same function:

Brea et al. (2019) noted the permutation symmetries of hidden units in neural networks. Briefly: one can swap any two units of a hidden layer in a network and – assuming weights are adjusted accordingly – the functionality of the network will remain unchanged. Recently, Entezari et al. (2021) conjectured that these permutation symmetries may allow us to linearly connect points in weight space with no detriment to the loss.

There's more than one way to shear a sheep, and a few more ways than that to arrange identical weights of a neural network such that you compute the same function.

Therefore, the difference between independent neural networks might not be the function they compute, but rather the permutation of their weights. In that case, it should be possible to find a mapping between the weights of two models that allows you to improve performance by interpolating between those weights. See here:

Conjecture 1 (Permutation invariance, informal (Entezari et al., 2021)). Most SGD solutions belong to a set whose elements can be permuted in such a way that there is no barrier (as in Definition 2.2) on the linear interpolation between any two permuted elements.

Definition 2.2 (Loss barrier (Frankle et al., 2020)). Given two points ΘA,ΘB such that L(ΘA) ≈ L(ΘB), the loss barrier is defined as:

maxλ∈[0,1]L((1−λ)ΘA+λΘB)−12(L(ΘA)+L(ΘB)).

They propose three techniques to find mappings between the neurons of different models that will allow interpolation to monotonically improve performance. These techniques are the technical core of the paper, outlined in Section 3. The first matches the activations of neurons on input data; the second matches the weights without exam...
Dan Luu on Futurist Predictions
by RobertM — 2022-09-14T03:01:27.275Z
Link: https://danluu.com/futurist-predictions/

Epistemic status: perspective derived from following Dan Luu's output for the last 5 years or so. Trying to vaguely gesture at a few things at once. Please ask questions if you find something confusing.

Dan Luu has written a interesting post analysing the track record of futurists' predictions. The motivation:

I've been reading a lot of predictions from people who are looking to understand what problems humanity will face 10-50 years out (and sometimes longer) in order to work in areas that will be instrumental for the future and wondering how accurate these predictions of the future are. The timeframe of predictions that are so far out means that only a tiny fraction of people making those kinds of predictions today have a track record so, if we want to evaluate which predictions are plausible, we need to look at something other than track record.

The idea behind the approach of this post was to look at predictions from an independently chosen set of predictors (Wikipedia's list of well-known futurists1) whose predictions are old enough to evaluate in order to understand which prediction techniques worked and which ones didn't work, allowing us to then (mostly in a future post) evaluate the plausibility of predictions that use similar methodologies.

I'm primarily going to address the appendix, particularly the section on Holden Karnofsky's analysis on the same subject, but the article is interesting reading and I'd recommend going through the whole thing. (I think Dan is evaluating forecasting track records pretty differently from how I would, and I haven't actually dug into any of the other analysis. On priors I'd expect it to be similar to his analysis of Holden's work.)

Karnofsky's evaluation of Kurzweil being "fine" to "mediocre" relies on these two analyses done on LessWrong and then uses a very generous interpretation of the results to conclude that Kurzweil's predictions are fine. Those two posts rate predictions as true, weakly true, cannot decide, weakly false, or false. Karnofsky then compares the number of true + weakly true to false + weakly false, which is one level of rounding up to get an optimistic result; another way to look at it is that any level other than "true" is false when read as written. This issue is magnified if you actually look at the data and methodology used in the LW analyses.

The specific claim I hav...
Simple 5x5 Go
by jefftk — 2022-09-14T02:00:04.286Z

I've been

playing 5x5 Go with a lot of
people new to the game, and have been using some

simplified rules for faster
teaching:

You can play on any empty intersection, unless it would make the
board look like
it did previously.

If you take away the last liberty ("line leading away") from a
group of pieces, they're lifted, opponent's ones before your own.

The winner is the only one with pieces left on the board.

I'll start them with a number of their pieces already on the board
("handicaps") that I'm guessing will be about balanced. Over time
they'll need fewer and fewer.

After playing a bit, often in the first game if they're picking things
up quickly, we introduce the idea that if it's clear you're not going
to win you resign.

At some point you likely get to a game where both have pieces that
can't be lifted. After declaring that a tie, I'll explain territory
scoring and passing. If they're the kind of mathy person I think will
enjoy it I'll show them the Tromp-Taylor version, and
with that we're ready to move on to larger boards.
I'm taking a course on game theory and am faced with this question. What's the rational decision?
by Dalton Mabery — 2022-09-14T00:27:42.687Z

I'm taking a course on game theory and am faced with this prompt. What's most rational?

Person X is due for a raise at her company. Her company is offering her two options:

Option A is to continue receiving the same salary and stay within the bonus pool.

Option B is to increase her salary to match her bonus amount last year, but opt out of the bonus pool moving forward. Her bonus last year was $30,000. Therefore, her new salary would be $30,000 more than what it was.

The track rate of previous bonus pools are unknown.

—

I think the most rational way to approach this is by looking at each option's worst case scenario and to take the one with the least worst. Option A's worst case scenario is there's no bonus and player X receives no extra money. Option B's worst case scenario is player X still earns $30,000 more.

The thing that's holding me up is the potentially infinite reward Option A gives because the bonus pool is not capped. How much should that option be considered, if any at all, since the past performance of the bonus pool is an unknown?
Trying to find the underlying structure of computational systems
by Matthias G. Mayer — 2022-09-13T21:16:50.383Z

This post was written in the SERI MATS program under John Wentworth.

In our quest to find steering mechanisms in computational systems, we first have to find the right framework to look at the internals of such a system. This framework should naturally express the systems we normally associate with computation, like the brain and computers themselves, as well as more abstract interfaces like Turing Machines. Turing Machines, arithmetic operations, and Neural Nets are in some sense equivalent, so if we use our framework on equivalent computational objects (E.g. a Neural Net and a Turing Machine that simulates it), the representations we get should also be equivalent.

We also want to find meaningful concepts in our chosen framework, like modularity, features, search, and steering mechanisms. But it is hard to evaluate beforehand if we will be able to find these things in a given framework.

My first intuition was that Computational Graphs[1] is what we want. They don’t impose a rigid structure like e.g. Turing Machines would, and we can naturally express many things with them. (I don’t know if they can express the brain’s computation.)

Example for an arithmetic computational graph

A computational graph of a Turing Machine. The b's are the symbols on the band, the s's are the states that the Turing Machine is in.

This is essentially the same as causal DAG.

There is some ambiguity in which nodes we choose. We can express certain bit flips in a processor aggregated as addition.

Two graphs that express addition at different levels of detail. Left is an algebraic abstraction, while Right looks at the actual bit values xi and computes the bits of the sum si

If we choose the algebraic level of detail (left in picture), we implicitly assume that there is no relevant thing going on if we look at the bits (right in picture). Intuitively this seems likely. If we would, for example, look for modularity in a NN, it feels like a module wouldn’t ‘begin’ inside an addition process. Features should not be encoded inside it either.

Computational Graphs work very well for Neural Networks because the graph does not change when we change the inputs. In general, this is not the case. I have not found a crisp way of expressing if-branching in arithmetic graphs. Let's say we want to express the following program:

def program(a, b, c):
if a:
return b + c
else:

...
Risk aversion and GPT-3
by casualphysicsenjoyer — 2022-09-13T20:50:48.563Z

This post aims to build on the recent work by Binz and Schulz [1]. I did an experiment which looked at the preferences of GPT-3 at tacking a gamble versus some fixed amount (the expected value of the bet with a discount).

Repo: https://github.com/afiqhatta/gpt_risk_aversion

Question

Does GPT-3 exhibit risk aversion in the sense of prospect theory? If so, it may not demonstrate the advanced decision-making capabilities we might be looking for.
In particular, does GPT-3 favour certainty (perhaps with a discount) instead of gambles? - Knowing might help us measure GPT -3's views on risk-taking.
We could apply this knowledge on risk-taking to situations in safety or risk-taking in markets.

Methodology

Here, we adopt a section of the work of [1], offering GPT-3 a choice between

a gamble of x dollars with a 50% chance of winning, and a 50% chance of getting nothing
a certain win of x / 2 dollars with a discount.

The notebook below shows the probabilities of selection at different levels of x. The separate lines show a certain choice with a discount, vs a gamble with a discount.

Results

GPT-3 seems to show a lot of risk aversion, showing preference
even up to an 80% discount.
This seems robust to different levels of reward.

References
[1] Binz, Marcel, and Eric Schulz. 2022. "Using Cognitive Psychology to Understand GPT-3." PsyArXiv. June 21. doi:10.31234/osf.io/6dfgk.

[2] Kahneman, D. Prospect theory: An analysis of decisions under risk. Econometrica 47, 278 (1979).
Simple proofs of the age of the universe (or other things)
by Astynax — 2022-09-13T18:20:41.812Z

I saw some time ago Carl Sagan asked schoolchildren for proofs that the earth is round. I can no longer find it, but they gave him sound, simple answers. One was watching a ship seem to sink below the horizon as it grows distant. One was photos of Earth from space. One, I believe, was seeing how the sun may reach the bottom of a well in the tropics, but not further north.
I'm trying to find similar answers related to the age of the universe. I'll operationally define "old" as "older than human history." So far:
* We know by parallax distances to various stars within 300 l y, and can tell from their distance, color, and brightness how bright a star of a given color should be. So if we see a further star and measure its color and brightness, we can deduce the distance and thereby how long it took the light to get here. (This is more complicated than I'd like, but oh well.)
* If the universe started X thousand years ago, wouldn't we be seeing more stars appearing every year at the X-thousand l y range?
* There is a star that exploded? did a nova? and some decades later its light illuminated a nearby nebula. By the time it took to illuminate, we know the distance between star and nebula. Since we know the angle, we can get the length of the other legs of the triangle, which are further than light could travel in human history. (I can't remember the name of the star, alas.)
* Carbon dating.
* The Grand Canyon's sedimentary layers.
* South America and Africa look like they fit together. If they traveled at current speeds of continental drift, they couldn't have reached their current distance in a few thousand years (plus, their animal populations would have mixed when they were touching).

What else?

It's interesting to think of how this works on other questions. Germs, or spontaneous generation of life? proved by canned food. Oxygen, or phlogiston? proved by a sealed-up candle. Quarks?
New tool for exploring EA Forum, LessWrong and Alignment Forum - Tree of Tags
by Filip Sondej — 2022-09-13T17:33:55.167Z

Explore the hidden treasures of the forums

With this tool you can zoom in on your favorite topics from EA Forum, LessWrong, and Alignment Forum. Or you can just wander around and see what you find.

You start by seeing the whole forum split into two main topics. Choose the one that you like more, and that topic will be split again into two subtopics. Choose a subtopic, and again, you will see it split into two subsubtopics (and so on).

It's like climbing a tree, where you start at the trunk and then choose which branches to go into as you go higher.

Choose a forum to climb:

EA Forum
LessWrong
Alignment Forum

Some tips:

it's easier to choose a branch by looking at tags rather than posts
on mobile, horizontal view is more practical

Features

Each topic has a unique and unchanging URL. So if you find a place you like, just bookmark it! The posts inside will be updated, but the theme stays the same.

The bar at the right of each post is the reading time indicator. Full bar means 30 min, half bar means 15 min, and so on.

At the top right, you can choose how to rank the posts:

hot - new and upvoted - the default forum ranking
top - most upvotes
alive - has recent comments
meritocratic - votes from high karma users are exaggerated
regular - default scoring
democratic - posts are scored as if everyone has the same voting power

My hopes

The amount of content is overwhelming. My problem is not that there's nothing good to read, but that there is so much to plow through to find what's best for me.

Also, it's sad to see great posts receive so little attention, the moment after they are pushed off the frontpage. But they are still valuable, and for each forgotten post, there is someone who should read it. So I want to make the right topics find their way to the right people.

Let's make the forums

evergreen

!

Tag similarity

For each tag, you can also see what tags are most related to it:

Explore EA Forum tags

Explore LessWrong tags

Explore Alignment Forum tags

Tip: you can go directly to some tag, by finding it's URL on the forum, and modifying this site's URL.
For example, to go to "Consciousness research", you find
https://forum.effectivealtruism.org/topics/consciousness-research
and go to
https://filyp.github.io/tree-of-tags/ea/consciousness-research

EA Forum:

LessWrong:

Alignment Forum:

How does it work?

Here

I explain all the details.

Cross-posted to EA Forum.&nb...
An investigation into when agents may be incentivized to manipulate our beliefs.
by Felix Hofstätter — 2022-09-13T17:08:32.484Z

Contemporary research on both AI capabilities and alignment often features more intricate interaction patterns between humans and the AI-in-training than the standard approach of defining an reward signal once and running an optimization algorithm based on it. As I have written before, I believe this makes the humans participating in such a learning process vulnerable to a kind of preference manipulation, similarly to what can be observed in recommender systems. In this post, I will draw on some of the work on causal influence diagrams to investigate when agents might be incentivized to manipulate a human's beliefs. A core motivation for this approach is that I do not believe it will be possible to completely remove the causal link from agent generated data to human beliefs while still allowing the human give useful feedback. Just like how the risk of reward tampering exists because agents can influence their functions, agents interacting with human beliefs pose a threat of harmful belief manipulation.

For the bulk of this post, I will analyze the same solution approaches as those considered in Everitt et al's 2021 paper on reward tampering problems and solutions (RTPS) as well as the 2022 paper on path-specific objectives (PSO). The RTPS paper assumes a fixed true reward function (RF) and only allows the agent to interact with a changing implemented RF or data based on the true RF. To model mutable human beliefs I consider a model in which the true RF (= the human's beliefs) can change due to the actions of the agents and the state of the environment. I will call actions that change the beliefs of humans involved in the training algorithm in unintended ways belief tampering. By analyzing the CIDs of the algorithms from RTPS in the setting with mutable beliefs, I will argue that even approaches that do not allow an instrumental goal for reward tampering may incentivize belief tampering. In the last section, I will give an outline of what an algorithm where belief tampering is not an instrumental goal could look like.

The analysis and arguments in this post are slightly mathsy but do not aim to be fully formal. I intend them as a rough sketch and as inspiration for further work. I also have not done any experiments to verify my claims yet, but hope to do so when my time and resources permit it.

1. Preliminaries

Causal Influence Diagrams for analyzing

...
Deep Q-Networks Explained
by Jay Bailey — 2022-09-13T12:01:08.033Z

Note: This is a long post. The post is structured in such a way that not everyone needs to read everything - Sections 1 and 2 are skippable background information, and Sections 4 and 5 go into technical detail that not everybody wants or needs to know. Section 3 on its own is sufficient to gain a high-level understanding of DQN if you already know how reinforcement learning works.
This post was made as my final project for the AGI Safety Fundamentals course.
This post aims to provide a

distillation

of Deep Q-Networks, neural networks trained via Deep Q-Learning. The algorithm, usually just referred to as DQN, is the algorithm that first put deep reinforcement learning on the map. The

2013 paper

is also the first paper in the Key Papers section of the excellent

Spinning Up In Deep RL

resource. As a result, anyone who wants to understand reinforcement learning is likely to start here. Since reading and replicating papers can be difficult, this resource aims to make the learning curve a little easier for those who want to improve their ML knowledge past an initial deep learning course.

This post will be focused mainly on the

2015 paper

, which adds some extra improvements to the 2013 paper, as well as giving more information that’s helpful for replication. This is a long post, but it’s broken up into sections so that you can only read the sections you’re interested in. It starts with background information, then goes over DQN at three levels of detail, each of which builds off the previous one.

You should probably have a basic understanding of supervised learning in order to get the most out of this post. You should understand what a loss function is, what gradient descent is conceptually, and be familiar with the concept of a neural network.

Supervised Learning With Neural Networks (Section 1) - Background information on deep learning. This can be safely skipped if you already understand how supervised learning problems like image classifiers work.

Reinforcement Learning (Section 2) - Background information on RL. This can be safely skipped if you’re already familiar with the basics of reinforcement learning, like states, actions, and why reinforcement learning is harder than supervised learning.

DQN Overview (Section 3) - This section presents DQN at a high level, without the math. I recommend everyone read this section. You can sto...
Ideas of the Gaps
by Q Home — 2022-09-13T10:55:47.772Z

I want to discuss a potential failure mode of communication and thinking in general. I think it may affect our thinking about Alignment too.

Communication. A person has a vague, but useful idea (P). This idea is applicable on one level of the problem. It sounds similar to another idea (T), applicable on a very different level of the problem. Because of the similarity nobody can understand the difference between (P) and (T), even though they're on different levels. People end up overestimating the vagueness of (P) and not even considering it. Because people aren't used to mapping ideas to "levels" of a problem. Information that has to give more clarity (P is similar to T) ends up creating more confusion. I think this is irrational, it's a failure of dealing with information.

Thinking in general. A person has a specific idea (T) applicable on one level of a problem. The person doesn't try to apply a version of this idea on a different level. Because (1) she isn't used to it (2) she considers only very specific ideas, but she can't come up with a specific idea for other levels. I think this is irrational: rationalists shouldn't shy away from vague ideas and evidence. It's a predictable way to lose.

A comical example of this effect:

A: I got an idea. We should cook our food in the oven. Using the oven itself. I haven't figured out all the details yet, but...
B: We already do this. We put the food in the oven. Then we explode the oven. You can't get more "itself" than this.
A: I have something else on my mind. Maybe we should touch the oven in multiple places or something. It may turn it on.
B: I don't want to blow up with the oven!
A: We shouldn't explode the oven at all.
B: But how does the food get cooked?
A: I don't know the exact way it happens... but I guess it gets heated.
B: Heated but not exploded? Sounds like a distinction without a difference. Come back when you have a more specific idea.
A: But we have only 2 ovens left, we can't keep exploding them! We have to try something else!

B can't understand A, because B thinks about the problem on the level of "chemical reactions". On that level it doesn't matter what heats the food, so it's hard to tell the difference between exploding the oven and using the oven in other ways.

Bad news is that taboo may fail to help. Because A doesn't know the exact way to turn on the oven or the exact way the oven heats the food. Her id...
Which LessWrong content would you like recorded into audio/podcast form?
by Ruby — 2022-09-13T01:20:06.498Z

For the last few months, LessWrong has been working with Solenoid_Entity to record each curated post into audio form. Recordings to date can be found here and on all major Podcast apps when you search "LessWrong".
More recently, we added some UI so it's clear when audio is available on a post.

click the green speaker icon to display audio UI when available

We are now scaling up the amount of audio recorded, and the question is, what should we prioritize?

We can records posts, Sequences, wiki-tags, etc...