trentmkelly/LessWrong-43k huggingface dataset:
Multi-Component Learning and S-Curves
by Adam Jermyn — 2022-11-30T01:37:05.630Z

(Thanks to Oliver Balfour, Ben Toner, and various MLAB participants for early investigations into S-curves. Thanks to Nate Thomas and Evan Hubinger for helpful comments.)

Introduction

Some machine learning tasks depend on just one component in a model. By this we mean that there is a single parameter or vector inside a model which determines the model’s performance on a task. An example of this is learning a scalar using gradient descent, which we might model with the loss function

L=12(a−~a)2

Here a is the target scalar and ~a is our model of that scalar. Because the loss gradients are linear gradient descent converges exponentially quickly, as we see below:

The same holds for learning a vector using gradient descent with the loss

L=12∑i(ai−~ai)2

because the loss is a sum of several terms, each of which only depends on a single component.

By contrast, some tasks depend on multiple components simultaneously. That is, the model will only perform well if multiple different parameters are near their optimal values simultaneously. Attention heads area a good example here: they only perform well if the key, query, value, and output matrices are all close to some target. If any of these are far off the whole structure fails to perform.

We think that such multi-component tasks are behind at least some occurrences of the commonly-seen S-curve motif, where the loss plateaus for an extended period before suddenly dropping. Below we provide toy examples of multi-component tasks, reason through why their losses exhibit S-curves, and explore how the structures of these S-curves vary with the ranks of the components involved. We additionally provide evidence for our explanations from numerical experiments.

Examples

Rank-1 learning

Suppose we’re using gradient descent on the regression task:

Aij=aibj~Aij=~ai~bjL=12∑ij(Aij−~Aij)2

Here there is a target matrix A which has rank 1, and which we model by an outer product of two vectors ~a and ~b.

The loss gradients are:

−∂L∂ai=a(b⋅~b)−~a~b2−∂L∂bj=b(a⋅~a)−~b~a2

If we write

∂~ai∂t=−∂L∂~ai

and likewise for ~b we get a non-linear ordinary differential equation (ODE) of the form

d~adt=a(b⋅~b)−~a~b2d~bdt=b(a⋅~a)−~b~a2

If the vectors are high-dimensional and we choose a random initialization then we approximately have a⊥~a and b⊥~b at early times. This means the above equations are...
EA & LW Forums Weekly Summary (14th Nov - 27th Nov 22')
by Zoe Williams — 2022-11-29T23:00:00.461Z

Supported by Rethink Priorities

This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'Effective Altruism Forum Podcast'.
Author's note: this week is a double-week, with an increased karma bar of 70. We'll be back on the regular schedule next week. FTX-related posts are also now separated into their own section, which you can see at the end of the post.

Top / Curated Readings

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all. Posts are picked by the summarizer, and don't reflect the forum 'curated' section.

First FDA Cultured Meat Safety Approval

by Ben_West

Linkpost for

this article

. “In a major first, the U.S. Food and Drug Administration just offered its safety blessing to a cultivated meat product startup. It completed its first pre-market consultation with Upside Foods to examine human food made from the cultured cells of animals, and it concluded that it had “no further questions” related to the way Upside is producing its chicken.”

Pre-Announcing the 2023 Open Philanthropy AI Worldviews Contest

by Jason Schukraft

Open Philanthropy will run an AI worldviews contest in early 2023. Prizes, judging, and other details will be different from the Future Fund competition, but they expect it will be easy to adapt entries for this contest.

Short Research Summary: Can insects feel pain? A review of the neural and behavioural evidence by Gibbons et al. 2022

by Meghan Barrett

A short summary of

Advances in Insect Physiology

by Gibbons et al. (2022), which summarizes >350 scientific studies to assess the scientific evidence for pain across six orders of insects. It finds strong or substantial evidence for pain in adult insects of five orders.

Trillions of insects are directly impacted by humans each year (farmed, managed, killed, etc.). Significant welfare concerns have been identified...
Distinguishing test from training
by So8res — 2022-11-29T21:41:19.872Z

Various people propose some variant of “align AGI by making it sufficiently uncertain about whether it’s in the real world versus still in training”. (The idea being that, insofar as it's in training, betraying the apparent-operators predictably gets high loss; so if the AGI is uncertain about which situation it's in, and assigns sufficiently high probability to "I'm still in training", it will always play it safe and avoid betraying its operators.)

This approach seems extremely unpromising to me. Setting aside the various reasons why the AGI kills you even if you manage to confuse it (such as an

inner alignment

failure, where the AGI turns out not to care about the predictably-high outer-loss from betraying the apparent-operators), let's consider why it's tricky to dupe a superintelligence into being uncertain about a fact that you yourself consider obvious.

Before reading the short list below, I recommend the following exercise. Suppose that some slow and individually-dumb aliens had a copy of your mind, and were putting you through a series of nested simulations. How would you tell when you finally made it to the outside?

.

.

.

.

.

.

.

A short list

(Note: As usual, treating this list as exhaustive is suicidal.)

1. Reality is the first place that's seriously large.

1.1. It's the first place large enough to contain a plausible explanation for how the AGI itself actually came to be.

1.2. It's the first place large enough to contain a plausible explanation for how the operators themselves came to be.

1.3. It's the first place with far more computing power than the AI itself, as can be readily verified by a variety of experiments (involving, say, fluid dynamics).

2. Reality is the first place where the AI's history may contain the signature of superintelligent optimization interacting with complex systems.

For example, suppose the AGI makes superintelligently-motivated trades in a stock market. 24 hours later, there are a billion additional human-intelligence trades made in response to it. The signature of a massive human-intelligence response to superintelligent-trades is implausibly hard for the operators to fake, even if generating a billion fake dumb trades is easy.

3. Reality is the first place that embeds the AGI and has laws of physics so simple that the hypothesis "this is being simulated ...
Progress links and tweets, 2022-11-29
by jasoncrawford — 2022-11-29T20:54:38.555Z
Link: https://rootsofprogress.org/links-and-tweets-2022-11-29

The Progress Forum

My AMA (ask me anything)
Are Technologies Inevitable? (Matt Clancy)
Comparing process improvement in manufacturing and construction: Duco vs Drywall (Brian Potter)

Announcements

An optogenetic retinal prosthesis (via @maxhodak_)
The Atlantic Progress Summit, LA, Dec 13 (via @DKThomp). There is a community meetup planned around this
A new mRNA vaccine against all known influenza virus subtypes (via @ScottEHensley)
Waymo One is opening to the public in San Francisco (via @Waymo)
AI plays Diplomacy (via @ml_perception). See Zvi Mowshowitz’s take

Links

Kevin Esvelt on how to prevent the next pandemic. In the inagural issue of the new Asterisk Magazine
Guinea worm disease is close to being eradicated (via @redouad)
Tyler Cowen on falling inequality

Queries

Concrete visions/narratives for technology over the next 20-30 years?) (@nabeelqu)
What’s the best economic case for space?
Examples of path-dependency in technology? (@_brianpotter)
Who should Milan Cvitkovic meet in London or Oxford? (@MWCvitkovic)
Anyone know about the early history of semiconductor research?
Who would you like to see speak at Peter Diamandis’s Abundance360 Summit? (@PeterDiamandis)
Suggested reading on the issue of preindustrial child abandonment?
What’s the most interesting thing/person/event in the world right now? (@pmarca)

Quotes

Turn-of-the-century horsesharing programs
The spirit of applied science, on the eve of the era of synthetic chemistry
Questions that arose when converting wood-burning houses to coal
If you like privacy, be glad you were born after indoor plumbing
The rapid transformation from containerization took everyone by surprise
Top athletes don’t see what they do as sacrificial at all. They like it (@mbateman)
Applying a theory to make predictions is not a matter of naive “extrapolation”
US-Russian relations, 1960s

Tweets & retweets

The challenges of long-distance transportation of hydrogen
Well-being as value-fulfillment
Hot takes cool down in prediction markets
When you make it harder to exit something, you make it harder to enter
Civil asset forfeiture delenda est
Americans are an amazing people, but perfectionist (@yashevde)
A potential US rail strike would cause major supply chain disruptions (@typesfast)
Preventing atherosclerosis, the easiest way to improve your life expectancy?
by Eli_ — 2022-11-29T20:05:23.933Z

Disclaimer: I’m not a doctor or specialized in this topic in any way.

Summary: Atherosclerosis is the number one cause of death and disability in the developed world. A large part might be preventable with lifestyle changes, medication and supplements. I am trying to figure out if I should start doing that and I am looking for feedback.

Note: Atherosclerosis is also known as Arteriosclerotic Vascular Disease (ASCVD); I will use these intertwined.

Introduction

I’m writing this article after listening to a podcast episode of the Drive by Peter Attia, because I’m looking for some input from the Lesswrong community. In this episode, #229 ‒ Understanding cardiovascular disease risk, cholesterol, and apoB, Peter highlights the importance of understanding cardiovascular disease and why early intervention is critical.

The outline of this article is as follows:

In Part 1 I will outline why I think that this is an extremely relevant topic and I share some introductory information on atherosclerosis.
In Part 2 I will discuss cholesterol and apolipoprotein B, which is a better, but underutilized marker of atherosclerosis.
In Part 3 I will introduce how atherosclerosis is diagnosed and why this seems flawed.
In Part 4 I will introduce possible prevention strategies and I will come to the conclusion that many people might want to start statin therapy really early—like in your twenties and thirties.
In Part 5 I will shortly highlight that this might be a surprising Effective Altruism cause.
I will end with my strategy going forward, discussing my personal testing, target and treatment strategy.

I have learned a lot in writing this, but I'm still looking for some convincing arguments that help in weighing the risks involved.

Part 1: Atherosclerosis?

Why should you care about atherosclerosis?

Let's start with some facts:

It is the number one cause of death and disability in the developed world.[1]
Almost everyone has sub-clinical levels of atherosclerosis. It takes about four decades for it to develop to the level that it becomes clinically significant.
The disease is almost completely asymptomatic until it transforms into a very serious condition. For a significant portion of people with atherosclerosis the first symptom is sudden cardiac arrest resulting in death.

To summarize it plainly: atherosclerosis is deadly, almost everyone has it and many people d...
Why Would AI "Aim" To Defeat Humanity?
by HoldenKarnofsky — 2022-11-29T19:30:07.828Z

Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.

I’ve argued that AI systems could defeat all of humanity combined, if (for whatever reason) they were directed toward that goal.

Here I’ll explain why I think they might - in fact - end up directed toward that goal. Even if they’re built and deployed with good intentions.

In fact, I’ll argue something a bit stronger than that they might end up aimed toward that goal. I’ll argue that if today’s AI development methods lead directly to powerful enough AI systems, disaster is likely1 by default (in the absence of specific countermeasures).

Unlike other discussions of the AI alignment problem,3 this post will dis

...
Why Bet Kelly?
by Joe Zimmerman — 2022-11-29T18:47:23.142Z

The Kelly criterion is an elegant, but often misunderstood, result in decision theory. To begin with, suppose you have some amount of some resource, which you would like to increase. (For example, the resource might be monetary wealth.) You are given the opportunity to make a series of identical bets. You determine some fraction f of your wealth to wager; then, in each bet, you gain a fraction f with probability p, and lose a fraction f with probability (1−p).[1]

In other words, suppose Wn is your wealth after n bets. We will define Zn=logWn, and we will suppose for simplicity that Z0=0. Then Zn=∑nt=1R, where R is a random variable defined as:

R={log(1+f)with probability plog(1−f)with probability (1−p)

Now suppose that, for some reason, we want to maximize E[Zn]. By linearity of expectation, E[Zn]=∑nt=1E[R]. Hence, we should simply maximize E[R]. This amounts to solving:

0=∂∂fE[R]0=∂∂f[plog(1+f)+(1−p)log(1−f)]0=(1−f)p−(1−p)(1+f)f=2p−1

This, f=2p−1, is known as the Kelly bet. For example, it says that if you have a 60-40 edge, then you should bet f=2(0.6)−1=0.2, i.e., bet 20% of your current wealth on each bet.

That all seems pretty reasonable. But why do we want to maximize E[Zn]? If we were to simply maximize expected wealth, i.e., E[Wn], then a straightforward calculation shows that we should not bet Kelly -- in fact, we should bet f=1 ("YOLO"), wagering the entire bankroll on every bet. This seems extremely counterintuitive, since, after n bets, our wealth would then be:

Wn={0with probability 1−pn2nwith probability pn

In other words, as n grows large, we would almost surely go bankrupt! Nevertheless, this would be the way to maximize E[Wn]. Kelly, whatever its merits, does not maximize E[Wn] -- not even in the long run. Especially not in the long run.

We now come to the perennial debate: why does Kelly seem "obviously right", and YOLO "obviously wrong"? There are many answers usually offered to this question.

First, what we believe to be the correct answer:

Utility is not linear in wealth. As originally observed by Bernoulli, utility tends to be approximately logarithmic in wealth. If utility happens to be exactly logarithmic in wealth, then the Kelly bet is optimal. F

...
Is Constructor Theory a useful tool for AI alignment?
by A.H. — 2022-11-29T12:35:07.462Z

This is an attempt to make sense of some vague ideas and explain some intuitions I've had recently. Written in the spirit of How To Write Quickly While Maintaining Epistemic Rigor .

"Any headline that ends in a question mark can be answered by the word 'no'." - Betteridge's Law of Headlines

Introduction

Recently, I have been thinking about constructor theory - a broad philosophical approach to providing explanations in physics. It was developed by David Deutsch, who claims that it has the potential to solve problems in physics that are hard to solve in other frameworks. When I read Deutsch's paper, I had the intuition that some of the problems in physics that constructor theory was meant to solve were analogous to problems associated with AI alignment. I thought that if constructor theory was able to solve these problems in physics, it might be helpful in the field of AI alignment.

I have been thinking about this on and off for the last few months and now believe that constructor theory probably does not have much to offer the field of AI alignment. I came to this conclusion broadly for two reasons. Firstly, constructor theory has few (zero?) concrete tools that could be imported to studying AI. Despite a lot of media coverage, constructor theory has only had limited success in solving the problems in physics that it set out to solve. It has produced a couple of interesting results, but they are very much results about physics, that (as far as I can see) cannot be easily imported for studying AI. Secondly, and maybe more importantly, the analogies between constructor theory and AI alignment break down in some important ways, once one starts digging a bit deeper.

In my estimation, this is a bad alignment idea. John Wentworth writes 'it takes most newcomers ~5 years to go from their initial Bad Idea to actually working on something plausibly useful' and suggests that 'progress along the Path of Alignment Maturity can be accelerated dramatically by actively looking for problems with your own plans'. In this spirit, in this piece, I will try to steelman the intuitions that constructor theory might be useful for AI alignment, and then point out why I now think that these intuitions are wrong.

The 'Dynamical Laws' Approach to Physics

Often, in physics, we describe systems using dynamical laws and initial conditions. The initial conditions specify the state of a system at...
Alignment allows "nonrobust" decision-influences and doesn't require robust grading
by TurnTrout — 2022-11-29T06:23:00.394Z

Definition. On how I use words, values are decision-influences (also known as shards). “I value doing well at school” is a short sentence for “in a range of contexts, there exists an influence on my decision-making which upweights actions and plans that lead to e.g. learning and good grades and honor among my classmates.”

Summaries of key points:

Nonrobust decision-influences can be OK. A candy-shard contextually influences decision-making. Many policies lead to acquiring lots of candy; the decision-influences don't have to be "globally robust" or "perfect."
Values steer optimization; they are not optimized against. The value shards aren't getting optimized hard. The value shards are the things which optimize hard, by wielding the rest of the agent's cognition (e.g. the world model, the general-purpose planning API).
Since values are not the optimization target of the agent with those values, the values don't have to be adversarially robust.
Since values steer cognition, reflective agents try to avoid adversarial inputs to their own values. In self-reflective agents which can think about their own thinking, values steer e.g. what plans get considered next. Therefore, these agents convergently avoid adversarial inputs to their currently activated values (e.g. learning), because adversarial inputs would impede fulfillment of those values (e.g. lead to less learning).

Follow-up to: Don't design agents which exploit adversarial inputs, Don't align agents to evaluations of plans

I: Nonrobust decision-influences can be OK

Decision-making influences don't have to be “robust” in order for a person to value doing well at school. Consider two people with slightly different values:

One person is slightly more motivated by good grades. They might study for a physics test and focus slightly more on test-taking tricks.
Another person is slightly more motivated by learning. They might forget about some quizzes because they were too busy reading extracurricular physics books.

But they might both care about school, in the sense of reliably making decisions on the basis of their school performance, and valuing being a person who gets good grades. Both people are motivated to do well at school, albeit in somewhat different ways. They probably will both get good grades, and they probably will both learn a lot. Different values simply mean that the two people locall...
Will chat logs and other records of our lives be maintained indefinitely by the advertising industry?
by mako yass — 2022-11-29T00:30:46.415Z

A sufficiently detailed record of a person's behavior could be used to fully reconstruct their psychology. This might constitute a form of immortality, but even if it doesn't, complete knowledge of the past would be a beautiful thing to have, it would make the future richer to know its past and to be able to bring back faithful replicas of the people who built it.

So this is an important question. To produce sufficiently detailed records... is that already happening by default? Is the ad industry already keeping all of that stuff? Will it all eventually make its way to light?
I've been assuming it's happening. Storage is cheap. The data has lots of buyers.
If not, then I'm going to need to start recording and backing stuff up more thoroughly.
Current Trends in Economics and their Shortcoming
by joshlevent — 2022-11-28T19:36:30.459Z

In economics, as in other sciences, we need universal theories, and we need to understand how they apply in specific contexts. This is quite distinct from the ideas of theoretical and empirical work or basic and applied science.

When we talk about theories of human behaviour (whether in economics, psychology, political science, etc.) we don't often consider the domain of those theories. The domain is of some importance, as it tells us how widely the theory can be generalised, and in some way, at what level the theory explains things. I can imagine three domains of human behaviour:

1. Contextual behaviour that applies only to some humans, or only some of the time. Those theories may be constrained by situation, culture, or any number of other aspects that can either change in the short or in the long run. For example, the crowding out of intrinsic motivation by extrinsic rewards (famously in some Israeli nursery schools, late pickups increased rather than decreased when a fine was introduced, presumably parents were able to absolve their guilt at coming late by paying the fine).[1]

2. Universal human behaviour, which applies to all humans, but not to other life forms or intelligences. There would be no reason to believe that intelligent aliens we might encounter would exhibit these behaviours. Recoiling from pain might be a good example.

3. Universal behaviour, which applies to all intelligent beings. These are “laws of nature”, and no intelligent being can violate them. A good example from economics is the Theory of Comparative Advantage. Although its applicability to nations is not fully resolved, it is clear that as applied to individuals it is an elegant, surprising, and powerful theory which must apply to all intelligent beings.[2]

One interesting thing to note about these three categories is that strictly speaking, the second category doesn't exist. If we can imagine aliens not following some behaviour that humans tend to exhibit, then we must also be able to imagine some humans not exhibiting that behaviour. We know of humans who don’t recoil from pain for example (a condition known as congenital insensitivity to pain).[3]

In economics in particular these distinctions appear to be mostly lost. It sometimes seems to me that economists working on different domains argue with one another about what is true when an explicit acknowledgement of the ...