trentmkelly/LessWrong-43k huggingface dataset:
Thoughts on safety in predictive learning
by Steven Byrnes — 2021-06-30T19:17:40.026Z

(Many thanks to Abram Demski for an extensive discussion that helped clarify my thoughts. Thanks Abram &

John Maxwell

for comments and criticisms on a draft.)

Background: Why do I care?

Way back in ancient history, i.e. two years ago, my first ever attempt to contribute something original to the field of AGI safety was the blog post

The Self-Unaware AI Oracle

. Part of my premise there was that I was thinking maybe you can make a safe AGI by using 100% self-supervised predictive learning—i.e., the model tries to predict something, you compare it to what actually happens, and then you improve the model by gradient descent or similar.

That wasn’t a great post, although it was a good experience thanks to helpful commenters who guided me from “very confused” to “somewhat confused”. Then I wrote a couple other posts about self-supervised learning, before moving on to other things because I came to feel that you probably need reinforcement learning (RL) to reach AGI—i.e., that predictive learning by itself won’t cut it. See

Section 7.2 here

for why I (currently) think that. Note that others disagree, e.g. advocates of

Microscope AI

.

Fast forward to a couple months ago, when I wrote

My AGI Threat Model: Misaligned Model-Based RL

, where one gear in the AGI is a world-model trained by 100% predictive learning. Now that, I can get behind, as an architecture that could plausibly scale to AGI. Here the whole AGI is using RL and planning; it’s just that this particular gear is not.

Inevitable neuroscience side-note: I try to talk in ML lingo, but I’m always secretly thinking mainly about human brain algorithms. So anyway, in the brain case, the “gear” in question corresponds roughly to the non-frontal-lobe parts of the neocortex, which build predictive models of sensory inputs, and AFAICT their only training signal is predictive error (

example ref

). See

Is RL involved in sensory processing?

The answer by the way is: No. We don’t want RL here, because RL is for making decisions, and this part of the brain is not supposed to be deciding anything. I’m oversimplifying in various ways, but whatever, I don’t want to stray too far off-topic.

So anyway, here I am, back on the market for an argument that world-modeling systems trained 100% on prediction error are not going to do dangerous things, like try to seize more computing power and take over the world or whatever. We want the...
Musings on general systems alignment
by Alex Flint — 2021-06-30T18:16:27.113Z

Epistemic status: Musings from the past month. Still far too vague for satisfaction.

Friends.

I have been away on a retreat this past week, seeking clarity on how to move forward with living a vibrant and beneficial life that resolves problems in the world, but I’m afraid that I have come back empty-handed. I have only vague musings of a bigger picture, but no clear sense for how to take decisive action. I’ll try to share what I have succinctly, so as not to take up too much time.

Understand alignment, not intelligence

Look, our task here is to align the systems that will most influence the future with what is actually good. To do that, we should look out at the world, identify which kinds of systems are most influential, and seek to align them to the benefit of all life on the planet. Intelligence is the means by which certain very powerful systems could have a very large influence over the future, and to that end we ought to be interested in understanding intelligence. But we need not have any particular interest in understanding intelligence for its own sake. What we should be interested in understanding is the means by which any system can exert influence over the future, and the means to align such powerful systems with that which is worth protecting.

Align systems, not AI

There has been great debate about what AI might look like. Will it look like a singleton, or like a tool, or like a set of cloud services, or like a society of competing entities? One person says that a powerful singleton might be dangerous, then another person says that AI might not look much like a powerful singleton.

Yet there is a single unifying issue to resolve here, which is this: how do we build things in the world that are and remain consistently beneficial to all life? How do we construct international treaties that are aligned in this way? How do we construct financial systems that are aligned in this way? How do we construct tools that are aligned in this way? How do we construct belief-forming, observation-making, action-taking agents that are aligned in this way? These questions are connected in a deep, not surface-level way, because they all come down to clarifying what is good and implementing it in a tangible system.

There is a hard problem of alignment

There are many difficult problems in AI alignment, but there seems to be one problem at the center that has an entirely dif...
[AN #154]: What economic growth theory has to say about transformative AI
by Rohin Shah — 2021-06-30T17:20:03.292Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Could Advanced AI Drive Explosive Economic Growth? (Tom Davidson) (summarized by Rohin): Some (AN #121) previous (AN #105) work (AN #145) has suggested that by 2100 there is a non-trivial chance that AI could lead to explosive growth, that is, a growth rate of 30% (i.e. a doubling time of 2-3 years), 10x the current growth rate of ~3%. What does economics have to say about the matter?

This report investigates the following three stories:

1. Ignorance story: In this story, we don’t know how growth is determined, and attempts to forecast it based on models of how growth works are likely to be wrong. Note that this is perfectly compatible with explosive growth. We know that the growth rate has increased by orders of magnitude over the past millennia; so on an ignorance story we certainly shouldn’t rule out that the growth rate could increase by an order of magnitude again.

2. Standard story: This story focuses on the last ~century of growth, noting that the growth rate has stayed relatively constant at 2-3% per year, and thus predicting that future growth will be exponential (i.e. a constant growth rate), or possibly subexponential.

3. Explosive story: This story focuses on growth models with positive feedback loops, in which increased output leads to increased inputs which leads to even larger outputs, resulting in superexponential (and explosive) growth.

The author is interested in whether explosive growth is plausible, and so is most interested in arguments that argue for the standard story and against the ignorance or explosive stories, or vice versa. The main empirical facts we have are that the growth rate increased (maybe continuously, maybe not, it’s hard to tell) until about a century ago, when it plateaued at the current level of 2-3%. What can we then learn from economic growth models?

1. Ideas-based models of economic growth suggest that growth in output is driven primarily by the rate at which we get id...
Progress on Causal Influence Diagrams
by tom4everitt — 2021-06-30T15:34:33.381Z

By Tom Everitt, Ryan Carey, Lewis Hammond, James Fox, Eric Langlois, and Shane Legg

Crossposted from DeepMind Safety Research

About 2 years ago, we released the

first

few

papers

on understanding agent incentives using causal influence diagrams. This blog post will summarize progress made since then.

What are causal influence diagrams?

A key problem in AI alignment is understanding agent incentives. Concerns have been raised that agents may be incentivized to

avoid correction

,

manipulate users

, or

inappropriately influence their learning

. This is particularly worrying as training schemes often shape incentives in

subtle

and

surprising

ways. For these reasons, we’re developing a formal theory of incentives based on causal influence diagrams (CIDs).

Here is an example of a CID for a one-step Markov decision process (MDP). The random variable S1 represents the state at time 1, A1 represents the agent’s action, S2 the state at time 2, and R2 the agent’s reward.

The action A1 is modeled with a decision node (square) and the reward R2 is modeled as a utility node (diamond), while the states are normal chance nodes (rounded edges). Causal links specify that S1 and A1 influence S2, and that S2 determines R2. The information link S1 → A1 specifies that the agent knows the initial state S1 when choosing its action A1.

In general, random variables can be chosen to represent agent decision points, objectives, and other relevant aspects of the environment.

In short, a CID specifies:

Agent decisions
Agent objectives
Causal relationships in the environment
Agent information constraints

These pieces of information are often essential when trying to figure out an agent’s incentives: how an objective can be achieved depends on how it is causally related to other (influenceable) aspects in the environment, and an agent’s optimization is constrained by what information it has access to. In many cases, the qualitative judgements expressed by a (non-parameterized) CID suffice to infer important aspects of incentives, with minimal assumptions about implementation details. Conversely, it has

been shown

that it is necessary to know the causal relationships in the environment to infer incentives, so it’s often impossible to infer incentives with less information than is expressed by a CID. This makes CIDs natural representations for many types of incentive analysis.

Other advantages of CIDs i...
Simulacrum Levels of Trust
by alkexr — 2021-06-30T10:14:24.512Z

Epistemic Status: Pattern matching. The model completes a pattern in an intuitive way. Little originality besides the act of pattern completion itself.

As I was reading the recent discussion about networks of trust, I noticed a mismatch between models described therein and the social environment around me. In particular, I found that I would be happy to participate in the kinds of networks that came up as examples. That confused me, because in practice, I actively try to distance myself from some local communities. In fact, I've always thought that trust-based relationships were just a burden without any practical benefits for anyone involved. As it turns out, the reason for the mismatch is that the little bubble where my intuitions came from is actually different.

In the networks of trust that I can observe out there people don't actually seem to be interested in helping others where help is actually needed. Or at least, they don't act like it. And it isn't just a bug, either, because no one involved asks for, or expects, any helpful help. In the meantime, there is a complex culture of ritual incantations and arbitrary norms around it that everyone is careful to follow for fear of losing social status points. In other words, it smells of the exact same kind of bullshit that simulacrum level 3 is notorious for.

So I completed the pattern. The trust that is actually trust will be simulacrum level 1 (S1). The "trust" I observe out there will be S3. The pattern does complete in an obvious way for the other two levels, and the resulting model does explain my two datapoints well.

Level 1: The Handshake

The definition of S1 is "I help you, and expect you to help me; you help me, and expect me to help you."

At this level, you may offer a hand to your neighbor because you think you could be helpful.

In the world of trust-proxies, a company may deliver what the customer has paid for, because they think they offer something of value for a fair price.

A suftaja, as described by Henrik Karlsson, also belongs to this level. As a quick summary:

The level of trust [...] between merchants in the great Malay entrepôt Malacca, gateway to the spice islands of Indonesia, was legendary. The city had Swahili, Arab, Egyptian, Ethiopian, and Armenian quarters, as well as quarters for merchants from different regions of India, China, and Southeast Asia. Yet it was said that its merchants shu

...
Is it true that most studies in Kahneman's 'Thinking fast and slow' are not replicable and the book is largely not supported by evidence?


by Davy Jones — 2021-06-30T07:14:39.238Z

So I stumbled on this article.

https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow

'So, replicability [of all studies in this book] is somewhere between 12% and 46%. Even if half of the results are replicable, we do not know which results are replicable and which ones are not.'

'Readers of “Thinking: Fast and Slow” should read the book as a subjective account by an eminent psychologist, rather than an objective summary of scientific evidence.'

I have no background in social sciences or statistics so I don't know if claims and math in this article are correct. Could somebody with more knowledge comment on this? This is HUGE if true.

I love this book to pieces but I don't want to go around spreading outdated science.. are there any similar books on human biases and thinking which are more recent and more robust when it comes to evidence and statistics?
Do incoherent entities have stronger reason to become more coherent than less?
by KatjaGrace — 2021-06-30T05:50:10.842Z

My understanding is that various ‘coherence’ arguments exist of the form:

If your preferences diverged from being representable by a utility function in some way, then you would do strictly worse in some way than by having some kind of preferences that were representable by a utility function. For instance, you will lose money, for nothing.

You have good reason not to do that / don’t do that / you should predict that reasonable creatures will stop doing that if they notice that they are doing it.

For example, from Arbital:

Well, but suppose I declare to you that I simultaneously:

* Prefer onions to pineapple on my pizza.
* Prefer pineapple to mushrooms on my pizza.
* Prefer mushrooms to onions on my pizza.

...

Suppose I tell you that I prefer pineapple to mushrooms on my pizza. Suppose you're about to give me a slice of mushroom pizza; but by paying one penny ($0.01) I can instead get a slice of pineapple pizza (which is just as fresh from the oven). It seems realistic to say that most people with a pineapple pizza preference would probably pay the penny, if they happened to have a penny in their pocket.[1]

After I pay the penny, though, and just before I'm about to get the pineapple pizza, you offer me a slice of onion pizza instead--no charge for the change! If I was telling the truth about preferring onion pizza to pineapple, I should certainly accept the substitution if it's free.

And then to round out the day, you offer me a mushroom pizza instead of the onion pizza, and again, since I prefer mushrooms to onions, I accept the swap.

I end up with exactly the same slice of mushroom pizza I started with... and one penny poorer, because I previously paid $0.01 to swap mushrooms for pineapple.

This seems like a qualitatively bad behavior on my part. By virtue of my incoherent preferences which cannot be given a consistent ordering, I have shot myself in the foot, done something self-defeating. We haven't said how I ought to sort out my inconsistent preferences. But no matter how it shakes out, it seems like there must be some better alternative--some better way I could reason that wouldn't spend a penny to go in circles. That is, I could at least have kept my original pizza slice and not spent the penny.

In a phrase you're going to keep hearing, I have executed a 'dominated strategy': there exists some other strategy that does

...
How do the ivermectin meta-reviews come to so different conclusions?
by ChristianKl — 2021-06-29T22:34:47.443Z

Ivermectin and outcomes from Covid-19 pneumonia: A systematic review and meta-analysis of randomized clinical trial studies comes to the conclusion: "Our study suggests that ivermectin may offer beneficial effects towards Covid-19 outcomes. More randomized clinical trial studies are still needed to confirm the results of our study."

On the other hand Ivermectin for the treatment of COVID-19: A systematic review and meta-analysis of randomized controlled trials comes to the conclusion: "In comparison to SOC or placebo, IVM did not reduce all-cause mortality, length of stay or viral clearance in RCTs in COVID-19 patients with mostly mild disease. IVM did not have an effect on AEs or severe AEs. IVM is not a viable option to treat COVID-19 patients."

What did the studies do differently to come to their conclusions? How do I go about interpreting which of them provides the better analysis?
D&D.Sci(-Fi) June 2021 Evaluation and Ruleset
by abstractapplic — 2021-06-29T21:02:20.072Z

This is a followup to the D&D.Sci post I made last week; if you haven’t already read it, you should do so now before spoiling yourself.

Here is the web interactive I built to let you evaluate your solution; below is an explanation of the rules used to generate the dataset. You’ll probably want to test your answer before reading any further.

Ruleset

This challenge runs mostly on Poisson distributions; the rest of this post will abbreviate “Take the result of a Poisson process with lambda=X” as “Po(X)”.

Heteropneums

There are two types of heteropneum. The most common is the garden-variety heteropneum, which exists as a natural consequence of the way soul energy clusters. This type has a randomly-generated amplitude which is always a multiple of 0.142 kCept. (You can infer that Earwax must not be of this kind because no amplitude which would round to 3.2 kCept is a multiple of 0.142 kCept.)

There are also a much smaller number of [Teeming] heteropneums, which exist because the [Teeming] hivemind does. These appear at times and with intensities not determined by the rules of this world (in other words, decided arbitrarily by the GM). Nicholas, Neville, Jasper, Athena, Endeavour, Spindle, Divisor, Desk, Pencil, Duchess, Economy, Sirocco, Constellation, Harbinger, Minimalist, Spackle, Tourniquet, Painting, Toast, and most importantly Earwax are [Teeming] heteropneums; all others are not.

Who Fights What

Heteropneums with amplitudes of >2.5 kCept are handled by Maria. Heteropneums in the 1.2-2.5 kCept range are split evenly between Maria and Janelle. Weaker heteropneums are allocated unevenly between all viable pilots.

Fighting (as Maria)

Let A be the amplitude of the heteropneum Maria is facing, in kCept.

Alpha

Alpha Resonance lets Maria create 3.4444 + 0.0285*Po(10.2) kCept of Effective Field Strength. On Floorday 502, a breakthrough was made which allowed the Sphere’s submarines to better utilize this form of Resonance; this adds an extra 0.5111 kCept.

Beta

Beta Resonance lets Maria create 2.4011 + 0.1034*Po(3.9) kCept of Effective Field Strength.

Gamma

Gamma Resonance gets stronger as the enemy does. It lets Maria create 0.6626 + A*0.66*Po(1.1) kCept of Effective Field Strength.

Delta

Delta Resonance is Maria’s preferred means of attacking. It lets her create A*0.32 + 3.253 + 0.0882*Po(19.2) kCept of Effective Field Strength.

Epsilon

Epsilon Resonance has a complex but pred...
Search by abstraction
by p.b. — 2021-06-29T20:56:57.314Z

A couple of years ago, when I delved into the details of machine learning algorithm/neuroscience model HTM, I had an idea about the nature of thought, specifically problem-solving thought. In the lingo of this site, it’s a theory about the internal workings of the Babble generator.

HTM theory, which now might go by „thousand brains hypothesis“ posits that the mammalian cortex learns a model of the world in the form of temporal sequences of sparse distributed representations. This world model provides you with predictions of future events. You can use it to create „rollouts“ and sample the future as predicted by your past experience.

Certainly sounds like a useful thing to have. The point where I got stuck, was when I considered novel problem solving.

Consider the case of trying to solve a mathematical problem. Sure enough our Babble generator creates plenty of tries and seems to continue doing so even after we have thrown the towel. If these were random rollouts our Eureka moments would consist of fully formed sequences that pop into our heads. Instead we experience an „insight“, a compact, non-sequential what-ever-it-is, that allows us to unroll the correct sequence consciously.

Now imagine a squirrel encountering a car tire in the woods. There is a nut on top of the tire. Triggered by the situation „standing in front of a unknown black round thing with a nut on top“ how does the babble generator create a solution - i.e. a sequence of actions that leads to a positive outcome? The squirrel has often jumped or climbed on things to retrieve nuts, but it has never seen a tire before. Why would the babble generator produce a jumping or climbing sequence?

What needs to happen, according to my theory, is that from the initial trigger, i.e. from the initial representation of the situation, many aspects have to be abstracted away until the final backbone of the situation triggers a realistic sequence with a positive outcome. For the squirrel all or most aspects of the tire have to be abstracted away until the situation is very similar to a more common situation like standing in front of a tree or stone with a nut on top.

The more complex the situation and the more uncommon the aspects that have to be abstracted away, the longer it will take until the babble generator will hit upon the correct abstract trigger to create a solution that is not pruned as un...