trentmkelly/LessWrong-43k huggingface dataset:
The Village and the River Monsters... Or: Less Fighting, More Brainstorming
by ExCeph — 2022-10-03T23:01:18.973Z
Link: https://ginnungagapfoundation.wordpress.com/2022/09/29/the-village-and-the-river-monsters-or-less-fighting-more-brainstorming/

Cross-posted from The Ginnungagap Foundation. This story contrasts the status quo of human politics with what I would expect rational politics to look like.

It is apparent to me that making human politics more constructive is a low-hanging fruit with enormous benefits for the human wellbeing and longevity, both individual and collective. I contend that we should be proactive and assertive about introducing humans to more constructive approaches on the problems and conflicts they face than the approaches their ideologies provide.

Thoughts? Questions? Criticisms?

The story begins:

If you guess how this story ends, humanity could use your help right now. (Well, you can help humanity regardless, so if you want to learn how, read on.)

Once upon a time, my people say, there lived a village of humans. The humans of this village had a desperate problem. They were always in short supply of a medicinal herb, which they regularly needed to treat illnesses and diseases. What little of the herb they found, they had to ration out sparingly. The herb was a vine that grew only on trees, and few trees grew near their village.

Idiot advisory: This free stock image in a short story should not be construed as medical, botanical, or any other kind of advice. Injury or illness resulting from ingestion of plants that look like this are your own fault. Litigants will be publicly mocked.

Well, to be more precise, many trees grew near the village–an entire forest, in fact. But the forest was separated from the village by a river, and in that river swam monsters that would, often as not, sink any boat they found and eat anyone on it. Only a few lucky villagers had ever returned from the forest to bring back the herb, so most of the time the village had to make do with searching the few trees on their side of the river.

The crocodile picture is just here to invoke menace and dread, not to imply that the river monsters can walk on land. I suppose that means they’re actually slightly less terrifying than crocodiles.

After years of watching their friends and family pass away from illness when there was no medicine to be had, around half of the villagers decided that enough was enough, that the village should have a steady supply of medicine. They demanded that the village regularly h...
Recall and Regurgitation in GPT2
by Megan Kinniment — 2022-10-03T19:35:23.361Z

The first half of this post uses causal tracing to explore differences in how GPT2-XL handles completing cached phrases vs completing factual statements. The second half details my attempt to build intuitions about the high-level structure of GPT2-XL and is speculation heavy.

Some familiarity with transformer architecture is assumed but hopefully is not necessary to understand the majority of the post.

Thanks to Euan McLean for editing and Nix Goldowsky-Dill for comments and advice. All views are my own.

Context and Confidence

This post grew out of my final project for

MLSS

, which was replicating

this paper

on causal tracing. Most of what I show below is generated with my own code. I used this code to successfully reproduce some plots from the paper, so I’m reasonably (but not 100% confident in it).

I originally planned for this post to be much shorter. But I got curious about certain patterns I found, so decided to dig deeper. The second half of the post (“Speculation on the high-level structure of GPT2-XL”) is an unpolished collection of various bits of evidence alongside my own interpretation and (weakly held) opinions.

Summary

Using causal tracing, I looked at how GPT2-XL completes prompts of differing complexity. Some prompts only required the model to regurgitate verbatim the completion of a common phrase (“regurgitation prompts”), while others required the recall of some factual information about a subject (“semantic prompts”).

Causal traces associated with regurgitation look different to those associated with recall. In each case, relevant information in early tokens is transferred into the last token via attention during the forward pass. But the average layer at which the transfer is centered differs between cases. With regurgitation prompts, the information transfer tends to happen in earlier layers (~ layer 10-15), compared with semantic prompts where transfer happens later (~ layer 30).

Causal traces showing just the noised token and the final token, split by and averaged over prompt type.

This is consistent with the fact that in semantic prompts final token attention is concentrated more heavily on the noised token in the later layers, compared to regurgitation prompts where final token attention concentrates more heavily on the noised token in earlier layers.

A possible explanation for the differing average laye...
Ivermectin: Much Less Than You Needed To Know
by George3d6 — 2022-10-03T15:02:16.001Z
Link: https://doyourownresearch.substack.com/p/ivermectin-much-less-than-you-needed

I've recently gotten in touch with Alexandros Marinos, which has been writing lengthy critiques of Scott's most famous post, his review of Ivermectin.

After hearing his point of view, I must admit I've been swayed to the thinking that:

There are some obvious statistical errors in Scott's post and his interpretation of Bitterman's epidemiological data
The evidence for ivermectin, while poor, is about as reasonable as that for any repurposed covid drug (e.g. fluvoxamine), and even purpose-made ones (plaxovid)

While I fancy myself an off-label statistician I'm prone to error and maybe my interpretation is wrong. Given that Alexandros has recently finished this post which condenses his core criticisms, I was curious to see what people on LW thought about it.
If you want to learn technical AI safety, here's a list of AI safety courses, reading lists, and resources
by KatWoods — 2022-10-03T12:43:08.808Z

If you're interested in technical AI safety, it's hard to know where to start. Lots of people have made courses and reading lists to help simplify things.

We made a spreadsheet of such resources for learning about AI safety so you can easily see what's available. It was for internal purposes here at Nonlinear, but we thought it might be helpful to those interested in becoming safety researchers.

Please let us know if you notice anything that we’re missing or that we need to update. This was made about a year ago, but I try to keep it up to date.

Highlights

There are a lot of courses and reading lists out there. If you’re new to the field, out of the ones we investigated, we recommend Richard Ngo’s curriculum of the AGI safety fundamentals program. It is a good mix of shorter, more structured, and more broad than most alternatives. You can register interest for their program when the next round starts or simply read through the reading list on your own.

The new one by Dan Hendrycks also seems great. Structured much more like a classic MOOC and since there are no co-horts, you can just start whenever you'd like.
A review of the Bio-Anchors report
by jylin04 — 2022-10-03T10:27:58.259Z
Link: https://docs.google.com/document/d/1_GqOrCo29qKly1z48-mR86IV7TUDfzaEXxD3lGFQ8Wk/edit

This is a linkpost for a review of Ajeya Cotra's Biological Anchors report (see also update here) that I wrote in April 2022. It's since won a prize from the EA criticism and red-teaming contest, so I thought it might be good to share here for further discussion.

Here's a summary from the judges of the red-teaming contest:

This is a summary and critical review of Ajeya Cotra’s

biological anchors report

on AI timelines. It provides an easy-to-understand overview of the main methodology of Cotra’s report. It then examines and challenges central assumptions of the modelling in Cotra’s report. First, the review looks at reasons why we might not expect 2022 architectures to scale to AGI. Second, it raises the point that we don’t know how to specify a space of algorithmic architectures that contains something that could scale to AGI and can be efficiently searched through (inability to specify this could undermine the ability to take the evolutionary anchors from the report as a bound on timelines).

Note that a link to a summary/review of the book Principles of Deep Learning Theory on page 8 has been moved here: More Recent Progress in the Theory of Neural Networks.
Data for IRL: What is needed to learn human values?
by j_we — 2022-10-03T09:23:33.801Z

This post is a summary of my research. If you are interested in experiments and proofs check the full document.
This work was done as part of the Swiss Existential Risk Initiative (CHERI) Summer Research Program. Much thanks to my mentor Markus Peschl.

TLDR:

Gathering data from which human values can be learned via Inverse Reinforcement Learning is non-trivial, but is important to solve the alignemnt problem with this approach. I identify properties of data that will make it impossible for IRL to learn human values from it:

The demonstrator is irrational
The demonstrator has wrong beliefs about the world and IRL doesn't know about it
The data is not informative enough, because it is too short or important features are missing

I also derive data requirements & opportunities from properties of human values:

Human values are dependent on the context => data from many diverse contexts is needed
Human values change over time => continuous stream of data is needed
Humans don't always act according to their values => more sophisticated models of human decision making and more data
Human values are often shared => to learn values of an individual, data from others can be leveraged
Human values are diverse => Learn from many people with diverse values

1. Introduction

Data is a core part of every machine learning pipeline. Thus it's plausible that quality and type of data are also crucial for the success of inverse reinforcement learning in learning human values. If this is true, then determining what data is needed and how to get it is an important part of solving the alignment problem with this approach. Determining this would also clarify the chances of success for IRL for learning human values and clarify what challenges need to be met by algorithms. However, thinking about data for IRL has been widely neglected (except for

Model Mis-specification and Inverse Reinforcement Learning

). Thus, I aim to shed some light on the question: “What are necessary and beneficial properties of data, so that IRL can correctly learn human values from it?”.

I aim to attack this question from two sides: In Approach 1, I investigate which properties of data would make it impossible for IRL to correctly learn rewards (Section 2). In Approach 2, I argue that certain properties of human values imply the need of data with certain properties to be able to learn the values (...
Statistics for objects with shared identities
by Q Home — 2022-10-03T09:21:15.884Z

I want to know if there exist statistics for objects that may "share" properties and identities. More specifically I'm interested in this principle:

Properties of objects aren't contained in specific objects. Instead, there's a common pool that contains all properties. Objects take their properties from this pool. But the pool isn't infinite. If one object takes 80% of a certain property from the pool, other objects can take only 20% of that property.

How can an object take away properties from other objects? What does it mean?

Example 1. Imagine you have two lamps. Each has 50 points of brightness. You destroy one of the lamps. Now the remaining lamp has 100 points of brightness. Because brightness is limited and shared between the two lamps.

Example 2. Imagine there are multiple interpretations of each object. You study the objects' sizes. Interpretation of one object affects interpretations of all other objects. If you choose "extremely big" interpretation for one object, then you need to choose smaller interpretations for other objects. Because size is limited and shared between the objects.

Different objects may have different "weights", determining how much of the common property they get.

Do you know any statistical concepts that describe situations when objects share properties like this?

Analogy with probability

I think you can compare the common property to probability:

The total amount of the property is fixed. New objects don't add or subtract from the total amount.
"Weight" of an object is similar to prior probability. (Bayes' theorem)
The amount of property an object gets depends on the presence/absence of other objects and their weights. This is similar to conditional probability.

But I never seen Bayes' rule used for something like this: for distributing a property between objects.

Probability 2

You can apply the same principle of "shared properties/identities" to probability itself.

Example. Imagine you throw 4 weird coins. Each coin has a ~25% chance to land heads or tails and a ~75% chance to merge with some other coin. At least one coin always remains. Not sure how the outcome of "merged" coins is determined, different rules are possible.

Here's an illustration of some possible outcomes of throwing 4 weird coins: image. Disappeared coins affect the remaining ones in some way. (If disappeared coins don't affect the remaining ones then "disappeared" is jus...
Visualizing Learned Representations of Rice Disease
by muhia_bee — 2022-10-03T09:09:12.442Z
Link: https://indecisive-sand-24a.notion.site/Visualizing-Learned-Representations-of-Rice-Disease-43178513deaa4f04b03c246e57ac16c7

By Brian Muhia

Status: Mid-project progress report after developing some thinking about potential future research directions. This is also part of a grant application [private communication].

https://github.com/poppingtonic/semisupervised-rice-africa

Motivation

The historical amount of pesticide use in agriculture has been a constant producer of illness as a secondary effect, that can be reduced by early plant disease detection. This document by Our World In Data shows which countries are most affected by massive amounts of pesticide use. Artificial intelligence should be helpful in more targeted, early interventions that help people to avoid using large amounts of pesticides. This project is to ensure the intelligence is interpretable, trustworthy, reproducible and generative. First we train, then we explain a family of self-supervised vision-only models in order to study the value of rigorous interpretability methods, and their potential for educating practitioners on their uses - leading eventually to a world where more practitioners routinely explore model capabilities and limits with more rigor than is the current norm. The goal here is to improve food safety, and also AI safety by exemplifying a norm of rigorous interpretability research.

Training Log Narrative

<aside> 💡 These loss curves can be reproduced during the training of an unsupervised encoder (from imagenet weights, and from scratch) that is further finetuned for classification in two ways: mlp-only, and discriminatively. The notebooks are in the linked repo. In the plot I included about 60 different iterations, for different image scales and architectures.

</aside>

This article starts with a walk-through of my thoughts about the different training runs from this public competition to help Egypt, while my memory is still fresh. First thought that occurs is that there seem to be two phases to training this model if you’re fine-tuning from ImageNet weights.

This is recommended as you can do this in a few hours on a laptop as a proof of concept, as opposed to a few days of waiting for the from-scratch model to randomly wander its way down to low-loss land.

First, we use the ImageNet weights to carve a significant chunk out of the error before plateauing. The dark red trajectory (scroll up to the cover image) shows a Res2Next50 making it out of what would have been several hours while training...
Is there a culture overhang?
by Aleksi Liimatainen — 2022-10-03T07:26:48.368Z

Culture is adapted for learnability, transmissibility and recombinability for humans. To the extent AI and natural intelligence operate on similar principles, these adaptations should be expected to carry over when culture is used as a training dataset for AI. If so, an AI trained on culture would catch up to the cultural state of the art fast but have more trouble progressing beyond it.
If the above holds, one would expect rapid AI progress up to the frontiers of human culture and nearby low-hanging fruit, after which progress would slow. More speculatively, depending on our compatibility with advancing AI's outputs, humans could have an unexpectedly easy time keeping up with the AI's (relatively slower) acceleration.
Are we currently living in a culture overhang? If so, how does it affect the picture of AI timelines?
Baby Monitor with Delay
by jefftk — 2022-10-03T01:40:01.371Z

Our youngest (15m) has recently started sleeping through the night on
her own, without us needing to get up for feeding or settling. But
she's still waking (

one
of) us up, because she often cries a bit before falling back
asleep. Turning the monitor off entirely is an option, but then if
something were pretty wrong (ex: vomiting) we might keep sleeping.
I'd be interested in a monitor with a configurable delay: only turn on
the monitor if they've been crying for, say, 8min.

A monitor like this would also be useful for sleep training. One
common approach is to pick some sort of timing pattern for when to go
back in to settle the baby, and not go back in before then. For
example, you might settle them, and then hope they stop crying and
fall asleep, but if they're still crying in 15min you repeat. Trying
to ignore someone you're very attached to cry while trying to remember
when exactly it will have been 15min while super sleepy is not fun.
If baby monitors had configurable snooze buttons, after settling the
baby you could snooze the monitor and try to fall back asleep. If
you're lucky they fall back asleep before the fifteen minutes are up,
and you more sleep.

There are a bunch of ways to make this fancier, including only
counting crying and not other noises or identifying unusual crying,
but even something basic like turning on when the average recent
volume reaches a threshold would be very helpful.

I did find someone
who built a prototype using a phone, with a much shorter
delay (10s). I can't find any products, though. Would other people
find this useful?

Comment via: facebook
What makes a probability question "well-defined"? (Part I)
by Noah Topper — 2022-10-02T21:05:25.364Z
Link: https://naivebayes.substack.com/p/what-makes-a-probability-question

See also:

Probability is in the Mind

,

You Are Never Entitled to Your Opinion

, and

“I don’t know.”

(Cross-posted from a new blog I'm trying to start. Sorry if some of the material is a bit basic for these parts, there's a larger point I'm trying to build up to.)

I. A puzzling probability problem

Suppose you have a random number generator, which gives values from 1 to 6. Like a die, but not necessarily a literal one; let’s call it a “die” anyway. You don’t have any other information about the relative frequency of the outcomes, and have no reason to expect one outcome more than any other. What’s your estimated probability distribution? That is, on the next roll, what’s the chance it’s a 1 (or 2, 3, 4, 5, or 6)?

“1/6 for each!”, you say cheerfully. Well, fair enough, I concur! You have no reason to say any one result is more likely than any other, so a uniform distribution is natural. But suppose I now make the problem slightly harder. I tell you the mean value is 4.5 (rather than 3.5, as it would be for a fair die). What’s your probability distribution now? What’s the probability that the next roll is, say, a 2?

I think many more people would now be tempted to say the question isn’t well-defined, there’s not enough information to answer. There are infinitely many probability distributions consistent with this average, after all, and no clear way to single out one amongst them all.

But, there’s something very funny about this response. In the second question, you are actually given more information about the die than in the first. How could this possibly make the question less well-defined? It can’t, of course!

II. Is the question really well-defined?

In fact, you could call the first question undefined on this basis, and some do! (Sorry for leading the witness before.) It’s not super common amongst ordinary people I’ve talked to, but a certain contingent are tempted to say that if you don’t precisely specify the frequencies of each outcome, then there’s nothing we can say about the probability of the next roll. (I would take the opposite end of this dilemma and say both questions are well-defined.)

If you yourself are so tempted, I would respond that you can call practically any probability question ill-posed on this basis. Consider an actually “fair” die. We all agree it’s a 1/6 chance for each outcome. But I could complain that you haven’t given me enough...
Against the weirdness heuristic
by Eleni Angelou — 2022-10-02T19:41:10.562Z

Crossposted from the EA Forum: https://forum.effectivealtruism.org/posts/DYmDGizxkcrZhmBCq/against-the-weirdness-heuristic

Slightly more than a decade ago (approximately 2009), I was imagining a pair of earphones that would look exactly like my iPod earphones but would have no wires. That pair was perfect in my imagination: it would fit in my ears just like my wired pair did, the sound would be great, and I wouldn't have to worry about disentangling wires. At the time, wireless meant "ugly Bluetooth (cool) kids never use". In 2016, Apple released the AirPods; they were pretty much what I had imagined plus you needed to charge them (I hadn't thought about that...)

About two decades ago, in the early 2000s, my father went abroad for a few months to work. We knew we wouldn't see his face for that period of time. I don't really remember calling with him either but maybe I'm confabulating here. His absence would have felt way less dramatic were we to video call. In 2010, FaceTime was released while Skype had already been around for many years.

I mention these two stories to argue that both "imagined" technologies became available very soon after I imagined them. I wasn't particularly a tech person, at least no more than the average child growing up at the time. I couldn't have predicted that these technologies should be available based on anything because, well, I didn't know anything. I was just thinking what it'd be nice to have and soon after this thing would be out for sale. (I personally didn't get AirPods till 2022, however).

Thinking about technology in the AI framework often creates the impression that the AGI future is weird. And if it's weird, it's also less likely to happen. This is the weirdness heuristic: I reject something not because I have evidence to do so but because we should generally not anticipate weird things to be true.

In retrospect, it's very easy to assume that having wireless earphones that you charge in a tiny case as well as video calling with someone on the other side of the planet whenever you want are both "normal", non-weird futures. But would we agree on that in the early 2000s?

Writing the "history of the future" would solve many of our problems. We would have clear timelines and plan accordingly. Maybe we would find that there isn't enough time to prepare or even if there is, that we still don't make it. Regardless, the arrow ...