trentmkelly/LessWrong-43k huggingface dataset:
Evidence Sets: Towards Inductive-Biases based Analysis of Prosaic AGI
by bayesian_kitten — 2021-12-16T22:41:41.668Z

Epistemic Status: Exploratory. My current but-changing outlook with limited exploration & understanding for ~60-80hrs.

Acknowledgements: This post was written under Evan Hubinger’s direct guidance and mentorship as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program. Thanks to particlemania, Shashwat Goel and shawnghu for exciting discussions. They might not agree with some of the claims made here; all mistakes are mine.

Summary (TL;DR)

Goal: Understanding the inductive biases of Prosaic AI systems could be very informative towards creating a frame of safety problems and solutions. The proposal here is to generate an Evidence Set from current ML literature to model the potential inductive bias of Prosaic AGI.

Procedure: In this work, I collect evidence of inductive biases of deep networks by studying ML literature. Moreover, I estimate from current evidence whether these inductive biases vary with scaling to large models. If a phenomenon seems robust to or amplified by scaling, I discuss it here and add it to the Evidence Set.

Structure: I provide interpretations of some interesting papers to AI safety in three maximally-relevant subareas of ML literature (pretraining-> finetuning, generalization and adversarial robustness), and demonstrate use cases to AI safety (in ‘Application’ subsections). I then summarize evidence from each area to form the Evidence Set.

Current Evidence Set: Given in Section “Evidence Set” (last section of this post)

Inspiration: I think developing good intuitions about inductive biases and past evidence essentially constitutes ‘experience’ used by ML researchers. Evidence sets and broadly analyzing inductive biases are the first steps towards mechanizing this intuition. Inductive bias analysis might be the right level of abstraction. A large, interconnected Evidence Set might give a degree of gears-level understanding of the black-box that is deep networks.

Applications: Any theory of inductive biases, if fully developed, can sufficiently explain the observations collected in the evidence set. Alternatively, we can use the evidence set to falsify mechanistic statements about training rationales in Training stories. However, I expect evidence sets to be helpful beyond these two applications.

Limitations: Evidence...
Housing Markets, Satisficers, and One-Track Goodhart
by J Bostock — 2021-12-16T21:38:46.368Z

There are two parts to what people generally refer to as the "Housing Crisis". One is the simple fact that homes are too expensive. The other part is that building new, expensive houses pushes existing, poorer, renting residents out of communities and scatters them across the outskirts of cities where they have to spend eight hours a day commuting.

Many people think the solution to the first part is building more houses. Many people also think that building more houses conflicts with solving the second part. To me the question is this: given that people like communities and presumably would be happy to pay money for them, why isn't this currently a factor in the housing market?

Regular Goodhart

Normal Goodhart's law goes like this:

A system Soptimizes for X
Xis correlated with U, which we want
So we like system Sand give it more power
With too much power, S makes lots of Xat the cost of U
Now we have no U

One case of this is distributional shift, where for very big X, it's no longer correlated with U. For example height and basketball ability.

An example in a typical market would be like this:

Company in market optimizes for making profit
Good products are correlated with profit
With too much power the strategies like "make good product" are dominated by other strategies like "form a total monopoly" or with enough power, extreme cases like "take over France for slave labour"

This is a case of standard Goodhart.

One-Track Goodhart

Now consider the following case:

Housing developers optimize for making profit
Various factors are correlated with house price: sufficient supply of places to live, nice communities, being pleasant to live in
With such an undersupply, the factor of supply/demand dominates everything else
Now housing developers are incentivised to remove existing housing in favour of servicing the parts of the market where undersupply is greatest

Note how we've gone out of the distribution where profits correlate with a bunch of human profits. Now they only correlate with producing as much housing as possible for the most underserviced part of the market (as a function of wealth to spend), which in this case is lots of expensive apartments. We've not done this by letting the market optimize harder for profit, we've done it due to making one factor dominate the optimization process by pushing the situation out o...
Covid 12/16: On Your Marks
by Zvi — 2021-12-16T21:00:01.788Z

The main action continues to be at the Omicron post. There’s still some background developments and near term developments worth noticing, but for now nothing that seems as important or that seems terribly urgent, so read that first.

Executive Summary

Omicron is here and will take over soon.
We might not do much about it until far too late.
And that’s probably better than the alternative.

Also some other stuff, like Will Eden’s excellent tweetstorm below.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 750k cases (+2%) and 8,800 deaths (+5%).

Results: 747k cases (+2%) and 8,813 deaths (+5%).

Prediction for next week: 862k cases (+15%) and 9,050 deaths (+3%).

Those were some pretty good predictions. Don’t count on it happening again any time soon. Stuff’s about to get weird.

The case growth depends on how far along we are in Omicron taking over, and when those tests results will come in and how many people will get tested if the cases are mild. If we take the 2.9% for the period ending 12/11 seriously, we now are moving ahead 12 days, I’ll guess the true general doubling time is 3 days, so for the sample we’ll get it’s going to be something like 18% (+12% from previous levels) Omicron, and cases of Delta likely remain roughly level, with more upside than downside for the count. It’s already at or last that level in the specific places we’re measuring, but it makes sense that they’d be ‘ahead’ of other areas with much larger populations. And we’ll likely see some mitigation. So I’ll go with +15%, but the error bars are wildly higher than before. So many variables.

Deaths

Deaths are straightforward, no reason to expect major change. Paxlovid remains illegal, Omicron isn’t here yet for long enough.

Cases

This is the question. We saw some regional movement last week, it’s probably largely random reporting fluctuations, except that the rise in the Northeast is very real. Going forward it’s all about Omicron. It’s super hard to time it, and what comes up on tests is several days behind what’s actually happening (e.g. a full doubling time) but my guess is that we’re already above 10% and increasing fast. London is our future, it’s merely unevenly distributed for a limited time.

So, how about Paxlovid? Can we hurry?

Paxlovid Remains Illegal

The final numbers are in, and they look good. Efficacy...
Reviews of “Is power-seeking AI an existential risk?”
by Joe Carlsmith — 2021-12-16T20:48:26.808Z

(Edited 10/14/22 to add Lifland review. Edited 7/10/23 to add Levinstein review. Edited 10/18/23 to add superforecaster reviews.)

Open Philanthropy solicited reviews of my draft report

“Is power-seeking AI an existential risk?”

(Edit: arXiv version here) from various sources. Where the reviewers allowed us to make their comments public in this format, links to these comments are below, along with some responses from me in blue.

Leopold Aschenbrenner

Ben Garfinkel

Daniel Kokotajlo

Ben Levinstein
Eli Lifland

Neel Nanda

Nate Soares

Christian Tarsney

David Thorstad

David Wallace

Anonymous 1

(software engineer at AI research team)

Anonymous 2

(academic computer scientist)

The table below (spreadsheet link

here

) summarizes each reviewer’s probabilities and key objections.

Screenshot of linked summary spreadsheet

An academic economist focused on AI also provided a review, but they declined to make it public in this format.

Added 10/18/23: With funding from Open Philanthropy, Good Judgment also solicited reviews and forecasts from 21 superforecasters regarding the report -- see here for a summary of the results. These superforecasters completed a survey very similar to the one completed by the other reviewers, except with an additional question (see footnote) about the "

multiple stage fallacy

."[1] Their aggregated medians were:

Good Judgment has also prepared more detailed summaries of superforecaster comments and forecasts here (re: my report) and here (re: the other timelines and X-risk questions). See here for some brief reflections on these results, and here for a public spreadsheet with the individual superforecasters numbers and reviews (also screenshot-ed below).

^

The new question (included in the final section of the survey) was:

"One concern about the estimation method in the report is that the multi-premise structure biases towards lower numbers (this is sometimes called the "

multi-stage fallacy

"; see also Soares

here

). For example, forecasters might fail to adequately condition on all of the previous premises being true and to account for their correlations, or they might be biased away from assigning suitably extreme probabilities to individual premises.

When you multiply through your probabilities on the individual premises, does your estimate differ significantly from the probability you would've given to "existential cat

...
The "Other" Option
by jsteinhardt — 2021-12-16T20:20:29.611Z

For this post, I commissioned Misha Yagudin, a top-ranked forecaster, to provide feedback and commentary. I include selected comments as quotes throughout.

For open-ended questions, it's easy to underestimate the number of plausible outcomes. For instance, while practicing for the forecasting class, the course staff were asked to predict the following question:

Which NBA player will have the highest average points per game (PPG) between October 20th and October 27th?

To answer this question, I thought about star players who were top scorers, and also looked at who had high PPG in the previous season. The names I came up with were Giannis Antetokounmpo, Steph Curry, Luka Doncic, Kevin Durant, and LeBron James, so I made the following forecast:

Giannis 19%

Curry 17%

Doncic 16%

Durant 13%

LeBron 12%

Other 23%

In fact, the top scorer was Ja Morant, and the second highest scorer also wasn't on my list. In retrospect, 23% on "other" was a pretty bad prediction, and I should have put more like 60-70% on "other".

This is a general challenge with forecasting when the set of options is large. Even if each individual choice in the "other" category isn't that likely, there are a lot of choices in that category, and they add up to a large number. For instance, there are 30 teams in the NBA, so even counting just the best player on each team, my top-5 list was missing 83% of the options. Even if the other 25 starters were each only half as likely as LeBron to top the PPG, they would account for 66% of the probability mass.

Our tendency to underestimate "other" is worsened by the narrative fallacy, which is our cognitive tendency to frame events in some logical coherent order, and the disjunction fallacy, where the whole is often estimated to have lower probability than the parts. It's easy to imagine Curry scoring lots of points, but harder to imagine Ja Morant (although he is quite good), and even harder to imagine "some random player" scoring a lot.

Misha: You bring up misc cognitive biases. My default explanation would be availability bias. Per wikipedia: "Availability bias, is a mental shortcut that relies on immediate examples that come to a given person's mind when evaluating a specific topic, concept, method or decision."

On the other hand, there are times when "other" really does have a low probability. For instance, in a given week there's really only 1 or 2 movie...
What Caplan’s "Missing Mood" Heuristic Is Really For
by DirectedEvolution — 2021-12-16T19:47:45.747Z

Bryan Caplan wrote and quoted the idea of a "proper reaction," the "specific package of moods" that a "reasonable person" holding an intellectual position would have. If this mood is missing, it is grounds for suspicion, and we can "learn a lot" from it. It's a "valuable clue." The appropriate mood suggests credibility and truth. He claims that the missing mood heuristic is "fallible... but we all use it and we’re wise to do so."

He then gave examples of three moods missing from his political opponents, and two missing moods among his fellow pacifists and libertarians.

In the spirit of Paul Graham's disagreement hierarchy, let's identify the central point in Caplan's argument, and see if we can refute it.

Caplan's examples revolve around political and economic policy questions, so let's stay in that territory.

Let's start with an analogy that a proponent of war, immigration restriction, or regulation might hold. What if we compared these policies to chemotherapy? Although such treatments come with serious side effects, proponents believe that the benefits outweigh them.

We might accuse a flippant oncologist of a "missing mood," but this is as much because of the gravity of cancer as the side effects of chemotherapy. And there is a space of feistiness and humor that might be contextually appropriate.

My sense is that Caplan thinks that the chemotherapy analogy doesn't capture his central point, and I think I know why. With chemotherapy, the biomedical side effects and therapeutic benefits exclusively affect the patient.

By contrast, with war, regulation, and immigration restrictions, and with political policies in general, the costs and benefits of such policies are usually suffered or enjoyed by different groups of people. Furthermore, those people may have little or no say in whether or not those policies are enacted. Often, it is the people with the least political power who are made to suffer. These policies might be net good, but still unfair.

Therefore, I think that Caplan's "missing mood" probably most relevant when a policy proponent focuses entirely on the net benefit of the policy, and does not express any sense of empathy for those who the policy treats unfairly.

A reason to be worried about policy activists with a "missing mood" is that the beneficiaries of an unfair policy have selfish reason to ignore the costs of even net-negative policies. If such cost...
Subway Slides
by jefftk — 2021-12-16T19:30:03.764Z

Here is the current layout of

Davis Square station:

If you're coming in from the primary headhouse, on the right, your path is
really indirect:

I think the solution is clear:

Put a fare gate at the top, or switch the entire MBTA to industry best
practice proof-of-payment.

Comment via: facebook
Virulence Management
by harsimony — 2021-12-16T19:25:36.046Z
Link: https://harsimony.wordpress.com/2021/11/24/virulence-management/

Cross posted from my blog

I have spent some time reviewing the literature on an approach to dealing with human pathogens known as “virulence management”. Here, I hope to provide a jumping-off point to the field for others. I have included a list of relevant papers at the end.

Acquired Immunity

Our journey starts with the first of three papers by Stephen M. Hedrick on what he calls “Disease Ecology”. Of all of the papers I link here, his paper The Acquired Immune System : A Vantage from Beneath is the one I recommend the most.

The final line of the abstract is perhaps the most striking:

By selecting for ever-more-devious parasites, the acquired immune system is the cause of it’s own necessity.

What does he mean by this? Hedrick reminds us that acquired immunity is a relatively recent invention. While all animals are beset by parasites, invertebrates lack an adaptive immune system. How can they survive without an immune system that can adapt to new pathogens? Hedrick’s main point is: they don’t have to.

Because pathogens rely on their host to survive, they face a strong evolutionary pressure to avoid killing their host. In animals without an adaptive immune system, the host and pathogen find a stable equilibrium. As Hedrick points out, invertebrates are doing fine despite having parasites.

In animals with an acquired immune system (read: humans), the immune system is constantly adapting to new pathogens, forcing these pathogens to adapt as well. Pathogens mutate quickly and have short generation times in order to adapt to changes in the host immune environment. Hosts recombine genes using sexual reproduction along with an amazingly sophisticated immune system in order to combat this.

Hedrick argues that the massive number and diversity of pathogens combined with their short generation times makes the task of the adaptive immune system futile, we don’t have a way to permanently out-adapt such a versatile threat.

But if the invertebrates are doing fine without acquired immunity, how did we get locked into this endless struggle with pathogens?

Evolution doesn’t care about whether a trait is useful to a population, it only cares about you out-competing your neighbor. In this sense, adaptive immunity can be highly beneficial. Unfortunately, this fitness advantage meant that adaptive immunity spread to the entire population, locking us in a wasteful struggle against an e...
Omicron Post #7
by Zvi — 2021-12-16T17:30:01.676Z

Here are some better worlds we might have lived in, but don’t.

In a better world, I could focus on this full time and also maybe even hire a research assistant, and be better able to scour for information.
In a better world than that, there would be a department at a newspaper, or some other such place, that scoured the world for data sources like SGTF deletion counts and wastewater concentrations that could help us understand spread, called the sources to attempt to get the best data possible, and complied it all in nice form.
In an even better world than that, there would be funding, either public or private, for the actual gathering of additional data.
In an even better world than that, we’d have run challenge trials the moment Omicron was detected, and we’d already have all our answers.

Alas, instead, we live in this world, where none of those things are true. I am grateful to all my news sources and those who help gather the information together, and everyone working tirelessly in the lab, but the whole thing is still a slapstick hodgepodge, and we know so much less than we could know so much slower than we could know it.

Still, we do the best we can.

This time out, we have a few new places whose data has been located, to supplement the United Kingdom and Denmark. We also have several important new lab results.

There is some uncertainty about how far along things are. There is uncertainty regarding how much milder Omicron is than Delta, from it being essentially the same (but much better on a per-case basis due to who it infects) to there being a substantial or even large difference.

But the main uncertainty that I’m now wondering about, that feels central, is how we will react.

What will happen when the rice grains on the chessboard suddenly get fully out of hand, stuff hits the fan and the hospitals overflow? Not if. When. How will governments react? How will the people react?

Last year, I created a toy spreadsheet to model how the rise of Alpha might go, and whether the control system (governments and individuals reacting to the levels of cases, hospitalizations and deaths by scaling up or down prevention measures) combined with vaccinations could handle it. I invite everyone to copy, modify, mess around and find out, and create their own toy models. There are a bunch of things in column AH that you can modify to see what they imply. Note that t...
Where can one learn deep intuitions about information theory?
by Valentine — 2021-12-16T15:47:01.076Z

I'm currently going through Brilliant's course on "Knowledge and Uncertainty". I just got through the part where it explains what Shannon entropy is. I'm now watching a wave of realizations cascade in my mind. For instance, I now strongly suspect that the "deep law" I've been intuiting for years that makes evolution, economics, and thermodynamics somehow instances of the same thing is actually an application of information theory.

(I'm honestly kind of amazed I was able to follow as much of rationalist thought and Eliezer's writings as I was without any clue what the formal definition of information was. It looks to me like it's more central than is Bayes' Theorem, and that it provides essential context for why and how that theorem is relevant for rationality.)

I'm ravenous to grok more. Sadly, though, I'm bumping into a familiar wall I've seen in basically all other technical subjects: There's something of a desert of obvious resources between "Here's an article offering a quick introduction to the general idea using some fuzzy metaphors" and "Here's a textbook that gives the formal definitions and proofs."

For instance, the book "Thinking Physics" by Lewis Carroll Epstein massively helps to fill this gap for classical physics, especially classical mechanics. By way of contrast, most intro to physics textbooks are awful at this. ("Here we derive the kinematic equation for an object's movement under uniform acceleration. Now calculate how far this object goes when thrown at this angle at this velocity." Why? Is this really a pathway optimized for helping me grok how the physical world works? No? So why are you asking me to do this? Oh, because it's easy to measure whether students get those answers right? Thank you, Goodhart.)

Another excellent non-example is the Wikipedia article on how entropy in thermodynamics is a special case of Shannon entropy. Its length is great as a kind of quick overview, but it's too short to really develop intuitions. And it also leans too heavily on formalism instead of lived experience.

(For instance, it references shannons (= bits of information), but it gives no hint that what a shannon is measuring is the average number of yes/no questions of probability 1/2 that you have to ask to remove your uncertainty. Knowing that's what a shannon is (courtesy of Brilliant's course) gives me some hint about what a hartley (= base ten version...