trentmkelly/LessWrong-43k huggingface dataset:
Covid 7/14/22: BA.2.75 Plus Tax
by Zvi — 2022-07-14T14:40:00.587Z

That’s the new variant for this week. I am not worried about it, but there is little other news on the Covid front and I’m still attempting not to reuse titles. As time goes by the events are still new but they tend to rhyme. This update is inessential.

In addition to various non-Covid stuff I’m putting at the end, which I anticipate will slowly grow as a section over time, this week’s other post is Criticism of EA Criticism Contest. If you find that is relevant to your interests, with an additional copy on the EA forum. If it is not relevant to your interests you can safely skip it.

Executive Summary

BA.2.75 is potentially a thing, unclear if it is a meaningful thing yet.
Spike in deaths that is likely a data artifact.
A reminder we could be working faster towards

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 700k cases (+5%) and 2,250 deaths (-1%)

Results: 746k cases (+10%) and 2,885 deaths (+27%).

Prediction for next week: 750k cases (+1%) and 2,450 deaths (-15%).

With BA.5 taking over it makes sense for there to be a modest rise in total cases about now, so even though I think the jump was partly a holiday backfill my guess is we catch back up there. For deaths the details look a lot like backfills and spikes in reporting rather than an actual increase in deaths, so my guess is that it largely reverts but uncertainty prevents predicting a full reversion.

Deaths

Diving into the state-by-state data shows a bunch of places that are getting increasingly lazy with when they report deaths. A bunch of them happened to report substantial numbers of deaths this week (and Florida, which reported zero, was adjusted while the others weren’t because it’s not obvious how to do that properly). The regional numbers end up looking mostly sane aside from the Western spike.

Cases

An estimate that at least in the UK we’re catching almost none of the child cases anymore, and not that many adult ones either. Seems likely to be right.

BA.2.75 Our Price Cheap

BA.2.75 is suddenly something people are talking about as rising rapidly, a subvariant of BA.2. We will not be calling it Centaurus. Here we go again?

Go go gadget log scale.

And yet:

My guess is this probably isn’t going to be an impactful thing, to the extent it is a thing at all. If it is, w...
Criticism of EA Criticism Contest
by Zvi — 2022-07-14T14:30:00.782Z

Back when it was announced, I toyed with the idea of criticizing the EA criticism and red teaming contest as an entry to that contest, leading to some very good Twitter discussion that I compiled into a post.

I finally found myself motivated to write up my thinking in detail.

Before I begin, I recommend reading the contest announcement yourself, and forming your own reaction. Ask yourself, among other things:

What are they telling people they are interested in and likely to reward?
What are they not interested in and telling people they will not reward?
What does this announcement say more generally about EA?

Praise First

I will start with some things that I think are good about the contest and announcement.

It exists at all. It is good to solicit (paid!) public criticism.
It awards at least $100,000. That’s good prize money.
It promises an even bigger prize for something that really hits home.
It suggests it will reward causing people to change their minds.
Guidelines offered to help with techniques includes red teaming.
The exercise I read this as being is still worth doing, if not as worthwhile as the thing I’d rather be happening, so long as it doesn’t create a story that the thing I want to happen did indeed happen and is thus handled, when it isn’t handled.
What actually wins will be an informative experiment. Stay tuned.

Core Critique

Effective Altruism has a core set of assumptions, and a core method of modeling and acting in the world.

The core critique I am offering is that the contest is mostly taking things like the list below as givens, rather than as things to be questioned.

It is sending a set of signals that such punches should be pulled, both for the contest and in general. Effective Altruism, in this model, very much wants criticism of its tactics, and mostly wants them also of its strategy, but only within the framework.

This parallels EA more broadly, where within-paradigm critiques are welcomed (although, as everywhere, often incorrect disregarded) but deeper critiques are unwelcome, and treated as mistakes.

Here is my attempt to summarize the framework, which came out to 21 points.

Note that the rest of the post does not depend on the exact points or their wordings. Although I did attempt to make the list match my perceptions as much as possible, the list is included primarily to give an idea of the type of thing that is assumed here.

Utilitarianism. Alterna

...
Humans provide an untapped wealth of evidence about alignment
by TurnTrout — 2022-07-14T02:31:48.575Z

This post has been recorded as part of the LessWrong Curated Podcast, and can be listened to on Spotify, Apple Podcasts, and Libsyn.

TL;DR: To even consciously consider an alignment research direction,

you should have evidence

to locate it as a promising lead. As best I can tell, many directions seem interesting but do not have strong evidence of being “entangled” with the alignment problem such that I expect them to yield significant insights.

For example, “we can solve an easier version of the alignment problem by first figuring out how to build an AI which maximizes the number of real-world diamonds” has intuitive appeal and plausibility, but this claim doesn’t have to be true and this problem does not necessarily have a natural, compact solution. In contrast, there do in fact exist humans who care about diamonds. Therefore, there are guaranteed-to-exist alignment insights concerning the way people come to care about e.g. real-world diamonds.

“Consider how humans navigate the alignment subproblem you’re worried about” is a habit which I (TurnTrout) picked up from Quintin Pope. I wrote the post, he originated the tactic.

A simplified but still very difficult open problem in

AI alignment

is to state an unbounded program implementing a

diamond maximizer

that will turn as much of the physical universe into diamond as possible. The goal of "making diamonds" was chosen to have a crisp-seeming definition for our universe (the amount of diamond is the number of carbon atoms covalently bound to four other carbon atoms). If we can crisply define exactly what a 'diamond' is, we can avert issues of trying to convey

complex values

into the agent.

Ontology identification problem

, Arbital

I find this problem interesting, both in terms of wanting to know how to solve a reframed version of it, and in terms of what I used to think about the problem. I used to[1] think, “yeah, ‘diamond’ is relatively easy to define. Nice

problem relaxation

.” It felt like the diamond maximizer problem let us focus on the challenge of making the AI’s values bind to something at all which we actually intended (e.g. diamonds), in a way that’s robust to ontological shifts and that doesn’t collapse into wireheading or tampering with e.g. the sensors used to estimate the number of diamonds.

Although the details are mostly i...
How to impress students with recent advances in ML?
by Charbel-Raphaël — 2022-07-14T00:03:04.883Z

I am preparing resources for an equivalent of the mlab camp, organized by EffiSciences, that will be held in France in a month to train students to become prosaic alignment researchers.

3/4 of the participants are already aligned with existential risks related to AI, but 1/4 of the students have been selected for their technical skills and epistemologies, but are not yet familiar with the AI safety issue.

We have the intuition that one of the main reasons why they do not consider this issue is their lack of knowledge about the very recent capabilities of AI.

If you had 30 minutes to impress them with the advances in deep learning and the speed at which the field is progressing, what resources would you recommend? What should I present? How would you pitch this?
Notes on Love
by David Gross — 2022-07-13T23:35:55.224Z

This post examines love as a virtue, as part of a sequence of posts about virtues. I mostly explore what other people have learned about love, rather than sharing my own research or opinions about it, though I’ve been selective about what I found interesting or credible, and my research has been scattershot. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will help people who want to know more about love and how to practice it skillfully.

What is this thing called love?

“Love” suffers from more than its share of terminological difficulty. People use the word to refer to all sorts of things—including sensations, emotions, dispositions, commitments, actions, judgments, predicaments, opinions, principles, personified or deified forces, and people. The love that people experience when they fall-in-love is very different from the Christian “love your enemies” sort of love or the love that parents have for their children or the love that a devotee has for God or the love that a gourmand has for pancetta or the love that a patriot has for her country or the love that misery has for company.

And I don’t think there is a primary meaning of love, with a bunch of secondary meanings hooked to it metaphorically. As a quick sanity-check, I did an informal poll of Facebook friends in which I asked, without further context, how they would sum up “love”. About half of the serious responses described love as a feeling. Another 25% described it as a sort of disposition (e.g. “the desire for the beloved to thrive,” “not thinking twice about sticking around during the tough times”). A few described it as a sort of activity (e.g. “helping”). Two examples did not attempt to describe what love is, but instead described its place in your life (“love completes you”, “it’s all you need”). Some answers were ambiguous (e.g. is “caring” a feeling or an action?). This reinforced my impression that “love” does not have an obvious central definition with other secondary meanings, but that it is a scrambled mishmash of a lot of things.[1]

While love is not well-defined it is nonetheless a very popular topic of inquiry in just about every literary variety. It is nigh impossible to do even a cursory literature survey of the topic. As a result, mine has been eccentric and there’s a good chance I would have come up with some very different concl...
Deep learning curriculum for large language model alignment
by Jacob_Hilton — 2022-07-13T21:58:33.452Z
Link: https://github.com/jacobhilton/deep_learning_curriculum

This is a deep learning curriculum with a focus on topics relevant to large language model alignment. It is centered around papers and exercises, and is biased towards my own tastes.

It is targeted at newcomers to deep learning who are already familiar with the basics, but I expect it to be pretty challenging for most such people, especially without mentorship. So I have included some advice about prerequisites and more accessible alternatives.
Artificial Sandwiching: When can we test scalable alignment protocols without humans?
by Sam Bowman — 2022-07-13T21:14:08.145Z

Epistemic status: Not a fleshed-out proposal. Brainstorming/eliciting ideas.

Thanks to Ben Mann, Pablo Moreno, and Jared Kaplan for feedback on early drafts.

Overview

I’m convinced sandwiching—the experimental protocol from Ajeya Cotra’s The case for aligning narrowly superhuman models—is valuable, and I’m in the process of setting up some concrete sandwiching experiments to test scalable oversight ideas.
Sandwiching experiments are generally fairly slow:

You have to design and pilot a strategy that allows humans to use (or oversee) a model for a task that they can’t do well themselves. The details matter here, and this can often take many iterations to get right.
Then, you need a bunch of humans actually try this. Even for very simple tasks, this is a high-cognitive-load task that should take at least tens of minutes per instance.
You have to repeat this enough times to measure average performance accurately.

I’m visiting Anthropic this year for a sabbatical, and some of my sandwiching work is happening there. Anthropic’s biggest comparative advantage (like that of similar teams at DeepMind and OpenAI) is easy access to near-state-of-the-art LMs that are fine-tuned to be helpful dialog agents.
In that context, I've heard or encountered this question several times: Can we speed up [some experiment I’m proposing] by replacing the non-expert human with a weaker LM?

This obviously doesn’t achieve the full aims of sandwiching in general, but it’s often hard to find a decisive rebuttal for these individual instances.
More broadly, I think there’s likely to be a significant subset of worthwhile sandwiching experiments that can be trialed more quickly by using an intentionally weakened model as a proxy for the human.
Which experiments these are, precisely, has been hard for me to pin down. This post is an attempt to organize my thoughts and solicit comments.

Background: Standard sandwiching (in my terms)

Prerequisites:

A hard task: A task that many humans would be unable to solve on their own.
A capable but misaligned language model assistant: A model that appears to have the skills and knowledge needed to solve the task better than many humans, but that does not reliably do so when prompted.

A non-expert human: Someone who can’t solve the task on their own, but will try to solve it using the assistant and some scalable alignment strategy.
[Secondary] Expert human: Someo

...
Any tips for eliciting one's own latent knowledge?
by MSRayne — 2022-07-13T21:12:13.542Z

I struggle to write. I cannot figure out how to access my own thoughts and ideas. They come in non-linguistic forms - vague feelings, 3D images, animations - with a verbal narrator on top, but my very bad memory, and the fact that my best thinking is done in the middle of the night when I ought to be asleep, means that I end up forming intuitions about things based upon my prior thoughts, but I'm unable to remember what those thoughts were, or even what intuitions have recently formed how. Instead, when something reminds me of a topic, I remember whatever is salient that I've already thought of at some unknown prior time.

I don't possess a map of my own mind, nor even a search function. Just a mysterious black box. My mind is totally disorganized. And when I look at a blank page, I have no idea what to write, where to start. I don't even know what questions to ask myself to prompt writing, nor how to answer any. I honestly have no idea how to think clearly. Particularly from a blank slate, without prompting or real-time communication with another person, who can dig things out of me I cannot retrieve on my own, by reminding me of things with their questions.

I've heard of the concept of "solve the whole problem day", but alignment itself is not in my wheelhouse - I'm more of a "philosopher" (though that's much too pretentious to call myself), and my "problems" are vague and undefined - nebulous clouds of concepts on which I fixate, returning again and again, like the nature (or, phenomenology?) of identity, boundaries, emotions - or plans for a digital platform for increasing collective intelligence, rationality, and coordination. And my "plans" are composed of an endless array of disconnected fragments I cannot put together - because, again, specific details only show up in my mind temporarily like electrons falling out of superposition, and then they return to being a cloudy orbital of pure intuition.

Does anyone else know what this feels like and have any tips for me? It's maddening, and I've been suffering from it for basically my whole life. I've tried using the Zettelkasten system to "organize" my thoughts but it doesn't work for me, the result is just a tangled mess as bad as my real brain, no easier to parse through. I want to be of use to this site, providing interesting perspectives, particularly in the search for "True Names" and the design of suppor...
Goal Alignment Is Robust To the Sharp Left Turn
by Thane Ruthenis — 2022-07-13T20:23:58.962Z

A central AI Alignment problem is the "sharp left turn" — a point in AI training under the SGD analogous to the development of human civilization under evolution, past which the AI's capabilities would skyrocket. For concreteness, I imagine a fully-developed mesa-optimizer "reasoning out" a lot of facts about the world, including it being part of the SGD loop, and "hacking" that loop to maneuver its own design into more desirable end-states (or outright escaping the box). (Do point out if my understanding is wrong in important ways.)

Certainly, a lot of proposed alignment techniques would break down at this point. Anything based on human feedback. Anything based on human capabilities presenting a threat/challenge. Any sufficiently shallow properties like naively trained "truthfulness". Any interpretability techniques not robust to deceptive alignment.

One thing would not, however, and that is goal alignment. If we can instill a sufficiently safe goal into the AI before this point — for a certain, admittedly hard-to-achieve definition of "sufficiently safe" — that goal should persist forever.

Let's revisit the humanity-and-evolution example again. Sure, inclusive genetic fitness didn't survive our sharp left turn. But human values

did

. Individual modern humans are optimizing for them as hard as they were before; and indeed, we aim to protect these values against the future. See: the entire AI Safety field.

The mesa-optimizer, it seems obvious to me, would do the same. The very point of various "underhanded" mesa-optimizer strategies like deceptive alignment is to protect its mesa-objective from being changed.

What it would do to its mesa-objective, at this point, is goal translation: it would attempt to figure out how to apply its goal to various other environments/ontologies, determine what that goal "really means", and so on.

Open Problems

There are three hard challenges this presents, for us:

Figure out an aligned goal/a goal with an "is aligned" property, and formally specify it.

Either corrigibility or CEV, or some clever "pointer" to CEV.
Requires a solid formal theory of what "goals" are.

Figure out how to instill an aligned goal into a pre-sharp-left-turn system.

Requires a solid formal theory of what "goals" are, again.
I think robust-to-training interpretability/tools for manual NN editing are our best bet for the "instilling" part.[1] Good news is th

...
Making decisions using multiple worldviews
by Richard_Ngo — 2022-07-13T19:15:02.621Z

Tl;dr: the problem of how to make decisions using multiple (potentially incompatible) worldviews (which I'll call the problem of meta-rationality) comes up in a range of contexts, such as epistemic deference. Applying a policy-oriented approach to meta-rationality, and evaluating worldviews by the quality of their advice, dissolves several undesirable consequences of the standard "epistemic" approach to deference.

Meta-rationality as the limiting case of separate worldviews

When thinking about the world, we’d ideally like to be able to integrate all our beliefs into a single coherent worldview, with clearly-demarcated uncertainties, and use that to make decisions. Unfortunately, in complex domains, this can be very difficult. Updating our beliefs about the world often looks less like filling in blank parts of our map, and more like finding a new worldview which reframes many of the things we previously believed. Uncertainty often looks less like a probability distribution over a given variable, and more like a clash between different worldviews which interpret the same observations in different ways.

By “worldviews” I include things like ideologies, scientific paradigms, moral theories, perspectives of individual people, and sets of heuristics. The key criterion is that each worldview has “opinions” about the world which can be determined without reference to any other worldview. Although of course different worldviews can have overlapping beliefs, in general their opinions can be incompatible with those of other worldviews - for example:

Someone might have

severe uncertainty

about

far-reaching empirical claims

, or about

which moral theory

to favor.
A scientist might be investigating a phenomenon during a

crisis period

where there are multiple contradictory frameworks which purport to explain the phenomenon.
Someone might buy into an ideology which says that nothing else matters except adherence to that ideology, but then feel a “common-sense” pull towards

other perspectives

.
Someone might have a strong “inside view” on the world, but also want to partly

defer

to the worldviews of trusted friends.
Someone might have a set of principles which guides their interactions with the world.
Someone might have different parts of themselves which care about different things.

I think of “intelligence” as the core ability to develop and merge worldvi...
App idea to help with reading STEM textbooks (feedback request)
by DirectedEvolution — 2022-07-13T18:28:06.505Z

Problem: STEM textbooks often reference figures and equations from earlier in the textbook. They usually do this with statements like "the shearing stress τ... may be obtained from the shearing-stress-strain diagram of Fig. 3.30." But Fig. 3.30 might be from 5 pages ago, or even earlier. Many textbooks don't have a list or index of equations, so the only way to find the referenced figure is to search page by page.

How this would ideally be solved: Every textbook would have an e-book version that links to, and previews, referenced equations every time they are mentioned.

Why not do this: Different textbooks identify and reference figures and equations different ways. Many readers of e-books use DRM-protected apps. These make it hard to create a universal solution that automatically identifies and annotates texts to provide this functionality. In addition, building an e-reader with this functionality requires building an e-reader to display PDFs and other formats, which seems very hard.

How I suspect most people solve the problem: Most people probably deal with this more or less like I do. The struggle to be diligent and actually look up the referenced figure. They might try to memorize the information. They also might put it into a list of reference notes containing the equations.

Why these solutions are suboptimal: Trying to look up or memorize information is cognitively and motivationally challenging. Learning dense material is already hard, and this adds to the burden. Taking notes can help. However, this leads to visual clutter when most equations aren't necessary at a given time, or when material that needs to be looked at simultaneously is distributed across different pages of notes. It's not always easy to get the required information into the note-taking system. Some note-taking systems also have trouble managing the sort of information a STEM student might need.

How this new app would solve the problem: This app depends on a "snip" feature, where we take a screenshot of part of the screen. Students would read an e-textbook, and specify in the app a collection of tags under which all new "infosnips" will be categorized (i.e. "Mechanics of Materials"). When it presents a new equation or figure, they take a "snip" of it, including its caption. Either instantly, or at the press of a hotkey, the app uses image-to-text to identify the reference number (i.e. the...