trentmkelly/LessWrong-43k huggingface dataset:
What ethical thought experiments can be reversed?
by Mati_Roy — 2021-03-06T15:12:09.826Z

Some thought experiments follow this template:

We have a moral intuition

We make some computation to what this intuition implies

We check how we feel about this implication, and it feels counter-intuitive

Then some people bite the (3) bullet. But bullets sometimes (always?) have a counter-bullet.

You can reverse those thought experiments: take ~(3) as your starting moral intuition, and then derive ~(1) which will be counter-intuitive.

For example, you can start with:

I would care about saving a drowning person even if it came at the cost of ruining my suit

There are a lot of metaphorically drowning people in the world

Therefore I should donate all my money to effective poverty alleviation charities

This is called "shut up and multiply".

But you can also use the reverse:

I don't want to donate all my money to effective poverty alleviation charities

A drowning person would cost more to save because it would ruin my suit

Therefore I shouldn't save a drowning person

This is called "shut up and divide" (also related: Boredom vs. Scope Insensitivity).

Step (2) might be eliminating a relevant feature which generates the counter-intuition, or it might be a way to open our eyes to something we were not seeing. And maybe for some thought experiment you find both the assumption and conclusion intuitive or counterintuitive. But that's not the object of this post.

Here I'm just interested in seeing what the reverse of ethical thought experiments look like. I'll put more examples as answer. I would like to know which other ethical thought experiments have this pattern -- that is, an ethical thought experiment that starts with an intuition to derive a counter-intuition, which can be reversed, to instead derive that the initial assumption is the wrong one.

Update: As I'm writing some of them, I realized some ethical thought experiment are presented as a clash of intuitions (so the "reverse" is part of the original presentation), whereas others seem to be trying to persuade the reader to bite the bullet on a certain counter-intuition, and omit to mention the reverse ethical thought experiment.
What (feasible) augmented senses would be useful or interesting?
by AprilSR — 2021-03-06T04:28:13.320Z

The BrainPort is a device which allows blind people to have their sight restored (partially) by sending signals through electrodes that rest on the tongue.

The signals are sent to the tongue via a "lollipop," an electrode array about nine square centimeters that sits directly on the tongue. Each electrode corresponds to a set of pixels. White pixels yield a strong electrical pulse, whereas black pixels translate into no signal. Densely packed nerves at the tongue surface receive the incoming electrical signals, which feel a little like Pop Rocks or champagne bubbles to the user.
Seiple works with four patients who train with the BrainPort once a week and notes that his patients have learned how to quickly find doorways and elevator buttons, read letters and numbers, and pick out cups and forks at the dinner table without having to fumble around.

- Scientific American

The brain seems to be remarkably good at reinterpreting sensory signals. What possible applications are there of this, especially which might be practically implemented by an average person today without needing to develop specialized hardware?
Check OK, babble-read, optimize (how I read textbooks)
by DirectedEvolution — 2021-03-06T03:17:22.325Z

A year ago, I didn't have any conscious ideas for how to read a textbook. I was frustrated by forgetting things, and had heard about memory palaces. When I was playing classical piano, I was able to memorize long complicated pieces of music with thousands of notes. The nice thing about piano music, besides the fact that it gets your muscles and ears involved, was that I had developed a set of methods for breaking down a piece and learning it effectively. I didn't really have any equivalent for book-learning.

After experimenting with memory palaces, I found that they just weren't getting me where I needed to go, although the practice at visualizing has proved extremely valuable. This post explains how I read now.

Level 1: The "check OK" method

Many sentences contain information that makes plain sense. You might not know which bits are going to be important. You might not know the point it's trying to make yet. And you're not necessarily committing to trying to memorize it, or even take notes on it. But you get it. You read it, and you think, "mmm-hmm, OK." An example of such a sentence is:

"Glucose is the major carbohydrate fuel for many cells."

Whole paragraphs can be composed of sentences like this. They're either not telling you anything new, or else they're so detailed that you consider them to be beyond what you need to remember.

At Level 1, you're checking for that "mmm-hmm, OK" reaction, so I call this the "check OK" method of reading. It's a safeguard against bleary-eyed skimming, and it's also meant to help you when the textbook is trying to make a point that actually requires some deeper comprehension.

Level 2: Babble-reading

The Pasteur effect involves the inhibition of glycolysis by oxygen. This effect makes biological sense because far more energy is derived from the complete oxidation of glucose than from glycolysis alone. What is the mechanism of this effect if oxygen is not an active participant in glycolysis?

If you're taking a biochemistry course, these sentences might be new information to you. Hopefully, if you were using the "check OK" method, you had an intuitive feeling of not fully understanding. "The inhibition of glycolysis by oxygen" is a pretty complicated idea. You just learned about glycolysis. Inhibition of glycolysis... what does that mean? And where does oxygen fit into the picture?

You go on to see that this "makes sense." The textbook...
MIRI comments on Cotra's "Case for Aligning Narrowly Superhuman Models"
by Rob Bensinger — 2021-03-05T23:43:54.186Z

Below, I’ve copied comments left by MIRI researchers Eliezer Yudkowsky and Evan Hubinger on March 1–3 on a draft of Ajeya Cotra’s "

Case for Aligning Narrowly Superhuman Models

." I've included back-and-forths with Cotra, and interjections by me and Rohin Shah.

The section divisions below correspond to the sections in Cotra's post.

0. Introduction

How can we train GPT-3 to give “the best health advice it can give” using demonstrations and/or feedback from humans who may in some sense “understand less” about what to do when you’re sick than GPT-3 does?

Eliezer Yudkowsky: I've had some related conversations with Nick Beckstead. I'd be hopeful about this line of work primarily because I think it points to a bigger problem with the inscrutable matrices of floating-point numbers, namely, we have no idea what the hell GPT-3 is thinking and cannot tell it to think anything else. GPT-3 has a great store of medical knowledge, but we do not know where that medical knowledge is; we do not know how to tell it to internally apply its medical knowledge rather than applying other cognitive patterns it has stored. If this is still the state of opacity of AGI come superhuman capabilities, we are all immediately dead. So I would be relatively more hopeful about any avenue of attack for this problem that used anything other than an end-to-end black box - anything that started to address, "Well, this system clearly has a bunch of medical knowledge internally, can we find that knowledge and cause it to actually be applied" rather than "What external forces can we apply to this solid black box to make it think more about healthcare?"

Evan Hubinger: +1 I continue to think that language model transparency research is the single most valuable current research direction within the class of standard ML research, for similar reasons to what Eliezer said above.

Ajeya Cotra: Thanks! I'm also excited about language model transparency, and would love to find ways to make it more tractable as a research statement / organizing question for a field. I'm not personally excited about the connotations of transparency because it evokes the neuroscience-y interpretability tools, which don't feel scalable to situations when we don't get the concepts the model is using, and I'm very interested in finding slogans to keep researchers focused on the superhuman stuff.

Ajeya Co...
The case for aligning narrowly superhuman models
by Ajeya Cotra — 2021-03-05T22:29:41.577Z

I wrote this post to get people’s takes on a type of work that seems exciting to me personally; I’m not speaking for Open Phil as a whole. Institutionally, we are very uncertain whether to prioritize this (and if we do where it should be housed and how our giving should be structured). We are not seeking grant applications on this topic right now.

Thanks to Daniel Dewey, Eliezer Yudkowsky, Evan Hubinger, Holden Karnofsky, Jared Kaplan, Mike Levine, Nick Beckstead, Owen Cotton-Barratt, Paul Christiano, Rob Bensinger, and Rohin Shah for comments on earlier drafts.

A genre of technical AI risk reduction work that seems exciting to me is trying to align existing models that already are, or have the potential to be, “superhuman”[1] at some particular task (which I’ll call narrowly superhuman models).[2] I don’t just mean “train these models to be more robust, reliable, interpretable, etc” (though that seems good too); I mean “figure out how to harness their full abilities so they can be as useful as possible to humans” (focusing on “fuzzy” domains where it’s intuitively non-obvious how to make that happen).

Here’s an example of what I’m thinking of: intuitively speaking, it feels like GPT-3 is “smart enough to” (say) give advice about what to do if I’m sick that’s better than advice I’d get from asking humans on Reddit or Facebook, because it’s digested a vast store of knowledge about illness symptoms and remedies. Moreover, certain ways of prompting it provide suggestive evidence that it could use this knowledge to give helpful advice. With respect to the Reddit or Facebook users I might otherwise ask, it seems like GPT-3 has the potential to be narrowly superhuman in the domain of health advice.

But GPT-3 doesn’t seem to “want” to give me the best possible health advice -- instead it “wants” to play a strange improv game riffing off the prompt I give it, pretending it’s a random internet user. So if I want to use GPT-3 to get advice about my health, there is a gap between what it’s capable of (which could even exceed humans) and what I can get it to actually provide me. I’m interested in the challenge of:

How can we get GPT-3 to give “the best health advice it can give” when humans[3] in some sense “understand less” about what to do when you’re sick than GPT-3 does? And in that regime, how can we even tell whether it’s actually “doing the best it can”?

I th...
D&D.Sci III: Mancer Matchups
by abstractapplic — 2021-03-05T19:07:17.473Z

Note: While not exactly *bad*, I believe - and the upvotes confirm - that this is my weakest D&D.Sci challenge by a considerable margin. If you're currently working through the archive, I'd recommend playing everything else before resorting to this one.

The voice that greets you from the darkness outside the summoning circle is low, gravelly, and – in your opinion – completely wasted on the polite and unthreatening tone its owner adopts.

“Hello there. First off, sorry if I interrupted anything by conjuring you like this. I’m rather new to the whole consorting-with-demons thing, so if I was supposed to book an appointment then, ah, mea culpa.”

Upon meeting you, most mortals immediately launch into a list of demands. This is odd, but you must admit, it’s a nice change of pace.

“Second off, you should know that I can’t see or hear you. Probably paranoid, but you lot are supposed to be worryingly good with words, and I’d hate to be talked out of my soul because the alternative was having a conversation be a bit one-sided.”

Prudent, but then how does he expect you to-

“Third off, the reason you’re here. There’s a war on, and the good wizards – the Vitamancers, the Geomancers, the Cryomancer – are trying to stop the Pyromancers and Necromancers taking over this plane of existence. I’ve been keeping a record of who wins what fights under what circumstances, but whenever anyone tries to use it to strategize, they get accused of trying to allocate themselves the easy jobs and there’s a big argument and then we go back to picking targets based on, ah, other factors.”

There’s a pause as he telepathically transmits his record of wins and losses to you. You have time to approvingly note the anonymization – apparently mortals have finally learned to not share their true names with your kind – before he starts talking again.

“But for the next set of fights, it’s really rather urgent that we get as many wins as possible. So we decided to bring in an external consultant to impartially decide whom to send where, for this round in particular.”

Translation: ‘we decided summoning a demon and doing whatever it says would be easier and safer than trying to address our intra-faction conflicts; and after the current crisis is over, we plan to go back to our dysfunctional approach’. For all his cleverness and caution, your summoner doesn’t seem able to stop bleeding information; you would...
GPT-3 and the future of knowledge work
by fowlertm — 2021-03-05T17:40:12.039Z

The most recent episode of the Futurati Podcast is a big one. We had Jungwon Byun and Andreas Stuhlmüller on to talk about their startup 'Ought' and, to the best of my knowledge, this is the first public, long-form discussion of their work around.

(It's also probably our funniest episode.)

Their ambition is to wrap a sleek GUI around advanced language models to build a platform which could transform scholarship, education, research, and almost every other place people think about stuff.

The process is powered by GPT-3, and mostly boils down to teaching it how to do something you want it to do by showing it a couple of examples. To complete a list of potential essay topics you'd just show it 3-4 essay topics, and it'd respond by showing you a few more.

The more you interact with it, the better it gets.

There's all sorts of subtlety and detail, but that's the essence of it.

This may not sound all that impressive, but consider what it means. You can have Elicit (a separate spinoff of Ought) generate counterarguments to your position, brainstorm failure modes (and potential solutions) to a course of action, summarize papers, and rephrase a statement as a question or in a more emotionally positive tone.

The team is working on some integrations to extend these capabilities. Soon enough, Elicit will be able to connect to databases of published scientific papers, newspapers, blogs, or audio transcripts. When you ask it a research question, it'll be able to link out to millions of documents and offer high-level overviews of every major theme; it'll be able to test your comprehensions by asking you questions as you read; it'll be able to assemble concept hierarchies; it'll be able to extract all the figures from scientific papers and summarize them; it'll be able to extract all the proper names, find where those people are located, get their email addresses where available, and write them messages inviting them on your podcast.

We might one day be able to train a model on Einstein or Feynman and create lectures in their style.

What's more, people can share workflows they've developed. If I work out a good approach to learning about the subdisciplines of a field, for example, I can make that available to anyone to save them the effort of discovering it on their own.

There will be algorithms of thought that can make detailed, otherwise inaccessible aspects of other people's ...
Takeaways from the Intelligence Rising RPG
by Quinn — 2021-03-05T10:27:55.867Z

Last Saturday, we played Intelligence Rising. Intelligence Rising is a role-playing game about the global development of transformative AI technologies, where each player or team is a major government or company. We had a great time and we highly recommend you seize any opportunity to play it!

What follows is the GM's summary of the game followed by some of our own perspectives and takeaways.

The participants were:

Game Master: Ross Gruetzemacher

USA: Quinn Dougherty & Roland Pihlakas

China: Remmelt Ellen

Tencent: Ben Greenberg

Alphabet: Viktor Rehnberg

Ross' (Game Master) Account

Early Game

Players learned the rules and how to effectively use their talents and actions. China was an experienced player and began using secret moves with large amounts of resources from the very beginning. The large resources used by China led to recurring benefits that yielded a powerful cyber advantage despite dice rolls not being successful in the first two rounds.

Midgame

The United States led an effort to ensure cooperation on AI safety research and to try to stop the development of any advanced AI capabilities via an international agreement. These efforts stretched over six years, and ultimately led to an agreement between all major actors to stop the vast majority of capabilities research and focus on safety. There were some limitations to the early agreement, like the lack of an arbiter or oversight assurances for governments, and the lack of a broader coalition of AI research companies (i.e., non-player characters) to sign the original agreement. There was a severe conflict in the South China Sea off the coast of Taiwan as a result of posturing from the Chinese, but the conflict deescalated quickly without growing into broader war because the international AI safety agreement required peace to move forward.

Lategame

Alphabet unilaterally violated the international agreement and pursued AGI, unlocking the AGI capability card. This led to dramatic chaos in the United States due to their failure to honor the agreement they sponsored coupled with Alphabet's seemingly imminent development of AGI. This took everyone by surprise, and the world was united in opposing the US's actions. Tom Cotton won the presidential election due in large part to this, further complicating the situation. In the US the right and left were both protesting leading to riots and terrorism.

In resp...
Multimodal Neurons in Artificial Neural Networks
by Kaj_Sotala — 2021-03-05T09:01:53.996Z
Link: https://distill.pub/2021/multimodal-neurons/

A paper investigating how individual neurons in a CLIP model (an image/text neural net combining a ResNet vision model with a Transformer language model) respond to various abstract concepts. This shouldn't be very surprising after GPT-3 and DALL-E but still, identifying multimodal neurons feels scarily close to "neural net that understands abstract concepts" and thus AGI for my comfort.

Some individual neurons that they isolated (see the article for more):

Spiderman neuron: responds to photos of Spiderman in costume and spiders, comics or drawings of Spiderman and spider-themed icons, the text “spider” and others. Associates him with "Peter Parker" and also responds to images, text, and drawings of heroes and villains from Spiderman movies and comics over the last half-century.
Yellow neuron: responds to images of the words “yellow”, “banana” and “lemon,” in addition to the color.
Jesus Christ neuron: detects Christian symbols like crosses and crowns of thorns, paintings of Jesus, his written name, and feature visualization shows him as a baby in the arms of the Virgin Mary.
Hitler neuron: learns to detect his face and body, symbols of the Nazi party, relevant historical documents, and other loosely related concepts like German food. Feature visualization shows swastikas and Hitler seemingly doing a Nazi salute.
Donald Trump neuron: strongly responds to images of him across a wide variety of settings, including effigies and caricatures in many artistic mediums, as well as more weakly activating for people he’s worked closely with like Mike Pence and Steve Bannon. It also responds to his political symbols and messaging (eg. “The Wall” and “Make America Great Again” hats). On the other hand, it most *negatively* activates to musicians like Nicky Minaj and Eminem, video games like Fortnite, civil rights activists like Martin Luther King Jr., and LGBT symbols like rainbow flags.
Happiness neuron: responds both to images of similing people, and words like “joy.”
Surprise neuron: responds to images of surprised people, and to slang like "OMG!" and "WTF", and text feature visualization produces similar words of shock and surprise.
Mental illness neuron: activates when images contain words associated with negative mental states (eg. “depression,” “anxiety,” “lonely,” “stressed”), words associated with clinical mental health treatment (“psychology”, “mental,” “disorder”, “th

...
Does making unsteady incremental progress work?
by peterbarnett — 2021-03-05T07:23:30.338Z
Link: https://peterbarnett.org/2021/03/05/unsteady-progress/

There's a commonly repeated piece of wisdom about incremental improvement which says it's important to know the difference between 1.01x and 0.99x as x gets very large. If you are making 1% improvement per day at a thing, over a year your progress goes like this (note that because of decreasing marginal returns your metrics probably won't go like this, but skill using some scale will):

While if you are neglecting a skill and so losing 1% a day your progress will go like this:

This is true and good in the world of maths; if you make steady incremental progress you keep improving, your gains compound and you shoot to the moon, while if you slowly decrease then things fall to zero. And this is kind of just a 2% difference, the difference between +1% and -1% per day.

I was a bit dubious of this advice for the "real world", because in the real world making 1% progress every day is often really hard. Sometimes you make big leaps, sometimes you fall back a bit. So rather than doing 1.01×1.01×1.01×1.01×⋯×1.01, it's more like 1.0055×1.0163×0.986×1.0246×⋯×1.0079.

So I decided to test this, drawing the percent improvement from a distribution which was centered on 1%, but could be bigger or smaller:

With this varying, but still on-average positive improvement, a single trajectory looks like this:

This looks good! It increases similarly to the case without any variation. This is pretty robust when we run the same thing a bunch of times:

Here the rainbow lines are individual runs with variation, and the solid black line is the case with no variation. With the randomness there is a bit more uncertainty about where it ends up, but it's still broadly clustered around the original, and we don't see any of the samples collapse to zero. The final distribution of improvement after 365 days looks like this:

Still centered around the same value but with a fair bit of variation. And in fact, a slightly longer tail on the right; so with random variation you're likely to end up better than the no-variation case... but this definitely relies on a bunch of assumptions and probably isn't very robust.

I guess the moral of the story is that it's great to make steady incremental progress every day, but it's also good to make unsteady progress as long as on average you improve every day.