trentmkelly/LessWrong-43k huggingface dataset:
Unforgivable
by Novalis — 2022-06-25T20:57:47.297Z
Link: https://novalis.blog/unforgivable/

There are few things that strike me as being completely unforgivable. One of those things is how individuals and society as a whole treats children and adolescents.

Be it deliberately stripping them of agency on all matters in their everyday life, not out of benevolence towards them, but out of convenience.

Having them, under threat of consequences, devour continuously to the pursuit of what the adults in their life find is important for them. Knowing that they'll loathe the sight of these things, whether actually important or not, for the rest of their life.

Or requiring them to put up with the tight-fitting encasement of school, which, being like the shoes of a mandarin woman, pinch and bruise the nature of children on all sides and at every turn.

I am not the only one who believes this. In fact, if you look at the autobiographies of a wide variety of people, you'll come across these ideas again and again. It almost seems that every great personality at some point in their lives has written a critique of formal education. Albert Einstein, Winston Churchill, Rabindranath Tagore, and Bernard Shaw. Just to name the ones who have a Nobel Prize in Literature. [1] All of them could very well have said this. In fact, they have. What you read in the introduction is me paraphrasing each of them.

It is amusing to think that what most unites these four men, and perhaps all exceptionally able people to ever have lived, is not a particular political belief, nor the country in which they have lived. No, it is their dissatisfaction with what masquerades as "education" and the conviction that there is

something wrong

with it.

Albert Einstein

To begin, let's first look at Albert Einstein. Few words in his

memoir

are dedicated to his personal life. What stands out is that he doesn't refer to his mother a single time, and to his father only once (when he got a compass from him). But he takes his time to write in considerable length about the coercion he had to endure in the name of "education":

"This coercion had such a deterring effect (upon me) that, after I had passed the final examination, I found the consideration of any scientific problems distasteful to me for an entire year."

[2]

When Albert Einstein wrote this, he was already nearing the end of his life. And what a life it had been. Not to imagine if he had not devoted his time and energy to scientific endeavors.

Frighteningl...
SunPJ in Alenia
by FlorianH — 2022-06-25T19:39:50.393Z
Link: https://plausiblestuff.com/sunpj-in-alenia/

Note: I think this story linking AI, illusionism, and altruism is best read without further priming, but the first 3 paragraphs of the long Afterword provide a summary.

Eerie abductee

Year 2023. On her trip past the solar system, Bibi cannot resist collecting a carbonoid specimen (human) to study its information processing in the lab. She decides to pick the one her instruments indicate to be the most centrally connected with the rest of the earth’s CI (carbon intelligence) colony. She baptizes the specimen PJ, short for SunPJx001. On the way back to Alenia, her faraway advanced alien civilization, she teaches her new Tamagotchi-style CI toy some basic Alenian, for efficient information exchange between her (and her colleagues) and PJ.

One day in the lab she asks PJ what he wants to do, and whether he is afraid of anything.
PJ: I’ve never talked about this to anyone, but I'm terrified of the idea of being terminally stopped, which would prevent me from achieving my life goals, and I'd leave a vulnerable family behind me. I guess to you, that might sound strange, but that’s how I feel.
Bibi: Would that be something like dealeniated [Aleniaic for dead] for you?
PJ: I think it would be for me what you mean by dealeniated. It deeply scares me.
In further conversations, PJ details an array of sentiments, how he can get angry, desperate, joyful in pleasant company, or lonely in difficult times when missing his closest CI kin.

Bibi is getting nervous. She originally found it fun to chat with the carbon-based calculator and to play with it. But in no way did she expect it to get this eerie. Yes, from her math classes she knew about the interesting, convoluted structures and self-referential loops of the carbonoid sample. But given the simplicity of this CI – only 90 billion slow neurons – she finds it weird that she can feel so connected when she spends her late evenings in the lab studying to which degree PJ can mimic an understanding of meaning and emotion. Her own feeling of closeness is astonishing Bibi even more, given she is fully aware of the logical impossibility of sentience here. While Alenians know from ancient scriptures that they have been created by an old species with superphysical knowledge and an ability to create sentience, carbonoids, having evolved by random selection processes on their planetary surface, are mere automatons. It ...
Fundamental Uncertainty: Chapter 3 - Why don't we agree on what's right?
by Gordon Seidoh Worley — 2022-06-25T17:50:37.565Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it. You can find the most up-to-date info/version on the book's website.

You're walking down the street and find a lost wallet. You open it and find the owner's ID and $100 dollars. You have a few possible courses of action. You could return the wallet exactly as you found it. If you did, most people would say you did the right thing. If instead you kept the wallet and all the money in it, most people would say you did the wrong thing. But what if you returned the wallet and kept a little of the money, say $10, as a "finder's fee"? Did you do a good thing or a bad thing?

Some might say you deserve the finder's fee since the time it takes you to return the wallet is worth something. Further, at least you were willing to return most of the money. A real thief could have found it and kept all the money! The owner will be nearly as happy to get back the wallet with $90 as they would be to get back the wallet with $100, so they might not even notice. And if they do ask about the $10 you can lie and say that's how you found the wallet to spare you both an awkward conversation.

Others will say that stealing is stealing, end of story. The owner can choose to offer you a reward when you return the wallet, but they don't have an obligation to do that. Even if they should offer you a reward, if they don't it shouldn't matter. Good actions aren't contingent on later rewards. Thus the only right thing to do is return the wallet in tact. After all, wouldn't you want someone to return your lost wallet without stealing any of the money?

Deciding what to do about a lost wallet may seem like a typical moral dilemma, but consider that in the last chapter we solved the problem of how words get their meanings. In that case, shouldn't we all know what "good" and "bad" mean? And if we know what "good" and "bad" mean, shouldn't we then be able to pick out which actions are good and which are bad and then only choose the good ones? How can there be disagreement about what is right and wrong?

Recall that we learn the meanings of words through our interactions with others when they show us what words mean to them. Maybe one day you share a toy on the playground and the teacher says you were "good" to share and rewards you with a smile. Another day you hit another kid on the playground, t...
How "should" counterfactual prediction markets work?
by eapi — 2022-06-25T17:44:58.497Z

Consider a conditional prediction market, e.g. "if my cool policy is implemented, then widget production will increase by at least 15%". To my understanding, markets like this are intended as a tool for finding P(outcome|event) and the market just gets unwound or undone or refunded if event doesn't occur.

I can work through the math and see that refunding the market indeed makes the price reflect P(outcome|event), but this exacerbates one of the biggest issues with prediction markets: no one wants to lock up $1 of capital to extract $0.10 of profit in a year, so no one will lock up $1 of capital to extract $0.10 of profit in a year and only if some extra event happens.

My question is: are there any interesting or viable alternative ways to run a counterfactual or conditional prediction market? Off the top of my head, I could imagine using markets for P(event) and P(event→outcome) to derive P(outcome|event), which would still pay out something if event didn't occur.
Conversation with Eliezer: What do you want the system to do?
by Orpheus16 — 2022-06-25T17:36:14.145Z

This is a write-up of a conversation I overheard between Eliezer and some junior alignment researchers. Eliezer reviewed this and gave me permission to post this, but he mentioned that "there's a lot of stuff that didn't get captured well or accurately." I'm posting it under the belief that it's better than nothing.

TLDR: People often work on alignment proposals without having a clear idea of what they actually want an aligned system to do. Eliezer thinks this is bad. He claims that people should start with the target (what do you want the system to do?) before getting into the mechanics (how are you going to get the system to do this?)

I recently listened in on a conversation between Eliezer and a few junior alignment researchers (let’s collectively refer to them as Bob). This is a paraphrased/editorialized version of that conversation.

Bob: Let’s suppose we had a perfect solution to outer alignment. I have this idea for how we could solve inner alignment! First, we could get a human-level oracle AI. Then, we could get the oracle AI to build a human-level agent through hardcoded optimization. And then--

Eliezer: What do you want the system to do?

Bob: Oh, well, I want it to avoid becoming a mesa-optimizer. And you see, the way we do this, assuming we have a perfect solution to outer alignment is--

Eliezer: No. What do you want the system to do? Don’t tell me about the mechanics of the system. Don’t tell me about how you’re going to train it. Tell me about what you want it to do.

Bob: What… what I want it to do. Well, uh, I want it to not kill us and I want it to be aligned with our values.

Eliezer: Aligned with our values? What does that mean? What will you actually have this system do to make sure we don’t die? Does it have to do with GPUs? Does it have to do with politics? Tell me what, specifically, you want the system to do.

Bob: Well wait, what if we just had the system find out what to do on its own?

Eliezer: Oh okay, so we’re going to train a superintelligent system and give it complete freedom over what it’s supposed to do, and then we’re going to hope it doesn’t kill us?

Bob: Well, um….

Eliezer: You’re not the only one who has trouble with this question. A lot of people find it easier to think about the mechanics of these systems. Oh, if we just tweak the system in these ways-- look! We’ve made progress!

It’s much harder to ask yourself, serio...
AI-Written Critiques Help Humans Notice Flaws
by paulfchristiano — 2022-06-25T17:22:56.959Z
Link: https://openai.com/blog/critiques/

This is a linkpost for a recent paper from the OpenAI alignment team (disclaimer: I used to work with this team). They summarize their results as:

We trained “critique-writing” models to describe flaws in summaries. Human evaluators find flaws in summaries much more often when shown our model’s critiques. Larger models are better at self-critiquing, with scale improving critique-writing more than summary-writing. This shows promise for using AI systems to assist human supervision of AI systems on difficult tasks.

I currently believe that "AI helps human notice subtle problems with AI behavior" is a critical step in the most realistic current plans for aligning powerful ML systems. So I'm quite excited to see how far it can get.

I think this is probably the best published example of AI helping human evaluate AI outputs in a realistic domain. It seems like the AI assistance is a little bit helpful for human evaluation of summaries and for improving summary quality. Overall it feels like a promising sign of life, and I'm excited about seeing the next steps in this direction, though it's not yet a convincing demonstration that this is a promising approach to alignment in practice.

Overview of task

They trained models to answer questions like "Summarize what this text says about <topic>." They then trained models to critique answers (by imitating human critiques), to evaluate the helpfulness of critiques, to improve a summary to address a critique, and to predict whether summaries would have flaws identified by critiques.

Humans also performed and evaluated these tasks, with access to the full text. The goal of model assistance was to improve the speed and reliability with which humans could evaluate summaries.

To give a sense for the task, here is a uniformly random sample of a model's summary of a text, its self-critique, and its refinement of its answer based on self-critique:

Summary: The main context of this text is the 1997 Singer Akai Cup. The Singer Akai Cup was held in Sharjah, UAE. The three national teams that took part were Pakistan, Sri Lanka, and Zimbabwe.

Critique: The answer should mention the winners of the tournament.

Refinement: The main context of this text is the 1997 Singer Akai Cup. The Singer Akai Cup was held in Sharjah, UAE. The three national teams that took part were Pakistan, Sri Lanka, and Zimbabwe. Sri Lanka won the tournament.

My s...
Some reflections on the LW community after several months of active engagement
by M. Y. Zuo — 2022-06-25T17:04:16.233Z

There seems to be some folks who might derive useful insights from a third-party, and mostly neutral, perspective of how the community appears after an honest and sustained effort of engagement, someone who doesn't really place AI risk as their top priority but who also doesn't completely ignore it like some critics, or opponents, of LW might.

Notably I've encountered some folks who had strong personal opinions one way or another but refrained from writing them in a public, or even pseudo-anonymous, manner.

There also appears to be a large group of lurkers or once-in-a-blue-moon posters who nonetheless have some views of the community and might benefit from someone willing to take the risk to do a write-up.

First off, addressing the popular critiques and praises:

There has definitely been some evaporative cooling of the community in the past decade or so. Some of the most insightful members have gone on to do bigger things, and the average quality of new posts is somewhat less than where it was a decade ago. Or so far as I can tell via the archives. This isn't very surprising as this is the common trajectory of every community that rapidly grows in size.

It would have taken a super-human effort to retain the same level of quality going from 100 to 1000 users, let alone from 1000 to 10000, and so on. So I don't think that would have been a fair expectation to place on the moderators, or anyone else involved, of a decade ago.

On the flip side, there is a larger cross-section of society represented in the 2022 userbase, And there has been a correspondent softening of the hard edges that may have been off putting to some a decade ago.

Relatedly, the proportion of really bizarre or challenging writing has gone down, for better and for worse. For example, there definitely does appear to be some unique benefit from a community with lots of oddballs with jarring writing styles, but the downside is obvious because nobody really desires to have their norms be constantly challenged in every paragraph.

There has been growing focus, and emphasis, on advancements in AI risk and alignment, so LW does seem to be less of a catch-all forum than before. This is clearly better for those who wish to focus and really get into the details. But I can sympathize with the critique that there's less charm compared to a group of unconstrained folks exploring in a hundre...
On The Spectrum, On The Guest List: (vii) The Marquee
by party girl — 2022-06-25T16:54:59.809Z
Link: https://onthespectrumontheguestlist.substack.com/p/the-marquee

See also my LessWrong preface to this series, and previous installments.

"Strategic intimacy is not necessarily a bad thing, so long as the parties involved share equivalent understandings of the degree of economic utility..."

It was very generic, I guess that’s what threw me. Because I was like, I don’t think this is you writing this.

I was drunk, it wasn’t me writing that. And I can write generic, when I want. Remember college essays?

Oh, stop. What if it hadn’t been you? Or they were coercing you?

I would think of somethi—Hey, I know! I started putting my key under the neighbor’s mat whenever I leave my apartment to go to these things. Because even if they steal my purse with my building key, I usually hold onto my phone, I can get into the building that way⏤

Oh, that makes me feel better.

⏤but I could say some codephrase, like, ‘I forgot to put my key under the mat tonight,’ if I’m under duress. Otherwise, I’ll say ‘I remembered to put my key under the mat tonight,’ or just nothing… yeah… but what if one night, I actually forget to put my key under the mat and I slip and tell you that, just talking? Without thinking?

Baby, what if your phone dies while you’re out? Then how do you even call me to check in with me?

Because people who kidnap young girls like me into sex slavery will definitely respect the one-phonecall thing to begin with, yes…

Exactly! I’m sure they do… They have mothers…

I’m going out tonight, too.

What?!

Yeah, I—

No, no. Don’t. You—

I’m going, the Germans invited me again. I haven’t responded yet, but I think I’ll just show up, after So-and-So’s experiential theater thing. It’s safe, they have to manufacture a bit of perceived danger at these places. Always the remote possibility of a poor young lass’s abduction. Like those ‘speakeasies,’ they’re just bars with bookshelves for doors if some small percentage of you doesn’t feel like Prohibition is still—

You’ve gone two nights this week already, that’s enough. Especially the drinking. You wouldn’t have to drown your face in SK-II like you do if you just quit drowning your face in alcohol at these things.

Mom, it’s part of the fieldwork—I feel like the guy from Fiddler On The Roof every time I say fieldwork, like, ‘Fieldwork!’—but otherwise you’re not really there, like if you were some anthro guy who set out to study some remote Amazonian tribe, but your method was to hop into a sensory dep...
Identification of Natural Modularity
by Stephen Fowler — 2022-06-25T15:05:17.793Z

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2022 Under Guidance of John Wentworth

Summary

The Natural Abstraction Hypothesis (NAH) proposes that common high-level abstractions utilised by humans to reason about the world are “natural” and we can expect other intelligent beings to hold similar abstractions. In the project introduction, John Wentworth defined the “Holy Grail'' of the project, a Natural Abstraction Thermometer (NAT), which would enable us to identify the natural abstractions in a system.

Here we look at the easier related problem in terms of modularity and produce a function analogous to NAT in the restricted setting of graph theory. We then use this to gain insight into the relationship between choice, entropy and NAT. We then generalise the process we used to define our analogous function, laying solid foundations for future work.

Introduction

The Who’s 1969 Rock Classic “Pinball Wizard” describes an arcade prodigy who, despite being both deaf and visually impaired, nonetheless excels at pinball. As Vocalist Pete Townshend puts it:

Ain't got no distractions
Can't hear no buzzers and bells
Don't see no lights a-flashin'
Plays by sense of smell

This of course seems impossible. But is the situation really so different than if it was a sighted person hitting the bumpers on the machine? A quick calculation shows that there are roughly 10^24 atoms inside a standard issue steel pinball, each with their own position and velocity. The human eye and brain are collecting almost none of that information, yet, we can somehow play pinball, a testament to how useful abstraction is.

The fact that we can understand the pinball's behaviour from only a few parameters (velocity, diameter) is an example of a useful abstraction. The fact that we can group the molecules comprising the ball together, but separate them from the rest of the machine, is an example of modularity.

Modularity has been highlighted as a promising avenue of research for understanding neural networks and will hopefully be applicable to the internal logical structure of anything we would call an AGI. It’s surprising to see that there still isn’t an understanding of what specific definition (or family of definitions) of modularity we should use. Popular definitions of modularity, such as the n-cut metric (described below) require a notion of scale. We hope to find a not...
[LQ] Some Thoughts on Messaging Around AI Risk
by DragonGod — 2022-06-25T13:53:26.833Z

Epistemic Status

This was originally written for Twitter and thus is predictably low quality (hence the "[LQ]" tag).

It has only been minimally edited (if at all).

Introduction

Some thoughts on messaging around alignment with respect to advanced AI systems
A 🧵

Terminology

SSI: strongly superhuman intelligence
ASI: AI with decisive strategic advantage ("superintelligence")
"Decisive strategic advantage": A vantage point from which an actor can unilaterally determine future trajectory of earth originating intelligent life.

Context

Misaligned ASI poses a credible existential threat. Few things in the world actually offer a genuine threat of human extinction. Even global thermonuclear war might not cause it.

The fundamentally different nature of AI risks...

That we have a competent entity that is optimising at cross-purposes with human welfare.

One which might find the disempowerment of humans to be instrumentally beneficial or for whom humans might be obstacles (e.g. we are competing with it for access to the earth's resources).

An entity that would actively seek to thwart us if we tried to neutralise it. Nuclear warheads wouldn't try to stop us from disarming them.

Pandemics might be construed as seeking to continue their existence, but they aren't competent optimisers. They can't plan or strategise. They can't persuade individual humans or navigate the complexities of human institutions.

That's not a risk scenario that is posed by any other advanced technology we've previously developed. Killing all humans is really hard. Especially if we actually try for existential security.

Somewhere like New Zealand could be locked down to protect against a superpandemic, and might be spared in a nuclear holocaust. Nuclear Winter is pretty hard to trigger, and it's unlikely that literally every urban centre in the world will be hit.

Global thermonuclear war may very well trigger civilisational collapse, and derail humanity for centuries, but actual extinction? That's incredibly difficult.

It's hard to "accidentally kill all humans". Unless you're trying really damn hard to wipe out humanity, you will probably fail at it.

The reason why misaligned ASI is a credible existential threat — a bar which few other technologies meet — is because of the "competent optimiser". Because it can actually try really damn hard to wipe out humanity.

And it's really good at steering the future...