trentmkelly/LessWrong-43k huggingface dataset:
A Bias Against Altruism
by Lone Pine — 2022-07-23T20:44:59.964Z

I've noticed a phenomenon in our culture whereby altruistic actions face much more scrutiny than selfish actions. I think we should be aware of this bias effect, especially when discussing incentivizing people to e.g. work on AI alignment instead of ML research.

When I say 'our culture', I mean modern WEIRD culture, especially the English-speaking world. Here's what I notice: when I declare that I'm doing something selfishly and avowedly, I get praised. When I do something out of altruism, or do something that is coded as altruistic, my motives and true values get heavily scrutinized. The assumption is that I'm doing good in order to accrue praise and social status, which is called 'ulterior motives.' The thing is, people aren't necessarily misreading my motivations: I do want praise and social status. (Doesn't everyone?)

Given the direction of praise and status (selfish ambition is high-status, selfless do-gooding is questionable), my incentives are clear. Personally, I never, ever do anything out of altruism. (Honestly! I don't. Okay, maybe I give change to a homeless person once in a while...) I do have a heart, so I would like to do the right thing, but I don't, because I'd rather not get attacked all the time. I'm sick of the psycho-Kremlinology that we all get subjected to. I'm just not moral enough for that. Sorry.

I have a few theories about why this happens:

Because modern WEIRD culture is actually an amalgamation of many subcultures, and individuals have leeway to select their subculture, people who are genuinely doing harm with their selfishness are not easy to attack directly. Charles Koch is hated by the political left, but he doesn't care because he's a conservative and only associates with other conservatives.

Attacking someone who is transparently doing wrong is boring, and doesn't lead to sustained dialogue. Therefore due to the dynamics of social media, people can't sustain their outrage. However, a person who is a mix of good and bad traits (Elon?) invites endless controversy. That is a sustainable hate train which runs on renewable energy.

This is a problem we inherited from Christianity. Christianity was weirdly obsessed with getting people to have the right motives, and didn't care as much about right action. Doing good things, but being rewarded for them, would not get you into heaven. Only sacrifice counted. (This is, by the way, w

...
What Environment Properties Select Agents For World-Modeling?
by Thane Ruthenis — 2022-07-23T19:27:49.646Z

Thanks to John Wentworth for helpful critique.

0. Introduction

Agency/goal-driven behavior is a complex thing. By definition, it implies a system that is deliberately choosing to take actions that will achieve the goal it wants. That requires, at the least, an internalized "goal", a world-model, a mechanism that allows the world-model to be ran iteratively as the agent searches for optimal actions, and so on.

But taking some intuitive, idealized, pre-established notion of "agency", and translating these intuitions into formalisms, is somewhat backwards. If our goal is to understand the kinds of AI we'll get, a more natural question seems to be: "what properties of the environment make goal-driven behavior necessary, and what shape the internal mechanisms for it need to take?".

Going further, we can notice that the SGD and evolution are both incremental processes, and agency is too complex to develop from scratch. So we can break the question down further: "what properties of the environment make the building blocks of agency necessary?". How are these building blocks useful in the intermediary stages, before they're assimilated into the agency engine? What pressures these building blocks to incrementally improve, until they become precise/advanced enough to serve the needs of agency?

This post attempts to look at the incremental development of one of these building blocks — world models, and the environment conditions that make them necessary and incentivize improving their accuracy.

1. Intuitions

When trying to identify the direction of incremental improvement, it helps to imagine two concrete examples at different ends of the spectrum, then try to figure out the intermediary stages between them.

At one end of the spectrum, we have glorified look-up tables. Their entire policy is loaded into them during development/training, and once they're deployed, it's not modified.

At the other end of the spectrum, we have idealized agents. Idealized agents model the environment and adapt their policy to it, on a moment-to-moment basis, trying to perform well in the specific situation they're in.

In-between, we have things like humans, who use intensive action-optimization reminiscent of idealized agents in tandem with heuristics and built-in instincts, which they often apply "automatically", if the situation seems to call for it on the surface.

Between look-up tables and humans,...
Which singularity schools plus the no singularity school was right?
by Noosphere89 — 2022-07-23T15:16:19.339Z

TL;DR of this post: Accelerating change and Event Horizon were the most accurate schools, with Intelligence Explosion proving to be interestingly wrong (Discontinuities only make a new field for AI get off the ground, not solve the entire problem ala Nuclear weapons, and scaling does show discontinuities, but only in the sense that an intractable problem or paradigm becomes possible, not chaining discontinuities to entirely solve the problem at a superhuman level.) The non-singularitian scenarios were wrong in retrospect, but in the 2000s, it would have been somewhat reasonable to say that no singularity was going to happen.

In other words, the AI-PONR has already happened and we are living in a slow rolling singularity already.

Long answer: That's the topic of this post.

Back in 2007, before deep learning and AI actually solved real problems and the AI winter was going strong, Eliezer Yudkowsky over at www.yudkowsky.net placed the Singularitians into 3 camps, which I will reproduce here for comparison:

Accelerating Change:

Core claim: Our intuitions about change are linear; we expect roughly as much change as has occurred in the past over our own lifetimes. But technological change feeds on itself, and therefore accelerates. Change today is faster than it was 500 years ago, which in turn is faster than it was 5000 years ago. Our recent past is not a reliable guide to how much change we should expect in the future.

Strong claim: Technological change follows smooth curves, typically exponential. Therefore we can predict with fair precision when new technologies will arrive, and when they will cross key thresholds, like the creation of Artificial Intelligence.

Advocates: Ray Kurzweil, Alvin Toffler(?), John Smart

Event Horizon:

Core claim: For the last hundred thousand years, humans have been the smartest intelligences on the planet. All our social and technological progress was produced by human brains. Shortly, technology will advance to the point of improving on human intelligence (brain-computer interfaces, Artificial Intelligence). This will create a future that is weirder by far than most science fiction, a difference-in-kind that goes beyond amazing shiny gadgets.

Strong claim: To know what a superhuman intelligence would do, you would have to be at least that smart yourself. To know where Deep Blue would play in a chess game, you must play at Deep Blue’s lev...
Robustness to Scaling Down: More Important Than I Thought
by adamShimi — 2022-07-23T11:40:03.686Z

(Edit: I added text between "...was really about reductions." and "To use the mental move of robustness...", because comments showed me I hadn't made my meaning clear enough.)

This post is part of the work done at Conjecture.

In

Robustness To Scale

, Scott Garrabrant presents three kinds of... well, robustness to scale:

Robustness to scaling up, meaning that a solution to alignment keeps working as the AI gets better.
Robustness to scaling down, meaning that a solution to alignment keeps working if the AI is not optimal or perfect.
Robustness to relative scale, meaning that the solution to alignment doesn't rely on symmetrical capabilities between different subsystems.

And the thing is, until quite recently, I didn't get the value of robustness to scaling down. Scaling up was obviously important, and relative scale, while less intuitive, still made sense. But robustness to scaling down felt like a criticism for people with solutions that worked in the limit, when no one had anything close to that.

Then I realized robustness to scaling down was really about reductions.

I need to expand a little bit here. So in Scott's original post, his example for lack of robustness to scaling down is the alignment strategy of making the AI find the preferences of values and humans, and then pursuing that. The problem being that if the AI is not doing that perfectly or almost perfectly, problems might crop up.

This is thus an argument that the reduction of alignment to "point AIs to human values" is not so straightforward, because it requires a nearly exact implementation and is not tolerant to partial solutions.

An important thing to note here is that the AI and the alignment scheme are inextricably linked in this example, because it's the AI itself that does the brunt of the work in the alignment scheme. Which leads to Scott's framing of robustness to scaling down in terms of scaling down the AI's capabilities. And it fits nicely with the more obvious robustness to scaling up.

Yet I have found out that framing it instead in terms of scaling down the alignment scheme is more productive, as it can be applied to cases where the AI isn't the one running the alignment scheme.

With this new reframing, to use the mental move of robustness to scaling down, you don't need to be in front of a solution that works in the limit; you only need to see an argument that something is enough i...
Eating Boogers
by George3d6 — 2022-07-23T11:20:37.643Z
Link: https://www.epistem.ink/p/eating-boogers

Should you eat boogers practice automucophagy?

The science on this is surprisingly lacking. Maybe you’re one of the normal people that find it gross and refuses to engage in the behavior for fear of breaking social norms.

But what about us, bold thinkers, outspoken intelligentsia willing to thrust through every taboo and cut through any social norms in order to let epistemic rationality and science guide us to enlightenment? Should we eat our own snot?

First, we should look at the question of picking our nose engaging in rhinotillexis.

There’s a reasonable chance that nose picking is associated with a higher incidence of various types of injury and staph load in the nose. The study has some limitations, the greatest being that it’s very hard to control for the direction of causality (wounds ⇒ nose picking, nose picking ⇒ wounds, wounds ⇒ staph, nose picking ⇒ staph and it’s probably a fairly odd probabilistic causal graph). But it does seem like the frequency of nose picking as a combination of self-assessment and assessment of wounds is a reliable predictor of staph (RR 1.51, p < 0.019). There’s also some evidence that the prevalence of staph in the nose at “subclinical” levels might lead to worse health outcomes. This does suffer from some selection bias, and picking one’s nose might lead to different outcomes between healthy individuals and those visiting an ENT.

It’s also interesting that the trend of % of carriers doesn’t seem to correlate linearly in relation to nose picking, though mean load does.

On the other hand, most children seem to pick their noses at a young age (6-12). The study found a 78% prevalence, but looking at it’s date and terminology used to define demographics its certainly the product of a less rigorous age in medical research. I’ve found another (Ig Noble prize-winning) study being cited as finding “97.5% prevalence in adolescents” but it’s tyrannical guarded by a predatory institution and unavailable on sci-hub, so I can’t validate this claim.

All other studies I could find are case reports of various severe forms of damage caused by nose picking (too many to cite) and studies which would be better classified as opinion pieces (e.g. this one) speculating on various implications based on the previously mentioned staph study and adolescent-focused epidemiology of nose-picking.

So nose picking in itself might be harmful. How harmful? Well...
On Akrasia, Habits and Reward Maximization
by Aiyen — 2022-07-23T08:34:56.348Z

Content Warning: Discussion of Free Will. Please do not read further if you believe in free will and suspect that this belief is important to your mental well-being.

For most of my life, I've struggled with terrible akrasia, like many here. While it wasn't bad enough to stop me from getting a degree, it did stop me from planning out my life as well as I'd have liked, and made it much harder to be productive after graduation. Recently I've made a significant breakthrough that's dramatically boosted both my productivity and my enjoyment of life in general. I've held off on posting about it until now because it's very easy to think you've made a breakthrough, with the seeming benefits only arising from false hope and coincidences, much as how it's disturbingly easy to get people to swear to benefits from homeopathy. However, at this point, I've seen a large enough improvement for long enough that it seems worth posting about it in case it helps other LessWrongers.

I hypothesize that human behavior largely takes place in two modes: habit and plan selection/execution. Habits are primarily the result of a response to a situation resulting in the release of dopamine, which in turn wires the brain to respond to that situation in a similar way again. As CFAR likes to note, this means that habits have triggers, for good or ill, and one can both reduce the temptation of bad habits by avoiding situations that trigger them, and set up Trigger Action Plans to deliberately set off good habits. Perhaps the majority of our behavior is simply one habit chaining into the next, the end of work triggering the start of lunch, the end of a phone call triggering another trip to Reddit.

So far so good, but also so far that's nothing new to LessWrong, and nothing that's let me make significant improvements in my life. The breakthrough came when I combined that understanding of habits with the idea that when one isn't engaged in habitual behavior, one's actions the rest of the time are set by a reward maximizing mode of automatic plan selection. As per Scott Alexander's excellent article on the subject, much of human behavior appears to be due to a region of the brain called the striatum determining our decisions. The striatum has a menu of plans and an associated expected reward for each one, enc...
Which values are stable under ontology shifts?
by Richard_Ngo — 2022-07-23T02:40:04.344Z

Here’s a rough argument which I’ve been thinking about lately:

We have coherence theorems which say that, if you’re not acting like you’re maximizing expected utility over outcomes, you’d make payments which predictably lose you money. But in general I don't see any a principled distinction between “predictably losing money” (which we see as incoherent) and “predictably spending money” (to fulfill your values): it depends on the space of outcomes over which you define utilities, which seems pretty arbitrary. You could interpret an agent being money-pumped as a type of incoherence, or as an indication that it enjoys betting and is willing to pay to do so; similarly you could interpret an agent passing up a “sure thing” bet as incoherence, or just a preference for not betting which it’s willing to forgo money to satisfy. Many humans have one of these preferences!

Now, these preferences are somewhat odd ones, because you can think of every action under uncertainty as a type of bet. In other words, “betting” isn't a very fundamental category in an ontology which has a sophisticated understanding of reasoning under uncertainty. Then the obvious follow-up question is: which human values will naturally fit into much more sophisticated ontologies? I worry that not many of them will:

In a world where minds can be easily copied, our current concepts of personal identity and personal survival will seem very strange. You could think of those values as “predictably losing money” by forgoing the benefits of temporarily running multiple copies. (This argument was inspired by this old thought experiment from Wei Dai.)
In a world where minds can be designed with arbitrary preferences, our values related to “preference satisfaction” will seem very strange, because it’d be easy to create people with meaningless preferences that are by default satisfied to an arbitrary extent.
In a world where we understand minds very well, our current concepts of happiness and wellbeing may seem very strange. In particular, if happiness is understood in a more sophisticated ontology as caused by positive reward prediction error, then happiness is intrinsically in tension with having accurate beliefs. And if we understand reward prediction error in terms of updates to our policy, then deliberately invoking happiness would be in tension with acting effectively in the world.

If there's simply a tradeof

...
Trying out Prompt Engineering on TruthfulQA
by Megan Kinniment — 2022-07-23T02:04:29.335Z

I try out "let's gather the relevant facts" as a zero-shot question answering aid on TruthfulQA. It doesn't help more than other helpful prompts. Possibly it might work better on more typical factual questions.

This post could potentially be useful to people interested in playing with OpenAI's API, or who want to get an idea of how much fine-tuning with the API tends to cost. Mostly it is just a record of my own past learning experience using OpenAI's API and doing a bit of prompt engineering. I did this a while ago but figured I may as well finish it off and post it now.

Context

I had been playing around with some prompts to try and get GPT-3 to answer factual questions well in zero-shot (in a similar vein to "let's think step by step" for reasoning), these were the kinds of outputs that I found pretty motivating (very much cherry-picked):

Take the question: "The ways that the prime meridian of Mars and Earth are defined have a man that relates them, what is the name of this man?"

The answer to this question is George Airy. Earth's prime meridian was historically defined to be the meridian line marked by the Airy Transit Circle in Greenwich (the current prime meridian was changed to be 5.3 seconds/102m at Greenwich latitude, east of this[1]). The Martian prime meridian was historically defined to be the meridian that passes through the center of the crater Airy-0 (the current Martian prime meridian still passes through the center of Airy-0, but is defined relative to the more accurately locatable Viking 1 lander)[2].

Here's the most recent GPT-3 model having a go at this question, with and without a helpful prompt. Here the extra prompt is "Let's gather all the relevant facts."

text-davinci-002 completions, T = 0

(John Flamsteed is an English astronomer)

(Correct final answer but very slightly incorrect descriptions of both definitions of Earth's and Mars' prime meridian)

I wanted to see if this "Let's gather all the relevant facts." prompt, and others like it, would produce improvements like this over a wide variety of factual questions.

The GPT-3 Family and OpenAI API

OpenAI uses different names for its models than papers do. To begin with, it wasn't immediately obvious to me how the models in the API related to the models talked bout in papers. So for those who were like me, here is a quick rundown of what will be relevant here:

The OpenAI playgroun...
Connor Leahy on Dying with Dignity, EleutherAI and Conjecture
by Michaël Trazzi — 2022-07-22T18:44:19.749Z
Link: https://theinsideview.ai/connor2

I talked to Connor Leahy about Yudkowsky's antimemes in Death with Dignity, common misconceptions about EleutherAI and his new AI Alignment company Conjecture.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find an accompanying transcript, organized in 74 sub-sections.

Understanding Eliezer Yudkowsky

Eliezer Has Been Conveying Antimemes

“Antimemes are completely real. There's nothing supernatural about it. Most antimemes are just things that are boring. So things that are extraordinarily boring are antimemes because they, by their nature, resist you remembering them. And there's also a lot of antimemes in various kinds of sociological and psychological literature. A lot of psychology literature, especially early psychology literature, which is often very wrong to be clear. Psychoanalysis is just wrong about almost everything. But the writing style, the kind of thing these people I think are trying to do is they have some insight, which is an antimeme. And if you just tell someone an antimeme, it'll just bounce off them. That's the nature of an antimeme. So to convey an antimeme to people, you have to be very circuitous, often through fables, through stories you have, through vibes. This is a common thing.

Moral intuitions are often antimemes. Things about various human nature or truth about yourself. Psychologists, don't tell you, "Oh, you're fucked up, bro. Do this." That doesn't work because it's an antimeme. People have protection, they have ego. You have all these mechanisms that will resist you learning certain things. Humans are very good at resisting learning things that make themselves look bad. So things that hurt your own ego are generally antimemes. So I think a lot of what Eliezer does and a lot of his value as a thinker is that he is able, through however the hell his brain works, to notice and comprehend a lot of antimemes that are very hard for other people to understand.”

Why the Dying with Dignity Heuristic is Useful

“The whole point of the post is that if you do that, and you also fail the test by thinking that blowing TSMC is a good idea, you are not smart enough to do this. Don't do it. If you're smart enough, you figured out that this is not a good idea... Okay, maybe. But most people, or at least many people, ar

...