trentmkelly/LessWrong-43k huggingface dataset:
Entitlement as a major amplifier of unhappiness
by VipulNaik — 2022-06-08T22:08:59.658Z

A crude simplification of an important part of Buddhist thought is
that desire is the root cause of all suffering. There are many
nuances to this, with the central theme being that attachment to
one's desires is the root cause of all suffering.

Based on both my lived experience and my reflections, I think that
neither desire nor attachment to desire would, on their own, lead to
significant suffering in most cases. Rather, I've found one key
ingredient that often significantly increases suffering (and a range
of negative emotions): entitlement to the fulfillment of one's
desires.

In this post, I'll look at the idea of entitlement and why I think
entitlement is a key amplifier of negative thoughts. I'll explore
related ideas. I'll also look at ways to reduce one's sense of
entitlement without (necessarily) reducing the intensity of one's
desires. This is probably important because for many people (and to
some extent, for myself as well) experiencing desires intensely is a
key part of a life well-lived, so trying to tamp down one's desires in
order to tamp down one's suffering isn't a worthwhile trade-off.

Entitlement has a crucial role as amplifier of negative emotions

Desire for something can be thought of as having a really strong
preference for it, relative to the alternative. For instance, when I'm
hungry, I have a strong preference for eating relative to the
alternative of not eating.

Attachment to desire can mean a few different things; my own
interpretation is that it means that the desire is very salient in
the mind of the person desiring, and the person is unwilling or unable
to alter the desire (e.g., by cultivating a preference for other
options).

Entitlement is a belief that one "deserves" or "is owed by (specific
people, or the world at large)" the fulfillment of a desire. For
instance, if I'm feeling hungry, I have a strong desire for food, but
that doesn't necessarily make me feel entitled to food. How might I
feel entitled to food? For instance, I might believe that my
housemates who are using the kitchen right now for their own food prep
should vacate the kitchen so I can fulfill my desire for food. Or I
might believe that my family members, or my past self, should have
kept the refrigerator well-stocked so my present self wouldn't have to
go hungry.

Based on these simple examples (and other similar ones) my sense is
that desire and attachment to d...
Silly Online Rules
by Gunnar_Zarncke — 2022-06-08T20:40:41.076Z

Silly Rules (found in the Alignment Newsletter) are rules that do not have functional value in themselves.

Silly rules help groups adapt to uncertainty about the stability of social order by enriching the information environment. They help participants in these groups track their beliefs about the likelihood that violations of important rules will be punished and thus the likelihood that important rules will be violated. These beliefs are critical to the incentive to invest resources in interaction.

Examples:

wearing a head covering in public
standing for the anthem
using the fork with the left and the knife with the right hand

What formal or informal rules can people use online where things like hats don't work?

One thing that I have come up with since I read this post is

Emoticons. :-)

Reacting to posts with suitable emoticons shows who pays attention and who smiles toward whom - something that is an important signal offline but hard to replicate online.

What are your suggestions?
Untypical SIA


by avturchin — 2022-06-08T14:23:44.468Z

TL;DR: If I am untypical, then many more observers exist. If this claim is necessarily true, then classical SIA is false.

There is a situation where we could prove something like SIA, let’s call it Untypical SIA. Imagine that there are 100 green dots and one red dot on a bow’s target (and they cover all the space). We know that there were either 10 or 100 attempts to hit the target with a random arrow. We only know that the red dot was eventually hit (maybe we are in it). Now we can give higher credence to the theory that there were 100 attempts (P1(red)=0.6339) over the theory of only 10 attempts (P2(red)=0.0956) to hit the red dot. Credence in 100 attempts = P1/(P1+P2) = 0.86.

The resonating above is made from God’s view and seems to have high epistemic status: it is not an assumption, it is a proof.

Examples of application of such untypical-SIA reasoning in the real life: if there is life on Earth, there are many other planets out there, which tried but fail to develop life. If humans are the only intelligent species, there are many others species that “tried” and fail to become intelligent. If our universe is fine-tuned, there are many other universes which are not fine-tuned, so the multiverse theory is true.

If Untypical SIA is a necessary condition for a larger set of observers, then classical SIA is false.

Untypical SIA requires that the red dot is different from the green dots. If all dots are red, and we learn that one of the dots was hit, it doesn’t give us any new information to update. If uniqueness is the necessary (=if and only if) condition for update when the lack of uniqueness means no update. In that case, classical SIA is false (it is not a problem in the infinite universe, where something like SIA still works, as we should look at relative concentrations of observers in different regions of the multiverse instead; all our SIA-based cosmological constructions are safe herein).

In other words, the fact of my existence is an argument in favor that more observers exist – only if I am special. In classical SIA, the mere fact that I exist at all is evidence that many more attempts to create me were made, without any mention that I should be different from all other observers.

Obviously, I can’t just say that my random name is rare. Because anyone can say that, and we are all in the same “ep...
Eliciting Latent Knowledge (ELK) - Distillation/Summary
by Marius Hobbhahn — 2022-06-08T13:18:51.114Z

This post was inspired by the

AI safety distillation contest

. It turned out to be more of a summary than a distillation for two reasons. Firstly, I think that the main idea behind ELK is simple and can be explained in less than 2 minutes (see next section). Therefore, the main value comes from understanding the specific approaches and how they interact with each other. Secondly, I think some people shy away from reading a 50-page report but I expect they could get most of the understanding from reading/skimming this summary (I'm aware that the summary is longer than anticipated but it's still a >5x reduction of the original content).

I summarized the

ELK report

and the results of the

ELK competition

. I personally think ELK is very promising and therefore want to make extra sure that the summary is readable and accurate. In case anything is unclear or wrong, please let me know.

Context and Introduction

Eliciting Latent Knowledge

was written by

Paul Christiano

,

Ajeya Cotra

and

Mark Xu

. Before getting to the main summary, I want to provide some background and context. In addition to the summary of the main report, I provide a summary of the results from the ELK prize contest in the end. My aim is to give a brief summary of “everything related to ELK” with links to the long versions.

Paul Christiano worked on various projects in AI safety before founding the

Alignment Research Center

(ARC). He lead the LLM alignment team at OpenAI and is known for, among other contributions,

AI safety via debate

and

Iterated Distillation and Amplification

(IDA).

From my very limited outside perspective and some of his podcast appearances (e.g. at

80K

or the

AI x-risks podcast

), Paul Christiano seems to run into similar problems with many current alignment techniques. Whether you choose inverse RL, AI safety via debate or IDA, there is always the risk that sufficiently powerful models create problems that can only be solved by understanding what the model “truly thinks”--by eliciting latent knowledge (ELK). In the case of AI safety via debate, for example, you have to trust the individual debate AIs. With a good solution to ELK, we could make sure the debate AIs are trustworthy.

Of course, other people want to understand what’s going on in Neural Networks (NNs) as well.

Chris Olah

, for ex...
Research Questions from Stained Glass Windows
by StefanHex — 2022-06-08T12:38:44.848Z

I just read Scott Alexander's recent ACX blog post on DALL-E 2 and I think it suggests many possible hypotheses about the internals of models. Could we test these with our current interpretability tools, e.g. for transformers? (DALL-E 2 seems to contain a transformer part)

Note: You might want to read or skim the blog post, otherwise the examples below will seem a bit out-of-context.

Case 1: Patterns that trigger styles

Example: Adding moose to the Tycho Brahe prompt

The stained glass has retreated from the style to become a picture element again. The style has become generally more modern.

(image linked from ACX)

Can we find out how the network decides what style to use? Can we use this to find out how to force the style? I would be surprised if there was no better way than saying "stained glass" to make DALL-E-2 produce a stained-glass style image (rather than a picture of a stained-glass window).

This might require 2 steps: Step 1 is to find out what neuron stores whether or not to make a stained-glass window (ideas see below), step 2 is to find inputs that maximize the excitation of this neuron (e.g. by iterating over random word input sequences, maybe enhanced with various Monte Carlo sampling techniques such as MCMC).

If we know how exactly the style information is stored, can we inject it somewhere along the way? I.e. can we write a "stained-glass image generator" by adding some inputs in feature-space (i.e. not just in the input sequence but e.g. between the encode- and decode processes of the Transformer), that always generates stained-glass style images?

Case 2: Triggering more advanced characteristics: Mood, Perspective, ...

Styles are a pretty straightforward thing people do with DALL-E-2, make picture of X in style Y. But look at the Occam's razor pictures, some of them ("GILA WHAMM") show a person looking quite scared or somewhat dreadful, others (like "William of Ockham" ones) show different kind of emotions (confusion? scared?).

(image linked from ACX)

Is this something the NN "knows"? Could we find out where in the network this is stored, and again, find patterns that provoke the behavior?

Possible idea: Use emotion-classifier network to classify outputs (generated from random inputs), group them, and look at what neuron excitation are common in a category, but different between the different categories. (Possibly difficult since the outputs will b...
Staying Split: Sabatini and Social Justice
by Duncan Sabien (Inactive) — 2022-06-08T08:32:58.633Z

Author's note: This is a contemporary post about an ongoing current event. It's not a timeless essay in the way that most of my essays are, though it does contain insight and thoughts on timeless topics.

Background

David Sabatini is a molecular biologist, previously employed as a tenured professor at MIT and a lead researcher at the Whitehead Institute. He was fired in August of 2021, after which most of his professional connections quickly dried up. He was almost restored to good standing in April of 2022 via a new position at NYU, but protests and external pressure caused NYU to withdraw their offer.

I shared a one-sided piece about Sabatini on Facebook, asking for people to falsify it. I was given various links and documents in response. I spent about five hours following up on sources, looking for original information, and trying to piece together a coherent take.

It proved to be impossible, and it proved to be impossible in a way that I think is interesting, and relevant to a lot of questions about how our culture functions (or doesn't). This essay is my attempt to digest and debrief, essentially writing to myself.

Why was Sabatini fired and blacklisted?

Option A: Because he engaged in romantic/sexual misconduct in conflict with the policies of his workplace, and created a hostile and sexualized environment that made work difficult or impossible for many of his subordinates.

Option B: Because a vindictive former lover enacted a revenge plot, partially enabled by an ideologue in the org's power structure who was looking for any pretext to shake things up.

Option C (for 'cynical'): Because scandal is costly regardless of whether it's grounded in fact, and there are a large number of highly-motivated people who have concentration of force against groups like MIT or NYU when it comes to highly charged questions like putative sexual misconduct.

If you buy reports like that of Suzy Weiss, the timeline looks something like this:

Sabatini runs a world-class, cutting-edge lab without any complaints or issues for two and a half decades.
At a conference in Maryland in 2018, Sabatini hooks up with Kristin Knouse, a cancer researcher in her own right, 21 years his junior.
They mutually establish some ground rules for their relationship, with Kristin in particular insisting that it remain open and low-key so she can carry on with other preexisting f

...
Puddle Temperature Alarm
by jefftk — 2022-06-08T02:10:02.162Z

I cook pancakes for my kids most mornings, and the griddle takes a
little while to heat up. If you look around for articles on how to
know when your pan is hot enough you'll see people suggest dropping in
a little water and seeing how it behaves. The problem is, this
requires you to be paying attention and guessing when it is about
right to test. I'm trying to do a lot of different things in the
morning, and don't want to allocate so much attention here. Instead, I
put the water in at the beginning:

Once the pan heats up, the water starts to sizzle: the temperature is
almost high enough:

The noise acts like an alarm, reminding me to stop unloading the
dishwasher or whatever and continue with the pancakes.

Comment via: facebook
Why I don't believe in doom
by mukashi — 2022-06-07T23:49:17.551Z

In my previous very divisive post I was said more than once that I was being downvoted because I was not providing any arguments. This is an attempt to correct that and to expand on my current model of doom and AGI. I think it can be more clear in the form of a Q and A

Q: First, what are you arguing against?

A: Against the commonly ( in this community) held belief that the first AGI means doom in a very short period of time (I would say days/weeks/months)

Q: What makes you think that?

A: We live in a complex world where successfully pulling off a plan that kills everyone and in a short of time might be beyond what is achievable, the same way that winning against AlphaZero giving it a 20 stone handicap is impossible even by a God- like entity with infinite computational resources

Q: But you can't prove it is NOT possible, can you?

A: No, the same way that you can NOT prove that is possible. We are dealing here with probabilities only, and I feel they have been uncritically thrown out of the window by assuming that an AGI will automatically take control of the world and will spread its influence to the universe at the speed of light. There are many reasons why things can go in a different way. You can derive from the orthogonality principle and instrumental convergence that an AGI might try to attack us, but it might very well be that is not powerful enough and for that reason decides not to.

Q: I can think plans to kill humans successfuly

A: So do I. I can't think of plans to kill all humans successfuly in a short span of time

Q: But an AGI would

A: Or maybe not. The fact that people can't come up with real plans makes me think that those plans are not that easy, because the absence of evidence is evidence of absence

Q: what about {evil plan}?

A: I think you don't realise that every single plan contains multiple moving parts that can go wrong for many reasons and an AGI would see that too. If the survival of the AGI is part of the utility function and it correctly infers that its life might be at risk, the AGI might decide not to engage in that plan or to bide its time, potentially for years, which invalidate the premise we started with

Q: does it really matter if it is not 2 weeks but 2 years?

A: it matters dearly, because it give us the opportunity to make other AGIs in that time window.

Q: What does it change? They would kill you too

A: No, we just established that it ...
"Pivotal Acts" means something specific
by Raemon — 2022-06-07T21:56:00.574Z

The term Pivotal Act was written up on Arbital in 2015. I only started hearing it discussed in 2020, and then it rapidly started seeing more traction when the MIRI 2021 Conversations were released.

I think people mostly learned the word via context-clues, and never actually read the article.

I have some complaints about how Eliezer defined the term, but I think if you're arguing with MIRI-cluster people and haven't read the article in full you may have a confusing time.

The arbital page for Pivotal Act begins:

The term 'pivotal act' in the context of AI alignment theory is a guarded term to refer to actions that will make a large positive difference a billion years later.

And then, almost immediately afterwards, the article goes on to reiterate "this is a guarded term", and explains why. i.e. this is a jargon term that people are going to be very tempted to stretch the definition of, but which it's really important not to stretch the definition of.

The article notes:

Reason for guardedness

Guarded definitions are deployed where there is reason to suspect that a concept will otherwise be over-extended. The case for having a guarded definition of 'pivotal act' (and another for 'existential catastrophe') is that, after it's been shown that event X is maybe not as important as originally thought, one side of that debate may be strongly tempted to go on arguing that, wait, really it could be "relevant" (by some strained line of possibility).

It includes a bunch of examples (you really should go read the full article), and then notes:

Discussion: Many strained arguments for X being a pivotal act have a step where X is an input into a large pool of goodness that also has many other inputs. A ZF provability oracle would advance mathematics, and mathematics can be useful for alignment research, but there's nothing obviously game-changing about a ZF oracle that's specialized for advancing alignment work, and it's unlikely that the effect on win probabilities would be large relative to the many other inputs into total mathematical progress.

Similarly, handling trucker disemployment would only be one factor among many in world economic growth.

By contrast, a genie that uploaded human researchers putatively would not be producing merely one upload among many; it would be producing the only uploads where the default was otherwise no uploads. In turn, these uploads could do deca

...
Embodiment is Indispensable for AGI
by P. G. Keerthana Gopalakrishnan — 2022-06-07T21:31:21.174Z
Link: https://keerthanapg.com/tech/embodiment-agi/

There are many paths on the road to AGI. Nando de Freitas recently wrote that “Scale is all you need” following the release of Deepmind’s Gato paper. Larger and larger transformers, with language interfaces doing supervised learning on large datasets is one such touted path. While Gato trains and evaluates on data from physical robots, there is a larger paradigm on getting to AGI that fully bypasses embodiment or robotics. This is the path pursued by OpenAI/Anthropic, who once famously

disbanded their robotics team

. I recently found myself in a car ride with a friend who was a “member of technical staff” at OpenAI and he asked “Do you think solving embodied AI is required for AGI or is it enough to be digital?” I have thought a lot about that question since.

Language as the end-all of intelligence

When it comes to conversations around AGI, language is often the start and end all of intelligence. From an evolutionary perspective, however, intelligence has not arised/existed outside of embodied agents and language itself evolved as a tool for multiagent communication. Language and structured coordination of agents helps with reward maximization in an environment, and could pose as an evolutionary byproduct of intelligence in line with the

Reward is Enough

hypothesis. Language certainly does encode a huge wealth of human knowledge and is a promising and defacto hammer to solve intelligence. But is it enough for AGI?

What is AGI?

It all comes to answering the question of what constitutes an AGI.

Andrej Karpathy says it is a feeling

. Several Anthropic researchers define it, weirdly so, as ‘that which would make the world a weird place’.

Nick Bostrom

defines AGI as “an intellect that is much smarter than the best human brains in practically every field”.

Wikipedia

defines it as “the ability of an intelligent agent to understand or learn any intellectual task that a human being can.”

Open Philanthrophy

describes “transformative AI” with an economic definition as that which would 10x the Gross World Product by bringing about as much difference as agriculture or the industrial revolution. There’s also the idea, embraced in OpenAI circles, that if you can hire a remote worker and they’re as good as a human, that is AGI. If an AI can work like a physicist or researcher and make scientific breakthroughs and write papers without ...
Stephen Wolfram's ideas are under-appreciated
by Kenny — 2022-06-07T20:09:37.636Z

Most of these "ideas" are described and explained in his book "A New Kind of Science" (NKS), available for free on the web:

Stephen Wolfram: A New Kind of Science | Online—Table of Contents

Here's a (long) recent post describing the rather extraordinary efforts of writing (and publishing) the book:

The Making of A New Kind of Science—Stephen Wolfram Writings

One of the main criticisms of the book, that I've come across, is that he's 'arrogant' and gave himself credit for other's discoveries. The above post contains some relevant info about him deliberately deciding to write the book mostly from 'his perspective'. Charitably, I think that makes his apparent 'arrogance' better understood as something like a 'literary convention', especially given the plausibility of him having 'independently re-discovered' some of the particular results he reports. (The book itself is about half detailed technical notes, tho still not in a 'proper' academic citation style. I can kinda understand why academics would hate that, but it's frustrating, to me, that that seems to reliably overshadow any real effort to engage with the ideas themselves!)

Here's another recent post describing his ongoing 'physics project' that's grown out of the work he began and covered in NKS:

Twenty Years Later: The Surprising Greater Implications of A New Kind of Science—Stephen Wolfram Writings

I don't have sufficient technical background to really judge the output of this physics work itself but it stills seem (at least) very intriguing!

In particular, this seems very enticing:

Instead of imagining that the universe follows some particular rule—albeit applying it multicomputationally in all possible ways—what if the universe follows all possible rules?

And then we realized: this is something much more general than physics. And in a sense it’s the ultimate computational construct. It’s what one gets if one takes all the programs in the computational universe that I studied in A New Kind of Science and runs them together—as a single, giant, multicomputational system. It’s a single, unique object that I call the ruliad, formed as the entangled limit of all possible computations.

That seems to me to very much 'rhyme' with some arguments for 'many worlds' quantum physics, e.g. 'all the branches are real!'.