trentmkelly/LessWrong-43k huggingface dataset:
I can see how I am Dumb
by Johannes C. Mayer — 2023-06-10T19:18:59.659Z

When I'm talking to somebody, sometimes I lose the conversational thread. Or sometimes I feel like there is this thing I want to say that seems relevant now, but I just can't remember it.

Or maybe I'm trying to solve a particular problem. I throw myself again and again at the problem, but it just won't budge. And then after some long amount of time, possibly hours, I realize that the solution was extremely simple. And I just failed to execute the right kind of solution-finding algorithm that would have found this very simple solution quickly.

I would expect that people with more intelligence, perform better in these domains. They have probably an easier time remembering and retaining the right things. Well, that alone might be sufficient to explain a large chunk of what makes a more intelligent person able to perform better.

If you remember the right things quickly that are relevant in the moment, and if you can keep track of more things in your head at the same time without losing track of what these things were, then that might account for a large chunk of how an intelligent person is better at performing any particular thing.

The core point here is that I think everybody, even somebody who would be much smarter than me, can see various failure modes in their own cognition and realize that they might be just so fundamental that there is no direct way of changing them.

I'm pretty sure that at some level what sorts of things your brain spits out into your consciousness and how useful that information is in the given situation, is something that you can't fundamentally change. I expect this to be a hard-coded algorithm, and I expect there to be many such hard-coded cognitive processes that can't be changed (at least not in major ways).

The cognitive improvements that you can apply will be at a higher level. To me, it seems that is what much of the Sequences are about. You can understand that there is something like the sunk cost fallacy, and understanding what it is, allows you to train yourself to recognize when you fall pray to it (Though that is a separate step from understanding what it is that you actually need to do to get most of the benefit).

And the way you would do this is by for example using TAPs. In a sense, it seems that tabs are a way to install a very small hook into your brain in the programming sense. My current model of it is that you install a ...
Ethodynamics of Omelas
by dr_s — 2023-06-10T16:24:16.215Z

Epistemic status: just kidding... haha... unless...

Introduction

A great amount of effort has been expended throughout history to decide the matter of "what is good" or "what is right" [citation needed]. This valiant but, let's face it, not tremendously successful effort has produced a number of possible answers to these questions, none of which is right, if history is any indication. Far from this being their only problem, the vast majority of these answers also suffer from the even more serious flaw of not being able to produce anything resembling a simple quantitative prediction or computable expression for the thing they are supposed to discuss, and instead prefer to faff about with qualitative and ambiguously defined statements. Not only they aren't right; they aren't even wrong.

More recently, and in certain premises more than others, a relatively new-ish theory of ethics has gained significant traction as the one most suited to a properly scientific, organized and rational mind - the theory of Utilitarianism. This approach to ethics postulates that all we have to do to solve it is to define some kind of "utility function" (TBD) representing all of the "utility" of each individual who is deemed a moral subject (TBD) via some aggregation method (TBD) and in some kind of objective, commensurable unit (TBD). It is then a simple matter of finding the policy which maximizes this utility function, and there you go - the mathematically determined guide to living the good life, examined down to however many arbitrary digits of precision one might desire! By tackling the titanic task head on, refusing to bow to the tyranny of vague metaphysics, and instead looking for the unyielding and unforgiving certainty of numbers, Utilitarianism manages one important thing:

Utilitarianism manages to be wrong.

Now, I hope the last statement doesn't ruffle too many feathers. If it helps reassure any of those who may have felt offended by it, I'll admit that this study also deals unashamedly into Utilitarianism; and thus, by definition, it is also wrong. But one hopes, at least, wrong in an interesting enough way, which is often the best we can do in such complex matters. The object of this work is the aggregation method of utility. There are many possible proposals, none, in my opinion, too satisfactory, all vulnerable to falling into one or another horrible trap laid by the cle...
Dealing with UFO claims
by ChristianKl — 2023-06-10T15:45:05.884Z

A common response in the recent LessWrong threads about UFO's is rationalists immediately going into a state of wanting to translate the news into probabilities of the existence of aliens instead of taking the facts for what they are and thinking about what should happen based on the revealed facts.

According to Ross Coulthart, David Grusch gave the ICIG, Congress and the Senate, the location where the vehicles are stored and the names of the people who control access to those programs.

While I would like to know whether or not aliens visited earth, I think it's more useful to simply take the stance "I don't know" instead of thinking in terms of probability.

From the "I don't know"-stance, the next step is obvious: There need to be congressional hearings where the people who were named has being in control of access to those programs get asked in public about the nature of those programs.

Given that there seem to be powerful people in the intelligence community who want to block public exposure of whatever the nature of those programs are, it's important that there's public pressure on Congress to investigate and hold public hearings that go into the details.

The mental moves of directly rounding down to "my priors against aliens are high" -> "no aliens" -> "no need to do anything" is bad as if enough people hold it we won't get more evidence.
A Theory of Unsupervised Translation
Motivated by Understanding Animal Communication
by jsd — 2023-06-10T15:44:00.238Z
Link: https://arxiv.org/abs/2211.11081

This is part of a larger project that aims to understand sperm whale communication. Jacob Andreas, Michael Bronstein, Antonio Torralba, and Shafi Goldwasser (an author of this paper) are part of the large team.

I'm linking to this here because folks who think about ELK/Interp/Learning the Prior may find it interesting, cf eg this post.
What are brains?
by Valentine — 2023-06-10T14:46:46.936Z

I see a self-reference problem with reductionism. I wonder if this has already been solved. So I'm asking.

Best as I can tell, there aren't actually things in reality. That's a human interpretation. It collapses upon almost any inspection, like the Ship of Theseus or the paradox of the heap. We also see the theory of thing-ness collapsing with physical inspection, which is why QM is "weird".

Best as I can tell, all thing-ness arises from reification. Like how we talk about "government" like it's a thing, but really we've just thingified a process. "The weather" is another clear-to-me example.

It seems to me that physical objects are exactly the same in this respect: a child interacts with swirling sense perceptions and reifies (i.e. thingifies) those experiences into "a ball" or whatever.

So how does reification happen?

Well, it's not like there's a thing that reification is; it's just a process that a human mind does.
Okay, so what's a human mind? Well, it's a process that the human brain engages in.
So what's a brain? A configuration of chemicals.
The chemicals are atoms, which are patterns of wiggling magical reality fluid from QM, which is maybe just made of mathematical structures.

So… when do we get to the place where we aren't using objects to explain how the impression of objects arises?

This puzzle shows up in the Many Worlds view of QM. It's roughly equivalent to "How do worlds arise?"

Two things (!) get entangled via an interaction. When one of those things is a human brain, we see the various possibilities, but as various brains which aren't interacting directly anymore from the perspective of those brains. So instead of seeing all quantum superposed configurations at once, each version of us observes just one configuration.

Okay, great.

So where are these brains that are getting entangled with other things? Aren't these brains made of the same quantum soup as everything else?

This Many Worlds thing makes a lot of sense if you're seeing the situation from the outside, where you can safely reify everything without self-reference. You can be a brain looking at a situation you're not in.

But we're in the reality where this happens. We're embedded agents. The brains doing this reification are somehow arising from the very process they're attempting to explain, which they meta-explain by… reifying themselves?

Which is to say, brains exist as reifications of brains.

So...
EY in the New York Times
by Blueberry — 2023-06-10T12:21:34.496Z
Link: https://www.nytimes.com/2023/06/10/technology/ai-humanity.html?unlocked_article_code=g8tzIOqlSdq4MsYnWh33QNXLjBP1AWRE3nn2VmFToIS8_ZExZucfbX0rMeP-HqAYyM3FAAy7KjSGZ3nfc_-BZF2BxgehlZb38nUsQSnXM_cX5hSpM1KCG2nZxsIH54uaF5oYNfb_RNzDHqTizWQBZo1FTH_QACtkRxVdGuV6m-A0rqlt-tzaR-Y8LCJCrxsYh8DbJb45QMS-qCFEQD9MdmXgixznve3vsqEfBliviywY335_eJZIRk5UfN38lkw7rBGkO_GAed0SkDfijxacro7FO3__iRI-A9mpUmeiIINrGrTMisYDDTj6S0khoEduIqDCw3Uq8fOW&smid=url-share

The Paper of Record had a good article today about existential AI risk, mentioning EY:

In the early 2000s, a young writer named Eliezer Yudkowsky began warning that A.I. could destroy humanity. His online posts spawned a community of believers. Called rationalists or effective altruists, this community became enormously influential in academia, government think tanks and the tech industry.

It even mentioned Clippy!

The worriers have often used a simple metaphor. If you ask a machine to create as many paper clips as possible, they say, it could get carried away and transform everything — including humanity — into paper clip factories.

Very happy to see this.
Goal-misgeneralization is ELK-hard
by rokosbasilisk — 2023-06-10T09:32:50.397Z

Consider an adversarial training-scheme for solving goal-misgeneralization, ( here i consider Redwood Research’s work on “Adversarial Training for High-Stakes Reliability”).

Consider a model that was trained to perform a specific task. To guarantee worst-case performance for this model, we need to have bounds for its outputs in adversarial examples. However, it is not feasible to run the model in the actual environment because it could lead to unwanted consequences.

Therefore, we can use a prediction-head to generate a simulated version of the trajectory that the model might undergo. We can train a classifier from the output of the prediction-head to classify whether the AI system exhibits goal-misgeneralization in an adversarial manner. The model can then be penalized for misgeneralization behavior, and the policy obtained can be distilled into a newer model iteratively. We can use this prediction-head to build another model that can act as a classifier and distill the conservative policy (model) into a newer one.

However, this is only possible when the prediction-head is not deceiving, which is essentially the same as the ELK problem.
What do beneficial TDT trades for humanity concretely look like?
by Stephen Fowler — 2023-06-10T06:50:21.817Z

Who do you think you're trading with and why?
What explicitly is being traded?
What mechanism enables both parties to trust the other?
I am quite pessimistic TDT related trades. Not because I think it isn't a real thing, but because I think the kinds of "trade offers" we encounter will be extremely negative. But I also know of at least two well established safety researchers who are extremely optimistic about the idea.
If I understand it correctly, Richard Ngo's "Ants and the Grasshopper" is suggesting (I think) that human civilization (the grasshopper) encountering an alien AI swarm (the ants) could trade us not fighting back against genocide with the possibility of human civilization being simulated in the distant future.

This idea appeals to me, but also seems to run into the problem that there is an extremely grim alternative trade that is usually cheaper for the ants.
Feel free to DM if you don't want to post publicly.
cloud seeding doesn't work
by bhauth — 2023-06-10T05:14:48.229Z

Alice: People should do more cloud seeding.

Carol: Cloud seeding doesn't work.

Alice: Governments spent a lot of money doing it, and there are a bunch of studies. And you're saying it just doesn't work?

Carol: Why is a cloud opaque? Because it has water droplets in it. If it already has droplets formed, then nucleating droplets does nothing.

Alice: Ah, a common misconception. Cloud seeding is about nucleating ice crystals, not water droplets. Scientists tested the concept before cloud seeding was deployed.

Carol: That was tested in unrealistic conditions: closed chambers with negligible suspended particles. And it's not that clouds are at altitudes with no particles, either - aerosol number density is similar up to around 9 km.

Alice: The fact is, cloud seeding was done on a large scale, it's still done some places, and there were a bunch of studies on its effectiveness.

Carol: Those studies showed positive effects only because of the same kind of publication bias you see for bad psychology studies. The better meta-analyses showed negligible effect sizes. That's why usage mostly stopped, but it was kind of an embarrassing failure with nobody motivated to publicize it, so people weren't very vocal about that.

Alice: No, if that was the case, there would be more articles and stuff about it, and besides, there are still governments doing cloud seeding today. After so much time to study things, governments wouldn't do cloud seeding at all if it just didn't work, which means it must work, so we should do more of it.

Who is it correct to believe here?
formalizing the QACI alignment formal-goal
by Tamsin Leake — 2023-06-10T03:28:29.541Z
Link: https://carado.moe/qaci-math.html

this work was done by Tamsin Leake and Julia Persson at Orthogonal.

thanks to mesaoptimizer for his help putting together this post.

what does the QACI plan for formal-goal alignment actually look like when formalized as math? in this post, we'll be presenting our current formalization, which we believe has most critical details filled in.

1. math constructs

in this first part, we'll be defining a collection of mathematical constructs which we'll be using in the rest of the post.

1.1. basic set theory

we'll be assuming basic set theory notation; in particular, A×B×C is the set of tuples whose elements are respectively members of the sets A, B, and C, and for n∈N, Sn is the set of tuples of n elements, all members of S.

B={⊤,⊥} is the set of booleans and N is the set of natural numbers including 0.

given a set X, P(X) will be the set of subsets of X.

#S is the cardinality (number of different elements) in set S.

for some set X and some complete ordering <∈X2→B, min< and max< are two functions of type P(X)∖{∅}→X finding the respective minimum and maximum element of non-empty sets when they exist, using < as an ordering.

1.2. functions and programs

if n∈N, then we'll denote f∘n as repeated composition of f: f∘…∘f (n times), with ∘ being the composition operator: (f∘g)(x)=f(g(x)).

λx:X.B is an anonymous function defined over set X, whose parameter x is bound to its argument in its body B when it is called.

A→B is the set of functions from A to B, with → being right-associative (A→B→C is A→(B→C)). if f∈A→B→C, then f(x)(y) is simply f applied once to x∈A, and then the resulting function of type B→C being applied to y∈B. A→B is sometimes denoted BA in set theory.

AH→B is the set of always-halting, always-succeeding, deterministic programs taking as input an A and returning a B.

given f∈AH→B and x∈A, R(f,x)∈N∖{0} is the runtime duration of executing f with input x, measured in compute steps doing a constant amount of work each — such as turing machine updates.

1.3. sum notation

i'll be using a syntax for sums ∑ in which the sum iterates over all possibles values for the variables listed above it, given that the constraints below it hold.

x,y∑y=1y=xmod2x∈{1,2,3,4}x≤2

says "for any value of x and y where these three constraints hold, sum y".

1.4. distributions

for any countable set X, the set of distributions over X is defined as:

ΔX≔{f|f∈X→[0;1],x∑x∈Xf(x)≤1}

a functio...
Expert trap: Why is it happening? (Part 2 of 3) – how hindsight, hierarchy, and confirmation biases break conductivity and accuracy of knowledge
by Paweł Sysiak — 2023-06-09T23:00:55.043Z

Crossposted from Pawel’s blog

This essay has three parts. In part one, I include notes on epistemic status, I give a summary of the topic. But mainly I describe what is the expert trap.

Part two is about context. In “Why is expert trap happening?” I dive deeper explaining biases and dynamics behind it. Then in “Expert trap in the wild” I try to point out where it appears in reality.

Part three is about “Ways out”. I list my main ideas of how to counteract the expert trap. I end with conclusions and with a short Q&A.

How to read it? In this part, there may be a lot of knowledge that you already are familiar with. All chapters make sense on their own. Feel free to treat this article like a Q&A page and skip parts of it.

Why is expert trap happening?

I think there are three main causes of the expert and they are: forgetting, hierarchy bias, and my-side bias.

Forgetting

First, forgetting. Keep in mind I am not quoting any relevant research here. This is my hypothesis. The core mechanism is the following. When we learn something we forget how was it not to understand it. We are in a constant process of forming new memories and forgetting old ones. Some are prioritized over others. In context of learning, as soon as we learn something we tend to deprioritize reasons why it was hard to understand it. It may be costly to keep the score of it because we need to keep a second, seemingly more important thread – how to understand the problem we just learned about. And these two are different. It is as if we unplug certain cables connecting old contexts and plug them into new ones. Maybe they seem redundant to our brain as the direction is to keep ascending further.

Also, if my-side bias is in play (on that later) we will deprioritize memories that don’t strengthen our self-image. Sate of confusion, not being able to understand things feel unpleasant like a shoe that is too tight. So we may deprioritize these memories and prioritize ones that are leaving us with positive states: Everything fits correctly. I am smart.

Hierarchy bias

I see the expert trap as largely caused by hierarchy bias. In Elephant in the brain Robin Hanson and Kevin Simler explain this distortion. The main thesis of the book is that we are deeply hierarchical creatures and at the same time don’t view ourselves this way.

Elephant in the Brain made a large impression on my thinking. Suddenly a lot of ...
Expert trap: What is it? (Part 1 of 3) – how hindsight, hierarchy, and confirmation biases break conductivity and accuracy of knowledge
by Paweł Sysiak — 2023-06-09T23:00:09.121Z

Crossposted from Pawel’s blog
This essay has three parts. In part one, I include notes on epistemic status, I give a summary of the topic. But mainly I describe what is the Expert trap.

Part two is about context. In “Why is expert trap happening?” I dive deeper explaining biases and dynamics behind it. Then in “Expert trap in the wild” I try to point out where it appears in reality.

Part three is about “Ways out”. I list my main ideas of how to counteract the Expert trap. I end with conclusions and with a short Q&A.

Intro

Version: 0.3. This is still a very early version of this article. Yes, this thing has versions and will improve with time.

Feedback: I would love to understand what I am missing or learn about any counter-arguments. Feel free to write a comment, dm me, or share anonymous feedback here: sysiak.com/feedback.

Writing style: I don’t normal write. English is not my native language. But I care about being simple and precise. Read more about my writing approach and values here: sysiak.com/about

Epistemic status: I think it’s a great practice, to begin with Epistemic status. That is state your certainty, effort, link to sources, and point to main counterarguments. In this case, however, feel free to skip it and come back to it at the end. I am proposing here quite a large statement and I am writing with quite a lot of uncertainty. It may be interesting to first evaluate claims on your own. At the end of the series, I will remind you about coming back to epistemic status and I will post a short Q&A that will hopefully clarify some gaps.

Epistemic status (optional)

Certainty: If I were to pick one confidence interval for the main claim it would be Likely (70%). There are also parts of the described dynamics that I think are Highly likely (90%) or Almost certain (>97%). These are for example explanations about my-side bias, confirmation bias, and hindsight bias which I think are established and largely non-controversial. But I also have quite a lot of uncertainty. There are a lot of claims that are around Maybe (50%) (hovering between 35% and 75%). I think the most uncertain part of the knowledge is Hierarchy bias and ideas from “Elephant in the Brain”. This is a sweeping reinterpretation of how human motivations work. Personally, I think it’s Likely (~70%) that hierarchy bias largely explains our behavior. But I understand others will find it a lot ...