trentmkelly/LessWrong-43k huggingface dataset:
Culture wars in riddle format
by Malmesbury — 2022-07-17T14:51:54.744Z

(Cross-posted from Telescopic turnip.)

I’ve come together with the fact that humans will never find the truth about culture-war issues. We just can’t help it. I tried checking the facts, listening to the other side, examining the social context, deconstructing the power dynamics. I tried mistake theory, conflict theory, critical theory, but the result is usually the same: after checking all the evidence, I come to the conclusion that my favourite political tribe is always right, and everyone else is an idiot. What a coincidence, the ultimate truth is whatever is most popular in my personal social circle!

This is unfortunate, because the prospect of stuff like social justice is actually pretty great. Everybody agrees that it’s better when society is fair. But social justice is a calibration problem, and a hard one: there’s only one way to be fair, and infinitely many ways to be unfair. To calibrate society, we need a unbiased bird-eye view of it, and that’s where we suck. The human brain is utterly incompetent at this task. Our perception of what is fair is tuned differently for different people. It’s becoming increasingly clear that people carry different genetic variants biasing their brains in different directions, so we may never be able to agree with each other. Did you know that there are documented cases of people turning into conservative authoritarians after brain trauma? Like, you have a car accident, and when you wake up, women must obey their husbands and Infowars makes sense. I have no doubt these patients can come up with very sound arguments about why they changed their mind. How are we supposed to balance the power imbalances in these conditions!?

A culture-wars riddle

In last resort, I’d like to try something different. Here is a culture-wars riddle. It’s like your typical math riddle, except it’s a sociological one. The argument is deliberately hidden and you have to come up with it yourself. Maybe (maybe) it will make it easier to put your existing opinions aside and have a fresher look at the issue. Let’s give it a try.

There is a phenomenon, related to [thunder noise] the gender pay gap, that I came across during my usual peregrinations on the information superhighways. I think this phenomenon is important and more people should take it into consideration. But I won’t tell you what it is. Instead, you will have to guess. Doesn’t that sound ext...
Resolve Cycles
by CFAR!Duncan — 2022-07-16T23:17:13.037Z

Epistemic status: Anecdotally strong

This technique was largely developed by Kenzi Amodei in the context of after-workshop followups and pair debugging. It has been refined and iterated, and has proven highly useful to our alumni, but all theorizing is post-hoc and untested, and direct research into (e.g.) an underlying theory of mind has yet to be done.

Consider the following scenarios:

You’ve been assigned a task that feels like it’s going to take about ten or fifteen hours of work, and you’ve been given three weeks to get it done (e.g. a document that needs to be written).
You’re facing a problem that you’ve tried solving off and on again for years, a problem that your friends and family never seem to run into (e.g. a struggle with motivation as you try to learn a new skill).
There’s a thing you need to do, but it seems impossibly huge or vague (e.g. to achieve your goals you’d need to found a company, emigrate to India, or cure a disease), and you don’t know where to begin.
You’re pretty sure you know all the steps between you and your goal, but there are about forty thousand of them (e.g. you’re hoping to run an actual marathon).
You’ve got a to-do list that’s long and growing, and you can only ever manage to get to the ones that are urgent (e.g. getting your car’s registration renewed, two months late).

Problems like the ones above can range from trivial to crucial, from simple to complex, and from one-time bugs to persistent, serious drains on your time, attention, and resources. There are a lot of elements in the mix—motivation, creativity, perseverance, prioritization—and a lot of justifiable reasons for thinking that solutions will be hard to come by.

Sometimes, though—despite every bit of common sense and experience telling us otherwise—those solutions aren’t hard to come by. Or rather, they might be hard, but they’re not elusive or mysterious or complicated.

The resolve cycle technique is one we offer up with a sort of shamefaced shrug, because it doesn’t sound like “real” applied rationality. It doesn’t have the rock-solid research underpinnings of TAPs or inner sim, or a carefully considered model like the ones behind turbocharging and double crux. It sometimes comes across like the worst possible advice—the sort of thing people say when they don’t actually want to help you with your problem:

“Have you tried setting a five-minute timer and just, y’k...
Alignment as Game Design
by Shoshannah Tekofsky — 2022-07-16T22:36:15.741Z

This essay is a continuation of the Naive Hypothesis approach wherein I attempt to write up and share thoughts on AI alignment before having absorbed much of the existing literature. The goal is to preserve a higher level of novelty of ideas by generating my own models a priori of having read the thoughts of others. Later I hope to update these models a posteriori with the knowledge from existing frameworks. The fusion of the two approaches hopefully maximizes the probability of finding new angles on the AI alignment problem. In expectation, my current thoughts are high on errors, already-solved issues, and already-refuted arguments, but also high on novelty and shifts in perspective compared to existing frameworks.

Outer Alignment as Gamification

Krakovna et al.'s list of specification gaming issues reads like an inventory of video game exploits. Which in turn makes me wonder if we can hire video game designers as outer alignment engineers. And then the next step would be to look for speedrunners and QA testers to test the objective specifications that they develop. Though eventually these would have to be superintelligent, so you'd have to train AI systems to preemptively seek out exploits in objective specifications for AGI.

Either way, the game design analogy seems surprisingly robust to me: To make AGI do what we want, we gamify the real world and feed it points for desirable actions. Following that reasoning further we can argue there is an art to building reward functions that lead reliably to desired behavior. But then, can we convert that art to a science? More specifically, what would happen if we team up world-class game designers with world-class AI engineers to extract this art and convert it in to a meta-specification of how to specify goals for AGI?

Inner Alignment as Hacking Prevention

The best way to maximize your score in any game, is to hack the game and set your score to maximum. The only reasons not to a hack a game are:

You are not able.
You are not willing.

AGI will be able, but is there any reason that it might not be willing to hack its reward function? In other words, can we build a game for AGI that's so good it won't want to escape?

But what would "good" mean in this case? Possibly it means that the reward function is such a strong attractor that there isn't much incentive to explore far outside the intended bounds of the "gam...
Risk Management from a Climbers Perspective
by Annapurna — 2022-07-16T21:14:01.671Z
Link: https://jorgevelez.substack.com/p/risk-management-from-a-climbers-perspective

Me on the summit of Aguja Guillamet, Fitz Roy Massif, Patagonia, Argentina

In the past year I have been going on climbing adventures that have forced me to be more mindful about the risks I am taking and making decisions accordingly. In making those decisions, I have seen some parallels between the risks I am taking while climbing and the risks I took while investing. A podcast I listen to did a deep dive into risk taking, risk management, and the decision making process. Listening to this podcast has helped me put into words the decisions and processes I have been honing while climbing.
In this post, I will be sharing my thoughts and comments on

Climbing Gold Season 3: A deep dive into the art of risk taking

, as well as providing my own interpretations on how many professional climbers look at risk and how those views can be applied to making all kinds of decisions, from climbing to investing to regular day to day life decisions.

What is risk?

I find

Alex’ definition of risk

to quite elegant despite its simplicity.

Risk = Severity of Outcome x Likelihood of Outcome

What is severity of outcome? The consequence of the negative result of the decision. In climbing, a negative result from a decision typically involves serious physical harm or death. In investing, it typically means losing money.

What is likelihood of outcome? It is the probability of the negative outcome occurring. This is the difficult part of the equation, as this probability sometimes is very hard to quantify. In some decisions, such as flying, it is quite easy to find this input.

A quick google search says that the likelihood of dying in a plane crash is 1 in 11 million.

Easy! So despite the severity of outcome being horrible (death), the likelihood is negligible, therefore the risk is quite low.

Other decisions, such as alpine climbing, finding the likelihood of outcome input can be quite complex. There are many factors to take into account when deciding to head up an alpine rock climb, which magnifies the complexity in trying to arrive at the likelihood of a negative outcome. I will give you an example: the sun.

Experience and deductive reasoning have taught alpinists that sun facing routes tend to be much more loose as the sun melts snow, ice, and permafrost, which could potentially rain down on you while climbing. When deciding to climb a sun facing route on the mountains, an alpinist has to cons...
Cognitive Instability, Physicalism, and Free Will
by dadadarren — 2022-07-16T13:13:01.915Z
Link: https://www.sleepingbeautyproblem.com/cognitive-instability-physicalism-and-free-will/

Against Boltzmann Brain

"Cognitive Instability" is a term coined by Sean Carroll for a type of argument against Boltzmann brain. Simply put: if you believe you are a disembodied brain that came into existence due to random fluctuation as the theory suggests, then you have no logical basis in believing said theory. Because the supporting evidence: your perceptions and memories, your very thought that the theory is correct, are the result of said fluctuation. You can't expect them to reflect true physics. It is unstable because it cannot be true and justifiably believed at the same time.

This counter doesn't completely rule out you being a Boltzmann brain as a logical possibility. No argument can do that. Yet it convincingly shows why we shouldn't put our faith in it: it's unlikely a fluctuation just happens to produce the brain, i.e. yourself, with the correct physical theory of the universe in its mind. Claiming the contrary would be arrogant and teleological. Like many ingenious arguments, "cognitive instability" has an element of self-reference in it.

What About Free Will?

Sean Carroll is probably not going to like this. But the same argument can be used against physicalism and in particular the denial of free will.

For this discussion, let's say something's behaviour is "free" as in it cannot be fully explained by physics. In another word, it's not physically causally closed. Here it's not about if we currently have the technical ability to analyze it, or if the world is ultimately deterministic or stochastic. Free will means it fundamentally cannot be physically reduced, not even in principle. Most people, physicalists for certain, would say this kind of free will cannot be real. Otherwise, it would be a direct violation of physical laws.

This view would regard your thoughts and actions as the outcome of the underlying physical processes. There is no other contributing factor. Our sense of free conscious choices can be seen as an illusion, or "what it feels like from the inside" of the actual processes, or maybe something else. But no matter how we choose to interpret that, it has no causal power.

But this means your belief is merely a product of this physical system called a person, by its internal dynamics and interactions with the environment. The feeling of thinking based on reasoning and rationality has no bearing on the outco...
All AGI safety questions welcome (especially basic ones) [July 2022]
by plex — 2022-07-16T12:57:44.157Z

tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb!

Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics.

As requested in the previous thread[1], we'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask.

It's okay to ask uninformed questions, and not worry about having done a careful search before asking.

Stampy's Interactive AGI Safety FAQ

Additionally, this will serve as a soft-launch of the project Rob Miles' volunteer team[2] has been working on: Stampy - which will be (once we've got considerably more content) a single point of access into AGI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that! You can help by adding other people's questions and answers to Stampy or getting involved in other ways!

We're not at the "send this to all your friends" stage yet, we're just ready to onboard a bunch of editors who will help us get to that stage :)

Stampy - Here to help everyone learn about stamp maximization AGI Safety!

We welcome feedback[3] and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase.[4] You are encouraged to add other people's answers from this thread to Stampy if you think they're good, and collaboratively improve the content that's already on our wiki.

We've got a lot more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets.

Guidelines for Questioners:

No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read either the WaitButWhy posts, or the The Most Important Century summary from OpenPhil's co-CEO first that's great, but it's not a prerequisite

...
QNR Prospects
by PeterMcCluskey — 2022-07-16T02:03:37.258Z
Link: http://www.bayesianinvestor.com/blog/index.php/2022/07/15/qnr-prospects/

Approximately a book review: Eric Drexler's QNR
paper.

[Epistemic status: very much pushing the limits of my understanding.
I've likely made several times as many mistakes as in my average blog
post. I want to devote more time to understanding these topics, but
it's taken me months to produce this much, and if I delayed this in
hopes of producing something better, who knows when I'd be ready.]

This nearly-a-book elaborates on his CAIS
paper (mainly chapters 37 through
39), describing a path for AI capability research enables the CAIS
approach to remain competitive as capabilities exceed human levels.

AI research has been split between symbolic and connectionist camps for
as long as I can remember. Drexler says it's time to combine those
approaches to produce systems which are more powerful than either
approach can be by itself.

He suggests a general framework for how to usefully combine neural
networks and symbolic AI. It's built around structures that combine
natural language words with neural
representations of what
those words mean.

Drexler wrote this mainly for AI researchers. I will attempt to explain
it to a slightly broader audience.

Components

What are the key features that make this more powerful than GPT-3 alone,
or natural language alone?

QNR extends natural language by incorporating features of deep learning,
and mathematical structures used in symbolic AI.

words are associated with neural representations. Those
representations are learned via a process that focuses on learning a
single concept at a time with at least GPT-3-level ability to
understand the concept.
BERT
exemplifies how to do this.

words can be related to other word via graphs (such as syntax
trees).

words, or word-like concepts, can be created via compositions of
simpler concepts.

corresponding to phrases, sentences, books, and entities that we
have not yet conceived.

QNR feels much closer than well-known AI's to how I store concepts
within my mind, as opposed to the stripped down version that I'm using
here to help you reconstruct some of that in your mind.

Drexler contrasts QNR to foundation
models, but I
don't find "foundation models" to be a clear enough concept to be of
much value.

Importance?

I've noticed several people updating this year toward earlier AGI
timelines, based on an increasing number of results demonstrating what
look to me like marginally more general-purpose intel...
To-do waves
by Paweł Sysiak — 2022-07-16T01:19:57.854Z

Crossposted from my Substack blog.

Main idea

Most of our tasks work like a wave. We need to do them at a certain interval. I need to pee and pay taxes every sometime.

To-do waves have different lengths. There are small waves like I need to pee, eat, and drink. There are daily waves like clean teeth, dress up, move my body. Couple of days long waves like socializing, grocery shopping, and bringing the trash out. And a week, year, or longer waves like cutting nails, refueling a car, renewing a passport, doing a checkup at the doctor, filing taxes. There is a high probability that each day there will be waves from all lengths buckets.

And there are so many different categories of tasks. Taking care of a human pet – an expression from the Wait But Why blog, meaning all the little things you need to do to take care of yourself, essential things for survival, and chores. There are things you simply like and gonna do: internet addictions, watch/read cool things in a save folder, go for ice cream. There are things that are super urgent “I run out of time with a parking meter” or agenda you feel is right to do like helping a friend or going to a protest. There are straight procrastination items – when you pick a thing of least resistance not to tackle a more demanding item. And there are tasks tiny but abundant like attaching a charging cable to devices, refilling a water bottle, or eating a snack. We usually underestimate how many things we need to-do in our lives.

By default, my expectation is that I have this open plane of time when I can work on the important to-do. The reality looks completely different. My life is full of to-do items that it is hard to foresee. In practice I am left with is shuttered timeline and slivers of time where I can try to accomplish something important.

FAQ / what to do about it?

💡 Feel free to treat the below section like an FAQ – delve into sections that interest you

Why this is important? Uninterrupted time as a foundation of productivity

I think we tend to underestimate to what extent context switching is demanding on our cognition. Paul Graham in his text on Good and bad procrastination sees that one of the most overlooked properties of productivity is the number of interruptions one has. I think it works this way because every time we do something complex we need time to load all the variables into our short-term memory. It’s like keepin...
Moneypumping Bryan Caplan's Belief in Free Will
by Morpheus — 2022-07-16T00:46:03.176Z

I was surprised to learn Bryan Caplan beliefs in Free Will. He also likes to bet on his beliefs.

So here's my plan for money pumping Bryan Caplan or other libertarians who confused themselves about free will to the point where they are arguing, by definition, that the causes of a choice cannot have a causal explanation or not be "free": get him to play successive rounds of rock-paper-scissors against world champions for small amounts of money. Of course, this would only work as long as he does not learn to run some pseudorandom number generator inside his head. But in my experience, most people are terrible at rock paper scissors and don't run a random strategy. So if free will were a thing, would that imply people good at rock-paper-scissors have more of it than others?
A summary of every "Highlights from the Sequences" post
by Orpheus16 — 2022-07-15T23:01:04.392Z

1

I recently finished reading Highlights from the Sequences, 49 essays from The Sequences that were compiled by the LessWrong team.

Since moving to Berkeley several months ago, I’ve heard many people talking about posts from The Sequences. A lot of my friends and colleagues commonly reference biases, have a respect for Bayes Rule, and say things like “absence of evidence is evidence of absence!”

So, I was impressed that the Highlights were not merely a refresher of things I had already absorbed through the social waters. There was plenty of new material, and there were also plenty of moments when a concept became much crisper in my head. It’s one thing to know that dissent is hard, and it’s another thing to internalize that lonely dissent doesn’t feel like going to school dressed in black— it feels like going to school wearing a clown suit.

2

As I read, I wrote a few sentences summarizing each post. I mostly did this to improve my own comprehension/memory.

You should treat the summaries as "here's what Akash took away from this post" as opposed to "here's an actual summary of what Eliezer said."

Note that the summaries are not meant to replace the posts. Read them here.

3

Here are my notes on each post, in order. I also plan to post a reflection on some of my favorite posts.

Thinking Better on Purpose

The lens that sees its flaws

One difference between humans and mice is that humans are able to think about thinking. Mice brains and human brains both have flaws. But a human brain is a lens that can understand its own flaws. This is powerful and this is extremely rare in the animal kingdom.

What do we mean by “Rationality”?

Epistemic rationality is about building beliefs that correspond to reality (accuracy). Instrumental rationality is about steering the future toward outcomes we desire (winning). Sometimes, people have debates about whether things are “rational.” These often seem like debates over the definitions of words. We should not debate over whether something is “rational”— we should debate over whether something leads us to more accurate beliefs or leads us to steer the world more successfully. If it helps, every time Eliezer says “rationality”, you should replace it with “foozal” and then strive for what is foozal.

Humans are not automatically strategic

An 8-year-old will fail a calculus test because there are a lot of ways to get the wrong answer and very fe

...