trentmkelly/LessWrong-43k huggingface dataset:
Counterfactuals are Confusing because of an Ontological Shift
by Chris_Leong — 2022-08-05T19:03:46.925Z

I recently became unstuck on counterfactuals. I now believe that counterfactuals are confusing, in large part, because they entail preserving our values through an ontological shift[1].

In our naive ontology[2], when we are faced with a decision, we conceive of ourselves as having free will in the sense of there being multiple choices that we could actually take[3]. These choices are conceived of as actual and we when think about the notion of the "best possible choice" we see ourselves as comparing actual possible ways that the world could be. However, we when start investigating the nature of the universe, we realise that it is essentially deterministic[4] and hence that our naive ontology doesn't make sense. This forces us to ask what it means to make the "best possible choice" in a deterministic ontology where we can't literally take a choice other than the one that we make.

This means that we have to try to find something in our new ontology that roughly maps to our old one. For example, CDT pretends that at the point of the decision that we magically make a decision other than that which we actually end up making. For example, we were intending to turn left, but at the instant of the decision we're magically altered to turn right instead. This works well for many scenarios, but notably fails for Newcomb-like problems. Here updateless counterfactuals are arguably a better fit since they allow us to model the existence of a perfect predictor. However, there is still too much uncertainty about how they should be constructed for this answer to be satisfactory.

Given that we're trying to map a notion[5] onto another ontology that doesn't natively support it, it isn't that surprising there isn't an obvious, nice, neat way of doing it.

Newcomb's problem exposes the tension between the following two factors:

a) wanting to ensure that the past is the same in each counterfactual to ensure that they are comparable with each other, in the sense of it being fair to compare the counterfactuals in order to evaluate the decision
b) maintaining certain consistency conditions that are present in the problem statement[6].

The tension is as follows: If we hold the past constant between counterfactuals then we've violated the consistency conditions as we have a point in time when the agent takes a decision despite the fact that this is inconsistent with the previous tim...
Gears-Level Understanding, Deliberate Performance, The Strategic Level
by CFAR!Duncan — 2022-08-05T17:11:16.678Z

Gears-Level Understanding

It seems to be important to distinguish between two kinds of knowing: the knowing that comes from listening to trusted sources, and the knowing that comes from seeing why the world couldn’t possibly be any other way.

Let’s imagine someone shows you a box with two gears partially sticking out of opposite sides:

At first, you don’t know what will happen if the gear sticking out on the left is rotated downward. It could send the right gear downward, or it could send it upward; it could have absolutely no effect whatsoever. If the person tells you that the result will be the right gear rotating downward, you’ll either take it on faith, or you won’t, depending on how confident they seem and how much you trust them.

If, on the other hand, you look inside the box yourself ...

... then you’ll gain a very different sort of confidence. It might take some work, but after a little thinking, you can know that the person’s claim is wrong. It doesn’t matter what sort of expertise they have, relative to you, or how much others respect their insight, or whether most people think you’re crazy for disagreeing with the established answer, because you can see how the gears must move. It would be deeply confusing for the other person to turn out to be right, in this case—it would violate your understanding of physics, similar to how an engineer who has spent some time understanding gyroscopes would be shocked if they suddenly started behaving the way a five-year-old expected them to.

There are two important takeaways, here. First, it’s important to recognize that it’s possible to achieve a “gears-level understanding” of any phenomenon, even if it might in practice be very difficult to achieve.

Second, you want to develop insight into whether or not you have it, in any particular case. For instance, you might find it confusing if turning a doorknob had no effect on the door opening, but it’s unlikely to violate your sense of reality. Does your System 1 seem to think that your car and your computer work by magic, and could just—stop working? Do your patterns of drive and motivation feel mysterious to you?

Gears-level understanding isn’t always a reasonable target, but all else being equal, having it is better than not, and seeking it is a good way to learn. It’s not the be-all-and-end-all, but it’s another lens to use when deciding whether or not...
COVID-19 Group Testing Post-mortem?
by gwern — 2022-08-05T16:32:55.157Z

As Covid is for the most part over and tests have not been in short supply for a very long time now, eliminating most of the rationale for complicated testing procedures meant to optimize test count/cost, it would be interesting for anyone who was involved in this to do a retrospective or post-mortem. Was pool testing useful? Why or why not? Can it be improved for future pandemics and should we care about it?

My impression, not having paid much attention to the details at the time (AI scaling was more important), is that various group/pool testing approaches were useful in a few niches, but were rendered useless in many places due to politics/organization/social flaws, and didn't apply to most places.

I read that use of pool testing was common in China early on in 2020 as part of the hotspot strategy to economize on testing extremely large groups of people where only a few would have active Covid. This is the most logical place to apply it, and it was a major government & societal priority, so that is a success.

I get the impression that they were able to switch to individual testing fairly early, in order to get faster turnaround and do more localized testing. But to the extent that pool testing was effective from the start, before testing manufacturing could scale up, and helped squash Covid when it was still squashable, then it was very valuable even if obsoleted.

Of course, this evaluation can change depending on how Covid ultimately shakes out in China. Was it part of a heavyhanded but ultimately highly effective Covid Zero policy which led to it turning into a hermit country enduring devastating economic fallout while merely postponed the inevitable variant which could defeat Covid Zero? In which case, pool testing was a tool which played a part in buying China entire years to vaccinate but its good was rendered useless when they bizarrely failed to make any good use of the time, dabbling in ineffective vaccines etc, and in practice, only doubling down repeatedly on coercive testing and surveillance until the martingale blew up in their face. Or was it part of a visionary resolute effort which spared China the mass death of the rest of the world as it came out the other side by persisting so long in country-wide lockdown until Covid just sorta stopped being a problem somehow? Then its good was unalloyed and it's sad so few other countries could bo

...
Where are the red lines for AI?
by Karl von Wendt — 2022-08-05T09:34:41.129Z

Thanks to Daniel Kokotajlo, Jan Hendrik Kirchner, Remmelt Ellen, Berbank Green, Otto Barten and Olaf Voß for helpful suggestions and comments.

As

AI alignment remains terribly difficult

and

timelines appear to be dwindling

, we must face the likely situation that within the next 20 years we will be able to build an AI that

poses an existential threat

before we know how to control it. In this case, our only chance to avert a catastrophe will be to collectively refrain from developing such a “dangerous” AI. But what exactly does that mean?

It seems obvious that an AI which pursues the wrong goal and is vastly more intelligent than any human would be “dangerous” in the sense that it would likely be unstoppable and probably lead to an existential catastrophe. But where, exactly, is the tipping point? Where is the line between harmless current AIs, like GPT-3 or MuZero, and a future AI that may pose an existential threat?

To put the question differently: If we were asked to draft a global law that prohibits creating “dangerous AI”, what should be written in it? Which are the things that no actor should ever be allowed to do, the “red lines” no one should ever cross, at least until there is a feasible and safe solution to the alignment problem? How would we even recognize a “dangerous AI”, or plans to build one?

This question is critical, because if the only way to avert an existential risk is to refrain from building a dangerous AI, we need to be very sure about what exactly makes an AI “dangerous” in this sense.

It may seem impossible to prevent all of humanity from doing something which is technically feasible. But while it is often difficult to get people to agree on any kind of policy, there are already many things which are not explicitly forbidden, but most people don’t do anyway, like

letting their children play with radioactive toys

,

eating any unidentifiable mushrooms they find in the woods

,

climbing under a truck to drink wine while it is driving at full speed on the highway

or

drinking aquarium cleaner as a treatment against Covid

. There is a common understanding that these are stupid things to do because the risk is much greater than the possible benefit. This common understanding of dangerousness is all that is needed to keep a very large proportion of humanity from doing those things.

If we could create a similar common u...
Bridging Expected Utility Maximization and Optimization
by Daniel Herrmann — 2022-08-05T08:18:26.481Z

Background

This is the second of our (Ramana, Abram, Josiah, Daniel) posts on our PIBBSS research. Our previous post outlined five potential projects that we were considering pursuing this summer. Our task since then has been to make initial attempts at each project. These initial attempts help us to clarify each project, identify the primary problems that need to be solved, and perhaps discover a promising line of attack or two.

This post is aimed at the second proposal from our previous post. There we asked: what is the connection between an agent that maximizes expected utility and an agent that succeeds in action? Here we will outline a few of the problems we see in this area and potential routes for solving them.

Expected Utility Maximization and Forming Expectations

In economics and formal philosophy, the standard characterization of a rational agent is an agent who maximizes expected utility. Informally, when such an agent has a set of options available to her, she chooses the one that maximizes the expectation of her utility function, where the expectation is taken relative to her subjective degrees of belief. The claim that expected utility (EU) maximization characterizes rationality is usually supported by representation theorems (see here for a good, quick introduction). Expected utility maximization plays a core role in the philosophical foundations of decision theory, game theory, and probabilism. Given that EU maximization plays such a central role in theories of rationality, and given that there is a vast literature surrounding it, it seems very plausible that EU maximization would help us think precisely about agency.

Despite this, it seems that that expected utility theory doesn’t seem to predict anything (or, at the very least, you need to combine EU maximization with certain complexity notions to get something that is weakly predictive). Obviously this is an issue, given that we want notions of agency to constrain our expectations about the behaviour and effects of concrete systems. This is the sense in which we want to bridge the gap between expected an agent’s utility maximization and its success in action (or, rather, our expectation that it will be successful in action). If we know that a (sophisticated) EU maximizer has a certain goal, then we should be able to infer something about the likely unfolding of the world.

The primary obstacle fo...
Deontology and Tool AI
by Nathan1123 — 2022-08-05T05:20:14.647Z

Please forgive me for being awkward on my first post, I just wanted to get these ideas out there to facilitate discussion on the topic. I also am sorry if I use any newly-discovered terminology incorrectly, but I am doing my best.

Introduction: Utilitarianism is Unaligned

Intuitively, I think that an important component of modeling a moral system for friendly AGI is to discuss what ethical philosophy such a system would be built off of.

From my experience, it seems that a lot of people who discuss a hypothetical moral system for friendly AI presumes that such a system would work on a utilitarian basis. For an array of generated policies, each policy is represented by some metric that represents the collective good of humanity or civilization, and the AI selects the policy where this value is maximized.

I feel that an approach like that tends to be naïve, and isn't an expectation we actually hold for humans in the real world, much less what we expect from an AI. From what I understand, Utilitarianism has been criticized for as long as it has been formally stated, despite such famous adages like "the needs of the many outweigh the needs of the few". Such thought experiments as the Trolley Problem are meant to give us the impression that killing n people is permissible if it allows n+1 people to survive, but this doesn't align with our moral conscience. This also doesn't work for a superintelligent AI, because a policy that will sacrifice 49% of the population for the benefit of 51% of the population should be totally unacceptable.

In reality, we want a moral system that says killing people is always wrong, at any time, for any reason, and under any circumstance. No matter how cosmically intelligent the AI is, there is no scenario where such a sacrifice is appropriate. Thus, a mathematical model based in Utilitarianism, while making sense on paper, is ultimately unaligned from our actual intentions in terms of morality.

Deontology as an Ethical Foundation for FAI

Now, I am aware there are more ethical systems out there besides Utilitarianism and Deontology, but I just wanted to outline here some thoughts I had on how utility functions based on a Deontological model could be constructed. I haven't seen anyone else on the site use this approach, but if there is any existing literature out there that discusses Deontological models for friendly AI, I would really lik...
An attempt to understand the Complexity of Values
by Dalton Mabery — 2022-08-05T04:43:25.495Z

Let's pretend one day you find magic machine that could grant any lifestyle you asked it for. Professional golfer? Done. Star of Pulp Fiction? Say no more. Enlightened writer? Easy. Any lifestyle you wished to live could be yours upon demand.

But like with any magical box or genie, there's a catch: The only thing it gave you was exactly what you asked it for.

If you asked to be a star of Pulp Fiction, it could do that, but who knows what Pulp Fiction would become. It certainly wouldn't be the genre-breaking film it was. Perhaps it would be a book about orange juice, but hey, you'd be the star!

You could be a professional golfer, but it would be the year 1100 and golf would be nowhere near the sport it is today. It would take your prompt, but it would not give you the assumed preferences–the things you value and care about–unless you specifically asked for every single one.

Let's look at a more concrete example: that of a young couple wishing to get married, buy a house with a white picket fence, and have two kids with a dog. If this couple gave the machine with that criteria for a lifestyle, they would get their house with a white picket fence and two kids, but they might be the only people on their side of the Mississippi. Since they didn't specify they wanted a car–or any other technology for that matter–they wouldn't have it.

But for the purpose of this thought experiment, let's say before the young couple entered their lifestyle preferences into the machine, they had a teacher instruct them on the complications of it:

"Make sure to specify exactly the kind of lifestyle you want to live. If you want to live in Paris in the 1950s, make sure to specify you want to live in the 1950s Paris, just as it was in the real 1950s," says the teacher. "This machine will do exactly what you tell it to do with no assumed preferences or inherited human belief systems, so be specific and precise about the things you care about. Don't forget anything!"

Even on a good day, that couple isn't getting their lifestyle exactly as it is today. Maybe they have the technology that they have today, but since they didn't specify that they also wanted the ability to love, they now hate each other.

Complexity of value

The task of defining your every preference is difficult. If you didn't follow along with that thought experiment, just imagine all the different ideals and preferences you have ab...
$20K In Bounties for AI Safety Public Materials
by Dan H — 2022-08-05T02:52:47.729Z

TLDR

We are announcing a $20k bounty for publicly-understandable explainers of AI safety concepts. We are also releasing the results of the AI Safety Arguments competition.

Background

Of the technologists, ML researchers, and policymakers thinking about AI, very few are seriously thinking about AI existential safety. This results in less high-quality research and could also pose difficulties for deployment of safety solutions in the future.

There is no single solution to this problem. However, an increase in the number of publicly accessible discussions of AI risk can help to shift the Overton window towards more serious consideration of AI safety.

Capability advancements have

surprised many in the broader ML community

: as they have made discussion of AGI more possible, they can also contribute to making discussion of existential safety more possible. Still, there are not many good introductory resources to the topic or various subtopics. If somebody has no background, they might need to read books or very long sequences of posts to get an idea about why people are worried about AI x-risk. There are a few strong, short, introductions to AI x-risk, but some of them are out of date and they aren’t suited for all audiences.

Shane Legg, a co-founder of DeepMind, recently said the following about AGI:[1]

If you go back 10-12 years ago the whole notion of Artificial General Intelligence was lunatic fringe. People [in the field] would literally just roll their eyes and just walk away. [I had that happen] multiple times. [...] [But] every year [the number of people who roll their eyes] becomes less.

We hope that the number of people rolling their eyes at AI safety can be reduced, too. In the case of AGI, increased AI capabilities and public relations efforts by major AI labs have fed more discussion. Similarly, conscious efforts to increase public understanding and knowledge of safety could have a similar effect.

Bounty details

The

Center for AI Safety

is announcing a $20,000 bounty for the best publicly-understandable explainers of topics in AI safety. Winners of the bounty will win $2,000 each, for a total of up to ten possible bounty recipients. The bounty is subject to the Terms and Conditions below.

By publicly understandable, we mean understandable to somebody who has never read a book or technical paper on AI safety and who has never read LessWrong or the...
Two Kids Crosswise
by jefftk — 2022-08-05T02:40:04.924Z

When traveling, we sometimes end up somewhere with the older two kids
sharing a double bed. Initially we would put them down the normal
way, as the bed was made to be used, but recently we realized it goes
much better if you think of the bed as being sideways:

Kids are pretty short, and if they sleep crosswise they're farther
from each other, which minimizes fighting. Our kids don't fight that
much normally, though they don't get along as well as they did a year
ago, but bedtime while traveling is one of the worst situations:
they're extra sleepy, and the sleepier they are both the worse their
impulse control is and the more irritable they are. Rotating the
arrangement makes it easy for them each to have their own space.
Separate sheets and blankets also help.

A Full
is 54" wide and a Queen is 60" wide, so the median
child
gets too tall for this at about 9y on a Full and 12y on a Queen.

Comment via: facebook
The Falling Drill
by Screwtape — 2022-08-05T00:08:26.069Z

Summary: The Falling Drill is intended to practice being wrong in a comfortable practice environment, intended to help us handle being wrong out in the world easier and with less stress.

Tags: Small, repeatable

Purpose: The first thing you learn in martial arts is the ability to fall. The corresponding rationalist skill is the ability to realize that you are wrong. Since modern western society (and possibly most human societies) discourage admitting you’re wrong, it can help to do it repeatedly and get used to it.

Materials: A device that can connect to wikipedia. A list of partial statements such as “The population of Boston is. . .” and “the melting point of mercury is. . .” A suggested list is here, and we suggest writing the questions down on individual cards before the meetup so each person only sees one card at a time.

Announcement Text: One of the most important parts of intellectual progress is learning to change your mind. The first step of changing your mind is realizing that you were wrong about something. Today we're going to practice that often painful realization, in a small way and in a low pressure situation. Without this skill, how can you debate an important issue or confront a challenging topic? You might argue long after it's clear to others that you've lost, because admitting it feels like defeat.

The first lesson any martial artist learns is often how to fall. There are ways to make the landing easier, but they all start from the knowledge that falling isn't the worst thing in the world.

Description:

1. Read one of the statements, then complete it as best you can. "The population of Boston is four hundred thousand."

2. Look it up on wikipedia. If you’re wrong, then announce to the room “I was wrong about the population of Boston. It's over six hundred thousand.”

3. Hand the questions to the next person in the circle, and the process begins again.

Variations: Practice saying “I don’t know” when you’re asked the question.

Practice saying your answer loudly and confidently, such as "The population of Boston is four hundred thousand! Only an idiot would think it was higher than five thousand!" It is good to know what you do not know, but it is also important to be able to back down from a strong claim.

Practice doing this with confidence intervals. "The population of Boston is between four hundred thousand and six hu...