trentmkelly/LessWrong-43k huggingface dataset:
Implications of the Grabby Aliens Model
by harsimony — 2021-12-06T18:34:44.985Z
Link: https://harsimony.wordpress.com/2021/12/05/implications-of-the-grabby-aliens-model/

Hanson’s Grabby Aliens Model is one way to explain the apparent lack of other intelligent life in the universe. Essentially, if you believe it’s possible for extraterrestrial civilizations to expand across the galaxy and transform large amounts of matter, then most observers (e.g. people) have to arrive earlier than these civilizations; since we wouldn’t exist if the extraterrestrials got here first. But others can explain it better than me, instead, I want to focus on the implications of the model assuming it’s true.

Hanson has previously claimed that the lack of alien civilizations suggests that it is typical for life to go extinct through an unknown process called the Great Filter. The Great Filter is pretty worrying, because it suggests there might be some unknown x-risk which is hard to avoid.

But the grabby aliens model offers an alternative. If we don’t see other civilizations because of selection effects, then we don’t have to worry as much about the great filter, and this reduces our expected probability of an existential catastrophe [1].

Grabby aliens also changes the value of existential risk reduction. On one hand, the upper bound on the size of our civilization is much smaller under grabby aliens since we must share the universe with many different civilizations. On the other hand, even if we went extinct, the universe wouldn’t remain empty since some other civilization would be there to take our place.

The presence of competing civilizations changes how we should prepare for the future. The universe fills up much faster if there are many many civilizations growing in parallel [2]. Moving quickly can allow humanity to gather a larger fraction of the universe for itself. Additionally, it is critical to develop tools for collaboration and resource sharing between alien species.

The possibility of meeting alien civilizations also means that we have to consider how our actions will effect our reputation [3]. Radio broadcasts have been leaving earth for a century. What do they say about us?

Being able to establish a reputation before meeting another civilization is crucial. The time period before we become aware of other civilizations gives us an opportunity to pre-commit to certain courses of action. For example, we might pre-commit to being friendly to cooperative aliens. Other civilizations may make similar commitments; but what is the equilibrium of th...
A Possible Resolution To Spurious Counterfactuals
by JoshuaOSHickman — 2021-12-06T18:26:41.409Z

Spurious counterfactuals (perhaps an easier handle than "the lobian inference issues described in Section 2.1 here, in the Embedded Agency paper ), are important to address because they lead to inference problems. Having a function (or agent) rely on proofs about itself to generate its output immediately introduces Lob's Theorem-related issues -- agents may very well want proofs that their behavior leads to an outcome to determine their actions, but that makes a lot of actions possible to find proofs for.

Uncertainty does not help much, but reveals a greater critique: the utility in a counterfactual is not information inferred from our making the decision -- we can't decide if a choice is good just because, in the circumstances we can imagine making that choice, well, it was the right choice, wasn't it? This seems relatively unavoidable for all agents who notice they are competent decision makers, under any level of uncertainty. A smart professor will notice, in the subjunctive case where he slashes the tires on his own car, it must have made a lot of sense, he wouldn't do that for no reason. Perhaps he needed an unimpeachable excuse for not being somewhere he desperately wanted to avoid. But seeing high reward in that case isn't relevant to the counterfactual imagining where he sees, if he slashed his tires right here and now, it would be a massive bummer.

I think explicit agent self-pointers resolve this issue. Taking the example from the paper (U being the universe and A being the agent):

A :=
Spend some time searching for proofs of sentences of the form
“[A() = 5 →U() = x] & [A() = 10 →U() = y]” for x,y ∈{0,5,10}.
if a proof is found with x > y:
return 5
else:
return 10

U :=
if A() = 10 :
return 10
if A() = 5 :
return 5

We can replace it with (I've added a parameter for "which decision is being made" because that clarifies the intent behind the counterfactual agent change, but obviously this is still a highly simplified view):

Situation :: Nonce
Agent :: Agent -> Situation -> Choice
Universe :: Agent -> Reward

CounterfactuallyDoing(PriorAgent, CounterfactualSituation, Choice) :=
NewAgent := \Agent, Situation ->
if Situation == CounterfactualSituation:
return Choice
else:
return PriorAgent(NewAgent, Situation)
return NewAgent

A(Self, Situation == OnlySituation) :=
Spend some time searching for proofs of

...
Information bottleneck for counterfactual corrigibility
by tailcalled — 2021-12-06T17:11:12.984Z

Previously, I wrote a post introducing a strategy for corrigibility based on counterfactuals.

There were various objections to this of various strengths. I don't think any defeating the notion of using this style of approach for the stop button problem specifically, but So8res presented one style of argument that, after thinking about it, I have to admit seems like a pretty strong challenge for applying my approach beyond the stop button. In this post, I will be trying to repeat the argument using my own frames.

Obviously anything bad about this argument should be blamed on me misrepresenting it, while anything good can be credited to So8res. 😉

Counterfactual stop button

The stop button problem is simple enough: We want humans to decide whether an AI gets shut down or continues optimizing. The AI shouldn't try to mess with humans' ability to do so, and it shouldn't be incentivized to manipulate humans to either shut it down or keep it running: It might even be good if it directly helped provide humans with control in noninvasive ways, but overall the point is that it should be up to the humans to decide what happens.

The stop button problem seems like a single instance of a more general problem in corrigibility and alignment, where you might want to provide people with some way of controlling AI, ideally in much richer ways than just a binary choice between shut down and continue.

My idea for this kind of problem is simple enough: Instead of rewarding the AI for the outcomes it produces, reward it for a causal connection between human wants and the relevant outcomes. Which theoretically is achieved via a utility function defined over counterfactual human behaviors, and practically achieved by training the AI across different counterfactual human behaviors.

Out of distribution

A problem that So8res pointed out is that the real world is out-of-distribution with respect to this training environment.

This is a result of a problem that I skip over in the original post. Back then I defined the utility function as:

U=Bs+Vf

Here, Xs and Xf represent the counterfactual values of X, with Xs being the value of X if humans want to stop the AI, and Xf being the value of X if humans don't want to stop the AI. But this is really undefined; there are lots of ways that one could want to stop the AI, depending on the exac...
Omicron Post #4
by Zvi — 2021-12-06T17:00:01.470Z

Previous Omicron updates: #1, #2, #3. Last weekly non-Omicron update.

An introductory word: Thanks to Dominic Cummings, I have a lot of new readers, many from the United Kingdom, so I want to welcome all of you, and I hope at least some of you will stay when I turn to non-Covid questions. I am American so these posts focus on the United States, but I keep an eye elsewhere too, and mostly we’re all in this one together and the same conclusions apply. If you have good news sources and follows to keep a better eye on the UK or Europe for Covid purposes, or data sources anywhere I may not have noticed, I invite you to share them in the comments.

The constant refrain from all sources is ‘we will know more soon.’ Soon is continuously arriving. The new information we are learning hasn’t completely ruled out many possibilities, but it is broadly consistent with Omicron spreading fast and having a high degree of immune erosion/escape in terms of infection but not protection against severe disease.

Thus, to headline the core takeaways at the top, here’s broadly where I’m at as of now (probability estimates at the end, as before). I’d be surprised if any of these were wrong.

Omicron spreads far more rapidly than Delta and is going to take over.
This will come at us fast. Omicron will be the majority strain by the end of January. This happening by end of year is on the table.
Omicron re-infects those who have already been sick, or breaks through to those who have been vaccinated, much more than Delta.
Previous infection continues to protect against severe disease, hospitalization and death.
Omicron doesn’t cause substantially more severe disease than Delta when it infects you, but we don’t know if it causes less severe disease yet, our evidence is ambiguous.
Omicron cases on average are much more mild because there will be a much higher percentage of re-infections and breakthrough cases, which are highly protected against severe disease.
The big peak is probably within a few months, so you don’t have time to wait for an Omicron-targeted shot. The current shots will still work against severe disease, so strongly consider getting vaccinated or boosted if you can.
The best thing we can do to minimize the harm Omicron causes is to ramp up production of therapeutics, especially Paxlovid, as rapidly as possible, along with the necessary tests and other logistics to get the treatments to

...
Life, struggle, and the psychological fallout from COVID
by Alex Flint — 2021-12-06T16:59:39.611Z

I have a friend in the effective altruism community who is facing an unusually slippery and disheartening struggle at the moment. This person is a long-time contributor to the community, and is held up as exemplary by other community members. Due to their integrity, they are in fact respected highly, yet this person often feels that this respect is about to slip away, and does not know whether the community will stand by them in the years to come. No matter how well the community takes care of its own, it is just not possible for any of us to know for sure that the community will stand by us through thick and thin, and this is a frightening thing to be uncertain about.

I have another friend who has recently been entrusted with quite significant resources as a leader within the community. They are expected to deploy these resources in service of the future of all life. Our society so prizes the position of allocating resources that we often simply call this "success", yet from the inside it is a very difficult position to hold, not just because it is difficult to allocate resources in service of the future of all life, but also because it is difficult to convey the extraordinary weight of this responsibility to others, and so a big lonely space opens up that almost no-one enters any more, and yet the furore of day-to-day decision-making continues hour by hour, and it is just not possible to put it aside, because it matters.

I felt this weight myself when I started a company a few years ago. I was only responsible for taking care of about 30 people, not the future of all life. Initially I was so happy that I was doing what I had wanted to do for so long, and was empowered to do it. But later, as we struggled to fund the company, I faced the furore of responsibilities, and realized that I was committed to a difficult path, which, due to integrity, I could not, and should not, put aside. I felt walls closing in on me, and I found myself holding tension in my body that just wound tighter and tighter, and still the mountain of decisions in front of me really mattered because they had real consequences.

I have another friend in the effective altruist community who is in full battle mode right now. This person feels that they are under attack, and, given actual circumstances, this is understandable. This person is actually very good at battle, has experience in it, has...
Modeling Failure Modes of High-Level Machine Intelligence
by Ben Cottier — 2021-12-06T13:54:38.147Z

This post, which deals with some widely discussed failure modes of transformative AI, is part 7 in our

sequence on Modeling Transformative AI Risk

. In this series of posts, we are presenting a preliminary model of the relationships between key hypotheses in debates about catastrophic risks from AI. Previous posts in this sequence explained how different subtopics of this project, such as

mesa-optimization

and

safety research agendas

, are incorporated into our model.

We now come to the potential failure modes caused by High-Level Machine Intelligence (HLMI). These failure modes are the principal outputs of our model. Ultimately, we need to look at these outputs to analyze the effect that upstream parts of our model have on the risks, and in turn to guide decision-making. This post will first explain the relevant high-level components of our model before going through each component in detail. In the figure below, failure modes are represented by red-colored nodes.

We do not model catastrophe itself in detail, because there seem to be a very wide range of ways it could play out. Rather, the outcomes are states from which existential catastrophe is very likely, or "points of no return" for a range of catastrophes. Catastrophically Misaligned HLMI covers scenarios where one HLMI, or a coalition of HLMIs and possibly humans, achieves a Decisive Strategic Advantage (DSA), and the DSA enables an existential catastrophe to occur. Loss of Control covers existential scenarios where a DSA does not occur; instead HLMI systems gain influence on the world in an incremental and distributed fashion, and either humanity's control over the future gradually fades away, or a sudden change in the world causes a sufficiently large and irreversibly damaging automation failure (see

What failure looks like

); alternatively, the use of HLMI leads to an extreme "Moloch" scenario where most of what we value is burned down through competition.

Moving one step back, there are three major drivers of those outcomes. Firstly, Misaligned HLMI is the technical failure to align an HLMI. Second, the DSA module considers different ways DSA could be achieved by a leading HLMI project or coalition. Third, Influence-seeking Behavior considers whether an HLMI would pursue accumulation of power (e.g. political, financial) and/or manipulation of humans as an instrumental goal. This third consideration is ...
A Framework to Explain Bayesian Models
by Jsevillamol — 2021-12-06T10:38:25.815Z

Bayesian Networks

are used to represent uncertainty and probabilistic relations between variables. They have an appealing graphical representation and a clear statistical meaning, which makes them a popular approach to manage uncertainty.

I have developed a new mathematical framework, inspired in message passing, to perform content selection and produce explanations of the reasoning in BNs.

In this article I will explain what it is possible to do with the framework, obviating the technical details of how it works. Those are available in the technical report

Finding, Scoring and Explaining Arguments in Bayesian Networks

.

An open source implementation of the framework is available in GitLab.

Previous work

After reviewing past work in Bayesian Networks, I concluded that current frameworks were inadequate to solve some key issues in explaining Bayesian Networks.

General explainability approaches like LIME

(Ribeiro et al, 2016)

and SHAP

(Lundberg and Lee, 2017)

already do a good job of helping us choose and explain the effect of the evidence that matters most for an explanation. But they cannot help us explain the intricate ways in which information propagates inside a Bayesian Network. So a solution particular to Bayesian Networks was desired.

Some existing frameworks addressed how to select the most important paths in a graphical model that linked the evidence to the nodes of interest. For example,

(Suerdmont, 1992)

described a method to extract such arguments in their INSITE system. The method was later refined in eg

(Haddawy et al, 1997)

and

(Kyrimi et al, 2020)

.

And other work focused on relating Bayesian Networks to argument theory, superficially studying the interactions between arguments.

(Vreeswijk, 2005)

,

(Keppens, 2012)

and

(Timmer et al, 2017)

are examples of this line of work.

But no previous work that I know of addressed both problems simultaneously. I propose a framework for selecting the most important ways the evidence relates to nodes of interest in a Bayesian Network, while being mindful of interactions in the network.

Introducing my framework

The best introduction to my work is an example. Consider the ASIA network depicted below:

The ASIA network, a common example of a Bayesian Network.

Suppose we learn that the patient’s xray showed an abnormality (xray = yes), that the patient has bronchitis (bronc = yes) and that the patient doe...
Anti-correlated causation
by DirectedEvolution — 2021-12-06T04:36:17.439Z

Example 1: exercise correlates with weight gain?

Let's assume that, when holding diet constant, exercise causes weight loss in the general population. Basic military training causes soldiers to exercise, but it also may cause weight gain in underweight soldiers. They need to build muscle to carry around their heavy gear, and must eat more in order to do so. This would result in a positive correlation between exercise and weight gain in this population.

So, in underweight soldiers:

Exercise causes weight loss.
Basic military training causes exercise.
Basic military training causes weight gain via increased eating. This effect is larger than that of the exercise-moderated weight loss.
Thus, exercise causes weight loss, yet is anticorrelated with it in this population.

Example 2: the cure is worse than the disease?

Likewise, a lab is developing microtube implants made of a known inert material to guide regenerating axons as a spinal cord injury intervention. It is hypothesized that microtube presence enhances functional recovery in their model.

Currently, it is extremely difficult to implant these tubes. Mice in the experimental group undergo longer surgeries, with more and longer potentially injurious insertions of instruments into the surgical site, than the control group.

Although microtube implantation causes recovery, the experimental surgery is more injurious to the experimental group mice than the control surgery. Therefore, presence of microtubes and survival/functional recovery are anticorrelated. Given the prior probability and expected utility of microtubes as an SCI intervention, it makes sense to invest more time trying to improve the surgical method.

Example 3: eating ice cream prevents hypothermia?

In summer, crime levels and ice cream sales increase, so criminal behavior and ice cream sales are correlated. By extension, winter drives ice cream sales down, while increasing rates of frostbite. Does eating ice cream prevent frostbite?

This is just the third variable problem in reverse!

In general: anticorrelation does not imply inhibition.

Suppose that A causes B.

C causes A, thereby causing B via one pathway.

C also inhibits B via another pathway, to a greater extent than the increase in B that it moderates via its influence on A.

We do a study or make an informal observation that involves manipulating C, intentionally or not; but our outcome measure is the correla...
ML Alignment Theory Program under Evan Hubinger
by ozhang — 2021-12-06T00:03:15.443Z

In the past six weeks, the Stanford Existential Risks Initiative (SERI) has been running a trial for the “ML Alignment Theory Scholars” (MATS) program. Our goal is to increase the number of people working on alignment theory, and to do this, we’re running a scholars program that provides mentorship, funding, and community to promising new alignment theorists. This program is run in partnership with Evan Hubinger, who has been providing all of the mentorship to each of the scholars for their trial.

As the final phase of the trial, each participant has taken a previous research artifact (usually an Alignment Forum post) and written a distillation and expansion of that post. The posts were picked by Evan and each participant signed up for one they were interested in. Within the next two weeks (12/7 - 12/17), we’ll be posting all of these posts to lesswrong and the alignment forum as part of a sequence, with a couple of posts going up each day. (There will be around 10-15 posts total.)

Community Engagement

Evan will be evaluating each post to determine whether participants make it to the next stage of the seminar program (where they have the opportunity to do novel research with a mentor), but we’d also be interested in hearing community feedback on each post. This could be just through upvotes or alternatively, via comments as well. We’ll run a conclusion post with our takeaways a week or two after the final blogpost has been released. If it’s interesting, we’d also be happy to post the story of MATS’ creation for other prospective community builders.

Additionally, if Evan knows you and you would be interested in mentoring one of the participants for the next stage of the program—e.g. you really liked their post and think it would be productive to work with them—you should reach out to Evan.

Program Description

From here on out, we’ll be discussing our program and future strategy. The previous two paragraphs are all the context needed to understand the series of blog posts.

The SERI MATS program aims to gather promising alignment theorists and give them an opportunity to learn from an experienced mentor.

The application process was as follows. We first asked the mentors of the Cambridge AGI safety fundamentals for promising participants and interviewed them. After the interviews, we put the scholars through a paid, six-week long trial period for 10 hours a week. During...