trentmkelly/LessWrong-43k huggingface dataset:
My default frame: some fundamentals and background beliefs
by Pontor — 2020-11-10T00:04:46.519Z

Following John Nerst’s noble example:

https://everythingstudies.com/2018/07/16/30-fundamentals

Only items after #18 are due to me, the rest are copied from the link above.

I subscribe to scientific materialism as a tentative metaphysics. Physical reality is what there is, and everything in the universe is a manifestation, however indirect, of the laws of physics. Ideas about human-centric phenomena (thought, emotion, choice, morality) being ontologically fundamental aren’t, in my view, worth taking seriously other than as interesting social or psychological phenomena.” It ought to go without saying that I still appreciate art and the humanities on their own terms as well as in naturalistic terms.
Many of the things we believe are not facts, they’re narratives — stories that are interpretations and generalizations of many facts. Narratives translate a strange, bewildering world into mentalese by using the building blocks of thought: causation, intention, agents, relationships, importance, good and evil, promises, rights and obligations.
The supposed facts that make up narratives can themselves be seen as sub-narratives, all in a fractal structure. The difference between a fact and a small narrative isn’t well defined. That’s because of how words work — they inherently contain different levels of generalization, interpretation and valuation. Thus, the structure and adoption of narratives shapes our collective worldviews and our societies, all the way up the chain from single words to full-blown paradigms. The struggle for the rights to shape them resembles war.
The study of rhetoric as usually defined is far too limited. We focus on rhetoric as trying to convince people to make a specific decision, but the constant reshaping of our shared conceptual world by the process of discourse is a more important arena for rhetoric.
The truth and accuracy of claims must take precedence over their social and political implications (in the public sphere, that is, because in personal situations tact is often more important). This is all for Kantian reasons: if we stop evaluating claims by their accuracy we can no longer trust them. Academic research that rejects this and sees itself as engaging in a political project (no matter how laudable) rather than a scientific one have severe legitimacy problems for this reason. Having a fundamentally political, strongly values-oriented way

...
Sleeping Beauty, Cloning Beauty, and Frequentist Arguments In Anthropic Paradoxes
by dadadarren — 2020-11-09T23:17:21.624Z

In a recent post, it has been said the Sleeping Beauty Problem is resoundingly considered resolved in academia. As someone who has been following the literature extensively for the past few years, I can positively say that is not true. The post also used a frequentist argument (in the form of a computer simulation) as support for thirdism. I want to point out the shortcoming of this argument. Most importantly, I want to show what I consider to be the correct frequentist model. And why perspectives shall be regarded as fundamental to solve anthropic paradoxes. as I have argued in this previous post.

Per Toss or Per Awakening

If the SB experiment is performed 100 times then roughly speaking there will be 150 awakenings. Out of these 150 awakenings, about 50 would be after Heads. Neither camp is contending this. A computer simulation reiterating it cannot settle the debate.

The problem is Halfers, in general, suggest the relative frequency should be divided over the total number of tosses: 50/100. Thirders on the other hand want to divide it over the total number of awakenings: 50/150. No consensus has been reached so far.

I argue the above interpretations are both wrong. I purpose an alternative model that would give the same relative frequency over toss or awakening. It requires treating perspectives as fundamental.

Cloning Beauty

For ease of expression, I will first use the following thought experiment I call Cloning Beauty. It presents the same anthropic uncertainty as SB. If you have doubt that SB and CB are equivalent, do not worry. I will go back to SB and present its frequentist model momentarily.

CB is set up the same way as the original Sleeping Beauty. Instead of memory wipes, the subjectively similar instances are obtained by cloning. Omega would toss a coin after Beauty falls asleep. If it lands Heads, nothing happens. If Tails, Omega would create a highly accurate clone of Beauty and put her in the empty identical room. The cloning is highly accurate even memories are preserved. So a Beauty would not be able to tell if she is physically new. Beauty wakes up the next day, what's the probability of Heads?

For clarity let's call the pre-existing agent the original, call the newly created agent the clone, and call any agent a copy.

The same debate of Halferism vs Thirderism also exists in CB. After waking up Beauty could say there is no new information so ...
The 300-year journey to the covid vaccine
by jasoncrawford — 2020-11-09T23:06:45.790Z
Link: https://rootsofprogress.org/immunization-from-inoculation-to-rna-vaccines

A covid vaccine has demonstrated 90% efficacy and no significant safety concerns in preliminary data from Phase 3 trials, according to an announcement today from Pfizer and BioNTech SE. The trials aren’t yet complete and the data hasn’t yet been released for independent verification, but this is very good news. (More from STAT News.)

Pfizer/BioNTech’s vaccine, like Moderna’s, is based on “mRNA” technology. If approved by the FDA, it will be the first such vaccine to reach that milestone. From a long-term progress perspective, this is a big deal.

Immunization technology has existed since the early 1700s (and the folk practices it originated in go back centuries further.) We can see the whole 300-year history of the technology as a quest to achieve immunity with ever-more safety and ever-fewer side effects. More recently, it has also become important to be able to react quickly to new epidemics, such as covid.

Here’s how immunization has advanced in stages:

Inoculation

All immunization is based on the observation that exposure to a disease often grants immunity (temporary if not permanent) to subsequent exposure. Long before we knew anything about antibodies or T-cells, people had noticed this simple correlation. Many people got smallpox in the past, but almost no one got it twice. The goal of immunization technology is to achieve that same immunity, but without having to suffer the disease or to risk death or other side effects.

The earliest form of immunization, then, was not a vaccine, but a method in which the patient was given the actual disease itself, in a manner that would cause a mild rather than a severe case of the illness. This was done with smallpox, and the technique was called inoculation or variolation.

This worked with smallpox for two reasons. One, infectious material was easy to obtain, from the pustules caused by the disease itself. Second, contracting the disease through a scratch on the skin caused a much more mild form than contracting it more naturally through inhalation.

Inoculation saved many people from smallpox. But there were downsides. First, the patient still had to contract the disease, causing mild symptoms. Second, there was still a small risk of a severe case; even the best inoculation methods had about a 0.2% death rate. Third, the patient was still contagious while going through the illness, and anyone who caught the disease natura...
Automating reasoning about the future at Ought
by jungofthewon — 2020-11-09T21:51:14.353Z
Link: https://ought.org/updates/2020-10-09-forecasting

Ought’s mission is to automate and scale open-ended reasoning. Since wrapping up factored evaluation experiments at the end of 2019, Ought has built Elicit to automate the open-ended reasoning involved in judgmental forecasting.

Today, Elicit helps forecasters build distributions, track beliefs over time, collaborate on forecasts, and get alerts when forecasts change. Over time, we hope Elicit will:

Support and absorb more of a forecaster’s thought process
Incrementally introduce automation into that process, and
Continuously incorporate the forecaster’s feedback to ensure that Elicit’s automated reasoning is aligned with how each person wants to think.

Our latest blog post introduces Elicit and our focus on judgmental forecasting. It also reifies the vision we’re running towards and potential ways to get there.
Notes on Self-Control
by David Gross — 2020-11-09T21:27:57.967Z

This post examines the virtue of self-control. I mostly summarize what others have learned about this virtue, rather than expressing my own opinions about it, though I’ve been selective about what I found interesting or credible, according to my inclinations. I wrote this not as an expert, but as someone who wants to learn more. I hope it will be helpful to people who want to know more about this virtue and how to nurture it.

What is self-control?

In another post, I cover the virtue of temperance, which involves having a well-chosen and well-regulated set of desires. Self-control is related but different. When an intemperate person gives into an unwise desire, they do so because they don’t see anything wrong with it; when a person without self-control does so, they do so in spite of knowing it’s wrong. Intemperance is a misuse of the will; lack of self-control is a malfunction of the will. Aristotle suggested this analogy: The intemperate person is like a city with bad laws; the person without self-control is like a city that has good laws on the books but that doesn’t enforce them.[1]

Some philosophers insist that lack of self-control is not really distinct from intemperance.[2] In their view, the person without self-control does not act contrary to their own temperate judgement, but merely reveals their real intemperate judgement at the last minute, having only deceived themselves into believing that they had good temperate judgement.

While temperance mostly has to do with our desires, self-control also comes into play in how we regulate our responses to other things, such as anger. (Lack of self-control in the face of fear falls under the domain of courage. Sometimes people split off response to anger under a distinct virtue like “good temper.”)

In addition to prompting unwise action, lack of self-control can also lead to unwise inaction, for example procrastination in which you commit to some wise course of action but then dawdle along doing something else instead, tempted by mere inertia or laziness.

Aside from lack of self-control, there is also a failure mode in which the pendulum swings too far the other way: pig-headed stubbornness, in which you stick with your resolutions even when the underlying facts change or when it turns out your resolutions were faulty. A fetish for being “decisive” can lead you to stick with bad decisions when a wiser person ...
Clarifying inner alignment terminology
by evhub — 2020-11-09T20:40:27.043Z

I have seen a lot of confusion recently surrounding exactly how outer and inner alignment should be defined and I want to try and provide my attempt at a clarification.

Here's my diagram of how I think the various concepts should fit together:

The idea of this diagram is that the arrows are implications—that is, for any problem in the diagram, if its direct subproblems are solved, then it should be solved as well (though not necessarily vice versa). Thus, we get:

inner alignment→objective robustnessouter alignment ∧objective robustness→intent alignmentintent alignment ∧capability robustness→alignment

And here are all my definitions of the relevant terms which I think produce those implications:

(Impact) Alignment: An agent is impact aligned (with humans) if it doesn't take actions that we would judge to be bad/problematic/dangerous/catastrophic.

Intent Alignment: An agent is intent aligned if the optimal policy for its behavioral objective[1] is impact aligned with humans.

Outer Alignment: An objective function r is outer aligned if all models that perform optimally on r in the limit of perfect training and infinite data are intent aligned.[2]

Robustness: An agent is robust if it performs well on the base objective it was trained under even in deployment/off-distribution.[3]

Objective Robustness: An agent is objective robust if the optimal policy for its behavioral objective is impact aligned with the base objective it was trained under.

Capability Robustness: An agent is capability robust if it performs well on its behavioral objective even in deployment/off-distribution.

Inner Alignment: A mesa-optimizer is inner aligned if the optimal policy for its mesa-objective is impact aligned with the base objective it was trained under.

And an explanation of each of the diagram's implications:

inner alignment→objective robustness: If a model is a mesa-optimizer, then its behavioral objective should match its mesa-objective, which means if it's mesa-objective is aligned with the base, then it's behavioral objective should be too.

outer alignment ∧objective robustness→intent alignment: Outer alignment ensures that the base objective is measuring what we actually care about and objective robustness ensures that the model's behavioral objective is aligned with that base objective. Thus, putting them together, we get that the model's behavioral objective must be al...
Open Problems Create Paradigms
by johnswentworth — 2020-11-09T20:04:34.534Z

A few weeks ago I had a fascinating conversation with Ruby about models of the research process and how to improve it. This post outlines one particular model which I took away from that conversation: open problems as the primary factor which create a paradigm.

There’s a cluster of things like research agendas, open problems, project proposals, and/or challenges; we’ll refer to that whole cluster as “open problems”. The unifying theme here is the function(s) these things serve:

Define a problem
Provide context: why is the problem interesting/valuable? Why is it hard? What would a solution look like? What background work exists?
Provide starting points/footholds/surface area - places for people to start working on the problem
Create a status reward for solving the problem - mainly by making the importance and difficulty public knowledge

Let’s walk through each of those pieces.

First, an open problem defines a problem. That sounds obvious, but it’s more difficult than it sounds: defining a problem means setting it up in such a way that anyone who understands the problem-explanation can recognize a solution. For pre-paradigmatic fields, this is hard. What would a solution to e.g. an embedded agency problem look like? If someone proposed a “solution”, and we asked 50 different researchers whether this was actually a solution to the problem, would their answers all agree? Probably not, though the embedded agency sequence brought us a lot closer to that point.

Second, an open problem provides context. Why is the problem interesting/valuable? Why is it hard? This goes hand-in-hand with defining the problem: we define the problem in a particular way because we expect a solution to provide some value. If a solution to a differently-defined problem would not clearly provide the same value, then that’s an argument in support of our particular problem definition.

Third, an open problem provides starting points/footholds/surface area for people to tackle the problem. The problem definition and value-prop inevitably ties to background work and existing techniques, and explains why those techniques are not already sufficient. That provides a jumping-off point for newcomers.

Finally, establishment of an open problem creates a status reward for solving the problem. We define what a solution looks like, so others can easily recognize success. We explain why a solution would be valuable...
Committing, Assuming, Externalizing, and Internalizing
by Scott Garrabrant — 2020-11-09T16:59:01.525Z

This is the tenth post in the Cartesian frames sequence.

Here, we define a bunch of ways to construct new (additive/multiplicative) (sub/super)-(agents/environments) from a given Cartesian frame. Throughout this post, we will start with a single Cartesian frame over W, C=(A,E,⋅).

We will start by defining operations from taking subsets and partitions of A and E. We will then define similar operations from taking subsets and partitions of W.

1. Definitions from Agents and Environments

1.1. Committing

Definition: Given a subset B⊆A, let CommitB(C) denote the Cartesian frame (B,E,⋆), with ⋆ given by b⋆e=b⋅e. Let Commit∖B(C) denote the Cartesian frame (A∖B,E,⋄), with ⋄ given by a⋄e=a⋅e.

CommitB(C) represents the perspective you get when the agent of C makes a commitment to choose an element of B, while Commit∖B(C) represents the perspective you get when the agent of C makes a commitment to choose an element outside of B.

Claim: For all B⊆A, CommitB(C)◃+C and Commit∖B(C)◃+C. Further, Commit∖B(C) and CommitB(C) are brothers in C.

Proof: That CommitB(C)◃+C and Commit∖B(C)◃+C is trivial from the committing definition of additive subagent.

Observe that CommitB(C)⊕Commit∖B(C)=(A,E×E,∙), where ∙ is given by a∙(e,f)=a⋅e if a∈B, and a∙(e,f)=a⋅f if a∉B. Let D⊂E×E be the diagonal, {(e,e) | e∈E}. We clearly have that (A,D,∙′) is in CommitB(C)⊞Commit∖B(C), where ∙′ is the restriction of ∙ to A×D, and that (A,D,∙′)≅C; so Commit∖B(C)◃+C and CommitB(C)◃+C are brothers in C. □

Claim: Commit∖B(C)≅CommitA∖B(C)

Proof: Trivial. □

1.2. Assuming

Assuming is the dual operation to committing.

Definition: Given a subset F⊆E, let AssumeF(C) denote the Cartesian frame (A,F,⋆), with ⋆ given by a⋆f=a⋅f. Let Assume∖F(C) denote the Cartesian frame (A,E∖F,⋄), with ⋄ given by a⋄e=a⋅e.

AssumeF(C) represents the perspective you get when you assume the environment is chosen from F, while Assume∖F(C) represents the perspective you get when you assume the e...
Time Travel Markets for Intellectual Accounting
by abramdemski — 2020-11-09T16:58:44.276Z

In a comment to Can we hold intellectuals to similar public standards as athletes? I said that one of the largest problems with using prediction accuracy (EG prediction markets) as the exclusive standard by which we judge intellectuals would be undervaluing contributing to the thought process.

Here, I propose a modification of prediction markets which values more types of contributions.

Arguments as Models

In many cases, a thought/argument can be reformulated as a model. What I mean by this is a formally described way of making predictions.

Adding models to prediction markets could increase their transparency; we want to create an incentive for traders to explain themselves.

A model has a complexity (description length; the negative log of its prior probability). We generally have more respect for predictions which come from models, in contrast to those which don't. And, given two models, we generally have more respect for the simpler one.

So, to adjust prediction markets to account for models, we would want something which:

Values making good predictions over everything else;
Values models over opaque predictions;
Values simple models more.

But how can we trade off between modelling and predictive accuracy?

Time Travel Markets

As I've discussed before, one of the major differences between prediction markets and Bayesianism is that prediction-market traders only get credit for moving the market in the right direction, which introduces a dependence on when a prediction is made: if you make an accurate prediction at a time when everyone else has already arrived at that conclusion, you don't get any credit; whereas, if you make an accurate prediction at a time when it's unpopular, you'll get a lot of credit for that.

That's a problem for model-makers. An explanation is still useful, even after all the facts needing explanation are agreed upon.

This is a generalization of the problem of old evidence.

Solomonoff induction solves the problem of old evidence for Bayesians as follows: a new hypothesis which explains old evidence is evaluated as if it were in our set of hypotheses from the beginning. (After all, it would have been in the Solomonoff prior from the beginning; it was only absent from our calculations because we are forced to approximate Solomonoff.) The prior probability is based on the complexity of the hypothesis; the updates are whatever they would have been....
Building AGI Using Language Models
by leogao — 2020-11-09T16:33:25.864Z
Link: https://leogao.dev/2020/08/09/Building-AGI-Using-Language-Models/

Despite the buzz around GPT-3, it is, in and of itself, not AGI. In many ways, this makes it similar to AlphaGo or Deep Blue; while approaching human abiliy in one domain (playing Chess/Go, or writing really impressively), it doesn’t really seem like it will do Scary AGI Things™ any more than AlphaGo is going to be turning the Earth into paperclips anytime soon. While its writings are impressive at emulating humans, GPT-3 (or any potential future GPT-x) has no memory of past interactions, nor is it able to follow goals or maximize utility. However, language modelling has one crucial difference from Chess or Go or image classification. Natural language essentially encodes information about the world—the entire world, not just the world of the Goban, in a much more expressive way than any other modality ever could. By harnessing the world model embedded in the language model, it may be possible to build a proto-​AGI.
Incentive Problems With Current Forecasting Competitions.
by NunoSempere — 2020-11-09T16:20:06.394Z

This post outlines some incentive problems with forecasting tournaments, such as Good Judgement Open, CSET, or Metaculus. These incentive problems may be problematic not only because unhinged actors might exploit them, but also because of mechanisms such as those outlined in

Unconscious Economics

. For a similar post about PredictIt, a prediction market in the US, see

here

. This post was written in collaboration with alexrjl, who should be added as a coauthor soon.

Problems

Discrete prizes distort forecasts

If a forecasting tournament offers a prize to the top X forecasters, the objective "be in the top X forecasters" differs from "maximize predictive accuracy". The effects of this are greater the smaller the number of questions.

For example if only the top forecaster wins a prize, you might want to predict a surprising scenario, because if it happens you will reap the reward, whereas if the most likely scenario happens, everyone else will have predicted it too.

Consider for example

this prediction contest

, which only had a prize for #1. The following question asks about the margin Biden would win or lose Georgia by:

Then the most likely scenario might be a close race, but the prediction which would have maximized your odds of coming in #1st might be much more extremized, because other predictors are more tightly packed in the middle.

This effect also applies if you think of "becoming a superforecaster", which you can do if you are in the top 2% and have over 100 predictions in Good Judgement Open, as a distinct objective from "maximizing reward accuracy".

Incentives not to share information and to produce corrupt information

In a forecasting tournament, there is a disincentive to sharing information, because other forecasters can use it to improve relative standing. This not only includes not sharing information but also providing misleading information. As a counterpoint, other forecasters can and will often point out flaws in your reasoning if you're wrong.

Incentives to selectively pick questions.

If one is maximizing the Brier score, one is incentivized to pick easier questions. Specifically, if someone has a brier score b2, then they should not make a prediction on any question where the probability is between b and (1-b), even if they know the true probability exactly. Tetlock explicitly mentions this in one of his

Commandments for superforecasters

: “Focus ...