trentmkelly/LessWrong-43k huggingface dataset:
Big Community Solstice
by jefftk — 2019-12-18T02:30:01.857Z

In 2011, Ray

started
a tradition in the broader rationalist community of having a
gathering around the winter solstice. As the person who started the
thing, if he says we should do it differently I'm going to pay
attention. But on the other hand I disagree with this pretty strongly:

"Big Solstice" is not Solstice.

The NYC Solstice Celebration is not Solstice.

The Bay Solstice Celebration is not Solstice

(according to me).

Big NYC Solstice was an _advertisement_ I created for Actual Solstice.

Actual Solstice is held on December 21st, or whenever Solstice is this
year, with your close family and friends. It is a holiday. It is
something you have a lot of ownership of.

—FB
post on the direction of Solstice

Reading the whole post, Ray describes Big Community Solstice and
Little Family Solstice, and why he thinks the latter should be the
focus. I'm atheist, in a family that's mixed, but that has a strong
internal tradition of a family Christmas celebration. Ray has talked
about how this kind of family tradition is what got him wanting to
make a Solstice holiday, and I see where he's coming from. But I
think Big Community Solstice fills a much more important role than
Small Family Solstice.

The former fills the role of church, and specifically the kind of
meaningful and serious church experience you have when people take
their religion seriously and honestly believe. Mainstream versions of
this aren't open to or attractive to atheists because they're built
around religion. They also don't emphasize the most important (but
challenging!) parts of the religion, let alone the ideas I think
are most important in general. Big Community Solstice can fill a
large need here, and that's why I host one.

The latter, however, fills the role of family Christmas celebrations.
These are much less dependent on religion than you might think. Yes,
there's Christian imagery and the best songs are pretty seriously
religious, but since it's primarily about family and tradition that's
ok. Christmas has been secular in parts of my family for a long time:
on the Jewish immigrant side it was taken up as an American
holiday. On the Quaker side was religious, but it has translated well
from the lightly-religious generation of my parents to my,
largely-atheist, generation.

Go back far enough and you see many of the traditions of Christmas,
bu...
On the role of abstraction in mathematics and the natural sciences
by Faustus2 — 2019-12-18T00:04:05.484Z

I'm a mathematics undergraduate in the UK, in my final year, and I have been thinking a lot about Platonism. I largely find myself completely unconvinced by attempts to naturalise the metaphysics behind the ontology of mathematical objects, though I am not unsympathetic to those attempts, and of those attempts I know, I can see their point.

I will state plainly here that as of this post I am a Platonist (by which, I believe that mathematical objects are abstract objects, but obviously that I am unsure of what this entails), and i want to sort of articulate why. I will mostly be focusing on the notion of 'mathematics is just a useful tool/language, and it has no abstract independent existence outside of this viewpoint'.

Most of the posts I've seen on lesswrong talk about natural numbers as the object of our attention when we discuss mathematics, though i could have missed some. To be sure, the natural numbers are about as quintessentially 'MATHEMATICS' as you can get, but that's not what i want to really discuss. I want to talk about abstractions. Hilbert space, Banach space, Measure space, Galois symmetries, Categories, Topoi, etc. are just some examples of extremely abstract notions that offer powerful and, from my perspective, necessary insight into applied mathematics.

Something that has almost plagued me over this final year is the notion of how the abstract notions of modern day mathematics has applicability to the real world. It is not that I am saying 'mathematics fits the world so well, we are so lucky to be able to have such a versatile tool'. But i am saying 'Why should it be the case that an abstract notion even has such powerful applicability at all? To the point where you can derive iff statements about applied structures.' I don't buy that we make mathematics fit the world, per se, although we certainly do that insofar as we are trying to model something. But its more, when we uncover structures built into the general areas of applied math, the mathematics says that 'if we assume such and such holds about, say partial differential equations, then this implies, through a chain of abstract reasoning, we get something like a concrete result within the real world.' Its essentially this process that disturbs me. This process can fail in the real world, especially when we try to mo...
Is Causality in the Map or the Territory?
by johnswentworth — 2019-12-17T23:19:24.301Z

steve2152 brought up a great example:

Consider a 1kΩ resistor, in two circuits. The first circuit is the resistor attached to a 1V [voltage] supply. Here an engineer would say: "The supply creates a 1V drop across the resistor; and that voltage drop causes a 1mA current to flow through the resistor." The second circuit is the resistor attached to a 1mA current source. Here an engineer would say: "The current source pushes a 1mA current through the resistor; and that current causes a 1V drop across the resistor." Well, it's the same resistor ... does a voltage across a resistor cause a current, or does a current through a resistor cause a voltage, or both, or neither? [...] my conclusion was that people think about causality in a way that is not rooted in physics, and indeed if you forced someone to exclusively use physics-based causal models, you would be handicapping them.

First things first: we're talking about causality, which means we're mainly talking about counterfactuals - questions of the form "what would the system do if we did X?". (See Pearl's book for lots of detail on how and why causality and counterfactuals go together.)

In the resistor example, both scenarios yield exactly the same actual behavior (assuming we've set the parameters appropriately), but the counterfactual behavior differs - and that's exactly what defines a causal model. In this case, the counterfactuals are things like "what if we inserted a different resistor?" and "what if we adjusted the knob on the supply?". If it's a voltage supply, then a voltage -> current model ("voltage causes current") correctly answers the counterfactuals:

Inserting a different resistor changes the current but not the voltage. In the voltage -> current model, we cut the arrow going into "current" and set that node to a new value.
Adjusting the knob on the supply changes the voltage, and the current adjusts to match. In the voltage -> current model, we set the "voltage" node to a new value, and the model tells us how to update the "current" node.

Conversely, if it's a current supply, then a current -> voltage model ("current causes voltage") correctly answers the counterfactuals. It is a mistake here to think of "the territory" as just the r...
Counterfactual Mugging: Why should you pay?
by Chris_Leong — 2019-12-17T22:16:37.859Z

The LessWrong Wiki defines Counterfactual Mugging as follows:

Omega

appears and says that it has just tossed a fair coin, and given that the coin came up tails, it decided to ask you to give it $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don't want to give up your $100. But Omega also tells you that if the coin came up heads instead of tails, it'd give you $10000, but only if you'd agree to give it $100 if the coin came up tails. Do you give Omega $100?

I expect that most people would say that you should pay because a 50% chance of $10000 for $100 is an amazing deal according to expected value. I lean this way too, but it is harder to justify than you might think.

After all, if you are being asked for $100, you know that the coin came up heads and you won't receive the $10000. Sure this means that if the coin would have been heads then you wouldn't have gained the $10000, but you know the coin wasn't heads so you don't lose anything. It's important to emphasise: this doesn't deny that if the coin had come up heads that this would have made you miss out on $10000. Instead, it claims that this point is irrelevant, so merely repeating the point again isn't a valid counter-argument.

You could argue that you would have pre-commited to paying if you had known about the situation ahead of time. True, but you didn't pre-commit and you didn't know about it ahead of time, so the burden is on you to justify why you should act as though you did. In Newcomb's problem you want to have pre-committed and if you act as though you were pre-committed then you will find that you actually were pre-committed. However, here it is the opposite. Upon discovering that the coin came up tails, you want to act as though you were not pre-commited to pay and if you act that way, you will find that you actually were indeed not pre-commited.

We could even channel Yudkowsky from Newcomb's Problem and Regret of Rationality: "Rational agents should WIN... It is precisely the notion that Nature does not care about our algorithm, which frees us up to pursue the winning Way - without attachment to any particular ritual of cognition, apart from our belief that it wins. Every rule is up for grabs, except the rule of winning... Unreasonable? I am a...
A dilemma for prosaic AI alignment
by Daniel Kokotajlo — 2019-12-17T22:11:02.316Z

Epistemic status: I predict that people who focus on

prosaic AI alignment

have thought of this before, in some way at least. But I don’t know what they would say in response, so I’m writing this up so I can find out! I’m making it a post instead of an email so that the discussion can be public.

Characterization of prosaic AI alignment: Prosaic AI methods—the sort of methods that we are using today, rather than hypothetical new methods based on a deeper understanding of intelligence—might be sufficient to make human-level AGI in the next two decades or so, and that if this happens we’d better be prepared. Thus we should think about how to take prosaic AI methods and combine them or modify them in various ways to make something which is as competitive, or almost as competitive, as cutting-edge AI. Examples of this approach are

debate

,

imitating humans

,

preference learning

, and

iterated distillation and amplification

.

Conjecture: Cutting-edge AI will come from cutting-edge algorithms/architectures trained towards cutting-edge objectives (incl. unsupervised learning) in cutting-edge environments/datasets. Anything missing one or more of these components will suffer a major competitiveness penalty.

Example: Suppose that the best way we know of to get general intelligence is to evolve a population of giant neural nets with model-free learning in a part-competitive, part-cooperative, very diverse environment consisting of an ensemble of video games. One year, the systems that come out of this process are at dog level, then two years later they are at chimpanzee level, then two years later they are at IQ-80 human level… It is expected that scaling up this sort of thing will lead, in the next few years, to smarter-than-human AGI.

The Dilemma: Choose plan 1 or plan 2:

Plan 1: Train a system into your scheme from scratch, using cutting-edge algorithms but not cutting-edge environments or objectives. (The environments and objectives are whatever your safety scheme calls for, e.g. debates, imitating humans, a series of moral choice situations, etc.)

Example: We take the best training algorithms and architecture we can find, but instead of training on an ensemble of video games, our AI is trained from scratch to win debates with human judges. We then have it debate on important topics to give us valuable information.
Problem with

...
Sections 3 & 4: Credibility, Peaceful Bargaining Mechanisms 
by JesseClifton — 2019-12-17T21:46:49.216Z

This post is part of the sequence version of the Effective Altruism Foundation's research agenda on Cooperation, Conflict, and Transformative Artificial Intelligence.

3 Credibility

Credibility is a central issue in strategic interaction. By credibility, we refer to the issue of whether one agent has reason to believe that another will do what they say they will do. Credibility (or lack thereof) plays a crucial role in the efficacy of contracts (Fehr et al., 1997; Bohnet et al., 2001), negotiated settlements for avoiding destructiveconflict (Powell, 2006), and commitments to carry out (or refuse to give in to) threats (e.g., Kilgour and Zagare 1991; Konrad and Skaperdas 1997).

In game theory, the fact that Nash equilibria (Section 1.1) sometimes involve non-credible threats motivates a refined solution concept called subgame perfect equilibrium (SPE). An SPE is a Nash equilibrium of an extensive-form game in which a Nash equilibrium is also played at each subgame. In the threat game depicted in Figure 1, “carry out” is not played in a SPE, because the threatener has no reason to carry out the threat once the threatened party has refused to give in; that is, “carry out’’ isn’t a Nash equilibrium of the subgame played after the threatened party refuses to give in.

So in an SPE-based analysis of one-shot threat situations between rational agents,
threats are never carried out because they are not credible (i.e., they violate subgame perfection).

However, agents may establish credibility in the case of repeated interactions
by repeatedly making good on their claims (Sobel, 1985). Secondly, despite the fact that carrying out a threat in the one-shot threat game violates subgame perfection, it is a well-known result from behavioral game theory that humans typically refuse unfair splits in the Ultimatum Game [1] (Güth et al., 1982; Henrich et al., 2006), which is equivalent to carrying out the threat in the one-shot threat game. So executing commitments which are irrational (by the SPE criterion) may still be a feature of human-in-the-loop
systems (Section 6), or perhaps systems which have some humanlike game-theoretic heuristics in virtue of being trained in multi-agent environments (Section 5.2). Lastly, threats may become credible if the threatener has credibly committed to carrying out the threat (in the case of the game in Fig. 1, this means convincing the opponen

...
Sections 1 & 2: Introduction, Strategy and Governance
by JesseClifton — 2019-12-17T21:27:30.496Z

This post is part of the sequence version of the Effective Altruism Foundation's research agenda on Cooperation, Conflict, and Transformative Artificial Intelligence.

1 Introduction

Transformative artificial intelligence (TAI) may be a key factor in the long-run trajectory of civilization. A growing interdisciplinary community has begun to study how the development of TAI can be made safe and beneficial to sentient life (Bostrom 2014; Russell et al., 2015; OpenAI, 2018; Ortega and Maini, 2018; Dafoe, 2018). We present a research agenda for advancing a critical component of this effort: preventing catastrophic failures of cooperation among TAI systems. By cooperation failures we refer to a broad class of potentially-catastrophic inefficiencies in interactions among TAI-enabled actors. These include destructive conflict; coercion; and social dilemmas (Kollock, 1998; Macy and Flache, 2002) which destroy value over extended periods of time. We introduce cooperation failures at greater length in Section 1.1.

Karnofsky (2016) defines TAI as ''AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution''. Such systems range from the unified, agent-like systems which are the focus of, e.g., Yudkowsky (2013) and Bostrom (2014), to the "comprehensive AI services’’ envisioned by Drexler (2019), in which humans are assisted by an array of powerful domain-specific AI tools. In our view, the potential consequences of such technology are enough to motivate research into mitigating risks today, despite considerable uncertainty about the timeline to TAI (Grace et al., 2018) and nature of TAI development. Given these uncertainties, we will often discuss "cooperation failures" in fairly abstract terms and focus on questions relevant to a wide range of potential modes of interaction between AI systems. Much of our discussion will pertain to powerful agent-like systems, with general capabilities and expansive goals. But whereas the scenarios that concern much of the existing long-term-focused AI safety research involve agent-like systems, an important feature of catastrophic cooperation failures is that they may also occur among human actors assisted by narrow-but-powerful AI tools.

Cooperation has long been studied in many fields: political theory, economics, game theory, psychology, evolutionary biology, multi-agent system...
Counterfactual Induction (Lemma 4)
by Diffractor — 2019-12-17T05:05:15.959Z

Lemma 4:

∀V∈Λ:∃V∈Ξ′:t(V)A(ϕ)≤VA(ϕ)

Notation:

For notational convenience, we will abuse notation a bit and abbreviate f(g(A)) as A. This is just turning A from a finite collection of axioms to a sentence, and from there to the set of worlds where that sentence is true. And if E∈L, we will abbreviate E/P(S) as ¬E. E,F denote arbitrary elements of L, and by our previous abuse of notation, A is an element of A when it appears as a subscript, and L otherwise, via the translation from groups of sentences to sets of worlds.

Our axioms of interest are:

1: Unitarity: ∀A:VA(A)=1

2: Subadditivity: ∀A,E,F:VA(E∪F)≤VA(E)+VA(F)

3: Empty Set: ∀A:VA(∅)=0

4: Law of Excluded Middle: ∀A,E:VA(E)+VA(¬E)=1

5: Sane Conditioning: ∀A,E:VA(E)=VA(A∩E)

6: Monotonicity: ∀A,E,F:A∩E⊆A∩F→VA(E)≤VA(F)

Notice that 6 implies 5. 6 is the powerset analogue of propositional monotonicity (where the precondition is A⊢pcϕ→ψ), and 5 is the powerset analogue of propositional equivalence (where the precondition is A⊢pcϕ↔ψ). Also note that 1,4, and 5 imply 3 because VA(∅)+VA(P(S))=1 by LEM, and VA(P(S))=VA(A)=1 by sane conditioning and unitarity.

Ξ′ is the set of all valuations that fulfill properties 1,2,3,4, and 6 (this also gets 5) Θ′ is the set of all valuations that fulfill properties 1,2, and 3, while Θ′lem,Θ′cond,Θ′mono are the subsets of Θ′ that fulfill properties 4, 5, and 6 respectively. Ξ′=Θ′lem∩Θ′mono.

Also, in some of the sublemmas, we will be working heavily with the powerset lattice L. It is the lattice of all subsets of worlds, ordered by set inclusion. Sup in this lattice corresponds to union, inf corresponds to intersection, and given some set A, A↓ refers to the ideal of all subsets of A, and A↑ refers to the filter of all supersets of A. Thinking of this lattice may help give some geometric intuition to the proofs for the reader, and we will freely move back and forth between the subset view and the lattice view.

The subvaluation property is defined as follows: V′≤V↔∀A,E:V′A(...
Counterfactual Induction (Algorithm Sketch, Fixpoint proof)
by Diffractor — 2019-12-17T05:04:25.054Z

So, to begin with, here's how the algorithm works.

The upstream algorithm iterates through all proofs, and records the lengths of all proofs of the form "a finite collection of sentences implies ⊥". Also the proof length accounting is set up such that if A,ϕ⊢L⊥ and A,ψ⊢L⊥, then A,ϕ∨ψ⊢L+L′⊥. Also, as soon as it checks all proofs of length n or shorter from A+propositional tautology given A, with no contradictions found, it reports that A is contradiction-free for at least n steps. (this doesn't require searching an infinite set because the number of propositional tautologies with length shorter than n is finite)

With S being the set of all math sentences, and Pfin,pc(S) being the set of all finite subsets of sentences that are propositionally consistent, the market P is a partial of type Pfin,pc(S)×S→[0,1], which fulfills the following four axioms.

(note: A appearing where a sentence would normally go refers to the collection of statements in A expressed as one big boolean and-statement, and ϕ⊢pcψ means that given ϕ, ψ is provable using only the rules of inference for propositional calculus)

1: Unitarity. ∀A:PA(A)=1

2: Subadditivity. ∀A,ϕ,ψ:PA(ϕ)+PA(ψ)≥PA(ϕ∨ψ)

3: Law of Excluded Middle. ∀A,ϕ:PA(ϕ)+PA(¬ϕ)=1

4: Propositional Monotonicity: ∀A,ϕ,ψ:A,ϕ⊢pcψ→PA(ψ)≥PA(ϕ)

We will also consider a fifth axiom, propositional equivalence, which is implied by axiom 4.

5: Propositional Equivalence: ∀A,ϕ,ψ:A⊢pcϕ↔ψ→PA(ψ)=PA(ϕ)

The market fulfills axioms 1-4, the worlds we are defending against fulfill axioms 1-3 and 5. Axioms 1, 3, and 5 suffice to show the empty set property that PA(⊥)=0, so that one comes for free and doesn't need to be specified.

Traders are poly-time algorithms that output a continuous circuit that takes as input the pricing P and output a nonnegative number for each pair of the form (A,ϕ). This should be interpreted as a bet against ϕ in the A counterfactual, and has a payoff of PA(ϕ)−VA(ϕ). Due to law of excluded middle for both V and P, selling these...
Counterfactual Induction
by Diffractor — 2019-12-17T05:03:32.401Z

One of the first attempts at defining counterfactuals was the conjecture that ϕ is a counterfactual consequence of ψ when there is a short proof of ψ→ϕ.

This post will present a variant of logical induction that implements these proof-length-based counterfactuals, so there are well-defined answers about what happens in impossible worlds, based on proof length. It required generalizing the notion of a probability distribution, so these generalized probability distributions, which I'll call "valuations", may be a subject of independent interest.

Now, to begin with, why would we want to use proof-length counterfactuals, instead of the existing notion of conditional probability? Well, the first issue is with provability induction. Due to the ability of logical inductors to learn that sequences of statements are false, and start preemptively assigning the statements extremely low probability even though the length of a disproof may rise greatly, it becomes extremely cheap for traders to mess with conditional probabilities, and as stated in the logical induction paper, we don't have good results for what happens to conditional probabilities on low-probability events. The second issue is that humans don't seem to think about what-ifs like "what's the probability of a decrease in the price of pencils along with a supply shock to lumber in Estonia, now what's the probability of the supply shock in Estonia, let me divide...". Humans instead go "in the imaginary world where there's a supply shock to lumber in Estonia, what happens to pencil prices". We seem to take conditional probabilities as primitive and compute the probability of conjunctions from those.

A Note On Booleans:

So, the original logical inductor algorithm just assigns values to statements as strings, while oracle induction and universal inductors operate by assigning an actual probability distribution to worlds (ie, functions from the set of atomic statements to {0,1}, with the probability of booleans given by the summed probability over all worlds that fulfill that boolean.)

The original logical inductor algorithm snuck the worlds in through the back-door via the worst-case scoring, where you consider only worlds that are propositionally consistent (no disproofs are discoverable by boolean algebra alone), thus forcing... ,