trentmkelly/LessWrong-43k huggingface dataset:
Proofs Section 2.1 (Theorem 1, Lemmas)
by Diffractor — 2020-08-27T07:54:59.744Z

Fair upfront warning: This is not a particularly readable proof section. There's a bunch of dense notation, logical leaps due to illusion of transparency since I've spent months getting fluent with these concepts, and a relative lack of editing since it's long. If you really want to read this, I'd suggest PM-ing me to get a link to MIRIxDiscord, where I'd be able to guide you through it and answer questions. This post will be recapping the notions and building up an arsenal of lemmas, the next one will show the isomorphism theorem, translation theorems, and behavior of mixing, and the last one is about updates and the decision-theory results. It's advised to have them open in different tabs and go between them as needed.

With that said, let's establish some notation and mental intuition. I'll err on the side of including more stuff because illusion of transparency. First, visualize the tree of alternating actions and observations in an environment. A full policy π can be viewed as that tree with some branches pruned off, specifying every history that's possible with your policy of interest. All our policies are deterministic. A policy stub πst is a policy tree that's been mostly pruned down (doesn't extend further than some finite time n). A partial policy πpa is just any policy tree in any state of specification or lack thereof, from tiny stubs to full policies to trees that are infinite down some branches but not others.

π∅ denotes the empty policy (a stub) which specifies nothing about what a policy does, and π¬h is some partial policy which specifies everything (acts like a full policy) everywhere except on history h and afterwards.

There's a distance metric on histories, as well as a distance metric on partial policies. Both of them are of the form γt where γ<1, and t is the "time of first difference". For histories, it's "what's the first time these histories differ", for policies, it's "what's the shortest time by which one partial policy is defined and the other is undefined, or where the policies differ on what to do". So, thinking of the distance as getting smaller as the time of first difference gets bigger is a reliable guide.
The outcome set F(πpa) is... take the tree corresponding to πpa, and it's the set of all the observation leaf nodes an...
Proofs Section 2.2 (Isomorphism to Expectations)
by Diffractor — 2020-08-27T07:52:08.121Z

Previous proof post is here.

Theorem 2: Isomorphism Theorem: For (causal, pseudocausal, acausal, surcausal) Θst or Θω which fulfill finitary or infinitary analogues of all the defining conditions, ↑(Θst) and ↓(Θω) are (causal, pseudocausal, acausal, surcausal) hypotheses. Also, ↑ and →st define an isomorphism between Θ and Θst, and ↓ and →ω define an isomorphism between Θ and Θω.

Proof sketch: The reason this proof is so horrendously long is that we've got almost a dozen conditions to verify, and some of them are quite nontrivial to show and will require sub-proof-sketches of their own! Our first order of business is verifying all the conditions for a full belief function for ↑(Θst). Then, we have to do it all over again for ↓(Θω). That comprises the bulk of the proof. Then, we have to show that taking a full belief function Θ and restricting it to the infinite/finite levels fulfills the infinite/finite analogues of all the defining conditions for a belief function on policies or policy-stubs, which isn't quite as bad. Once we're done with all the legwork showing we can derive all the conditions from each other, showing the actual isomorphism is pretty immediate from the Consistency condition of a belief function.

Part 1:Let's consider ↑(Θπst). This is defined as: ↑(Θst)(πpa):=⋂πst≤πpa(prπpa,πst∗)−1(Θst(πst))

We'll show that all 9+2 defining conditions for a belief function are fulfilled for ↑(Θst). The analogue of the 9+2 conditions for a Θst is:

1: Stub Nirvana-free Nonemptiness: ∀πst:Θst(πst)∩NF≠∅

2: Stub Closure: ∀πst:Θst(πst)=¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯Θst(πst)

3: Stub Convexity. ∀πst:Θst(πst)=c.h(Θst(πst))

4: Stub Nirvana-free Upper-Completion.

∀πst:Θst(πst)∩NF=((Θst(πst)∩NF)+Msa(FNF(πst)))∩Ma(F(πst))

5: Stub Restricted Minimals:

∃λ⊙,b⊙∀πst:(λμ,b)∈(Θst(πst))min→λ+b≤λ⊙+b⊙

6: Stub Normalization: infπstEΘst(πst)(0)=0 and supπstEΘst(πst)(1)=1

7: Weak Consistency: ∀πlost,πhist≥πlost:prπhist,πlost∗(Θst(πhist))⊆Θst(πlost)

8: Stub Extreme Point Condition: for all M,πlost:

M∈(Θst(πlost))xmin∩NF→

∃π≥πlost∀πhist:(π>πhist≥πlost→(∃M′∈Θst(πhist)∩NF:prπhist,πlost∗(M′)=M))

9: Stub Uniform C...
Proofs Section 2.3 (Updates, Decision Theory)
by Diffractor — 2020-08-27T07:49:05.047Z

Here are the previous two posts.

Now, what about updates? We'll use ugh (and suppress the π¬h that should be there) as shorthand for the function that maps (m,b) over Θ(π¬h∙πpa) to (c(m|h),b+m(0★hg)) in Ma(F(πpa)) (or the nirvana-free or sur variant of this), and also use ugh as a function from belief functions to belief functions (just map all the sets through)

Lemma 27: When updating, the closure adds no nirvana-free points that weren't present originally if Nonemptiness, Nirvana-free upper-completion and closure holds originally (works in the sur-case too)

Proof sketch: We take a sequence Mn limiting to M, and then take a preimage point of Mn, go to a minimal below it, find a limit point in our original set by Compactness, and map it back through the update, getting a point below M. Then, we find what we need to add to that to get M, and find something above our limit point that maps to M, so we didn't actually need closure anyways because we made M as an image of a nirvana-free point present in the original set.

Proof: Fix a sequence Mn in ugh(Θ)(πpa) (but without the defining closure part in the end) that limit to M which is nirvana-free.

Every Mn has a preimage point M′n∈Θ(π¬h∙πpa) with no nirvana off-h. For each M′n, find a minimal point M′lon below it, which have a λ⊙+b⊙ bound by bounded-minimals, so we can find a convergent subsequence limiting to M′lo (actually, might not be minimal, still a limit of minimal points, though). Shoving the M′lon (and the limit point) back through the update (which is a continuous function), we get a sequence Mlon limiting to Mlo (the thing you get from pushing M′lo through the update).

Since M′n lies above M′lon (upper-completion ordering), then updating preserves that property, because the update function is linear. Thus, all the Mlon lie below their corresponding Mn. Now, we can invoke Lemma 16 to conclude that Mlo lies below M. It lies below a nirvana-free point, so M′lo is nirvana-free as well. Now, we just need to show nirvana-free upper-completion because M=Mlo+M∗.

We can take M′lo and add on (m∗,b∗...
19th-century progress studies
by jasoncrawford — 2020-08-27T02:28:29.765Z
Link: https://rootsofprogress.org/19th-century-progress-studies

I wrote earlier about an 1857 plan for the transcontinental railroad (as yet unbuilt), and how it advocated funding the project through the sale of stock to a broad base of public investors. However, another aspect of the plan was even more striking: it explicitly advocating studying the history of progress, in order to judge what was possible and desirable for the railroad. It was like a 19th-century form of progress studies.

The author of the plan, Theodore Judah, first addresses the question of how fast the trains will travel from St. Louis to San Francisco: three days, he estimates. But then he points out that this estimate is only based on the speeds attainable by “our present class of engines.” The next paragraphs exemplify the 19th-century spirit of progress:

But—

Is there to be no improvement in our present class of engines? have we reached a point in the stage of progress where we must stop, beyond which we cannot go? Are we willing to admit that fifty miles per hour is the limit to speed? Are we contented, and do we desire to go no faster?

No—

However well we may be satisfied with the present rate of speed in traveling, we dare not admit the principle—we wish to go as fast as we can. Improvements are progressive and the future is before us. No, we have not arrived at the limit, at a final stopping place; we are only at a station, a way station—we have paused, but not to remain. We do not travel fast enough, nor will we, until a speed of one hundred miles per hour is attained with as much ease, and as little risk, as at present.

In defense of this idea, he explicitly appeals to history:

Does the idea seem preposterous? Is it foolish, visionary? Is it absurd?

Let us inquire into the matter a little: let us extract a few notes from the history of progress.

What follows is a long passage about the history of roads and horse travel in England starting in the 16th century. He points out that horseback riding was slow, uncomfortable, and dangerous. He describes the establishment of the English post office in 1660, which greatly sped up mail delivery; and the development of stage coach service, which was an improvement on riding horseback.

He also points out that advances in travel were often unappreciated, even opposed. He quotes several paragraphs from a 1673 pamphlet decrying the stage coaches and pining for the old days of horseback. The pamphlet claimed that ho...
What is the interpretation of the do() operator?
by Bunthut — 2020-08-26T21:54:07.675Z

To reductively explain causality, it has to be explained in non-causal terms, most likely in terms of total propability distributions. Pearl explains causality in terms of causal graphs which are created by conditionalizing the propability distribution on not event, but do(event). What does this mean? It's easy enough to explain in causal terms: You make it so event occurs without changing any of its causal antecedents. But of course that fails to explain causality. How could it be explained without that?
Introduction To The Infra-Bayesianism Sequence
by Diffractor — 2020-08-26T20:31:30.114Z

TLDR: Infra-Bayesianism is a new approach to epistemology / decision theory / reinforcement learning theory, which builds on "imprecise probability" to solve the problem of prior misspecification / grain-of-truth / nonrealizability which plagues Bayesianism and Bayesian reinforcement learning. Infra-Bayesianism also naturally leads to an implementation of UDT, and (more speculatively at this stage) has applications to multi-agent theory, embedded agency and reflection. This post is the first in a sequence which lays down the foundation of the approach.

Prelude:

Diffractor and Vanessa proudly present: The thing we've been working on for the past five months. I initially decided that Vanessa's scattered posts about incomplete models were interesting, and could benefit from being written up in a short centralized post. But as we dug into the mathematical details, it turned out it didn't really work, and then Vanessa ran across the true mathematical thing (which had previous ideas as special cases) and scope creep happened.

This now looks like a new, large, and unusually tractable vein of research. Accordingly, this sequence supersedes all previous posts about incomplete models, and by now we've managed to get quite a few interesting results, and have ideas for several new research directions.

Diffractor typed everything up and fleshed out the proof sketches, Vanessa originated almost all of the ideas and theorems. It was a true joint effort, this sequence would not exist if either of us were absent. Alex Mennen provided feedback on drafts to make it much more comprehensible than it would otherwise be, and Turntrout and John Maxwell also helped a bit in editing.

Be aware this sequence of posts has the math textbook issue where it requires loading a tower of novel concepts that build on each other into your head, and cannot be read in a single sitting. We will be doing a group readthrough on MIRIxDiscord where we can answer questions and hopefully get collaborators, PM me to get a link.

Introduction:

Learning theory traditionally deals with two kinds of setting: "realizable" and "agnostic" or "non-realizable". In realizable settings, we assume that the environment can be described perfectly by a hypothesis inside our hypothesis space. (AIXI is an example of this) We then expect the algorithm to converge to acting as if it already knew the correct hypothesis....
WebBundles and URL Randomization
by jefftk — 2020-08-26T17:50:05.692Z

Disclosure: I work on ads at Google; this is a personal post.

Peter Snyder of Brave recently wrote
a post
on the proposed WebBundle specification arguing that it makes URL
randomization easier and so is harmful to content blocking. The core
of the post is incorrect, however, and based on a misunderstanding of
what can be done today and what WebBundles make easier. Snyder and I
discussed this some in on HN and I
wanted to take a step back and try to write up the issues more
clearly.

A WebBundle (see explainer)
allows you to serve a single file that the browser can treat as
multiple files. This solves many long-standing problems, some of
which I've been working around for a long time. A classic one is that
many sites will have a large number of small files. Everyone who
visits the site will need essentially all of these files, and you lose
a lot of performance in individually requesting each one. When I
worked on open source web page optimization software (mod_pagespeed)
it could combine CSS, JS, and
images
so you could request a single 'bundle' for each, but it was not ideal:

Instead of one bundle containing the resources the site needs,
you have at least three: CSS, JS, images.

There are differences in error handling and recovery that mean
you can't just concatenate CSS or JS files. Even with code to work
around these issues there were still differences that meant automatic
bundling was a bit risky.

Image combining (spriting) Is especially difficult, and our
tool was less of a fully automatic image combiner than a tool to take
the toil out of doing it manually.

Instead of using a dynamic tool like mod_pagespeed, you can also do
this kind of concatenation at build time (webpack, browserfy, parcel, rollup, etc) but this
doesn't fix any of these problems.

A WebBundle is a much more straight-forward approach, where you tell
the browser explicitly about your bundling instead of trying to glue a
bunch of things together and pass them off to the browser as one.

I'm no longer working on mod_pagespeed, but these issues still come
up; in my work now I'm interested in using WebBundles
to allow a single ad response to provide multiple ads.

My understanding of Snyder's view, from their post,
HN
comments, and spec
comments, is that bundles make it much easier to randomize URLs,
and bypass URL-based content blockers. Specificall...
[AN #114]: Theory-inspired safety solutions for powerful Bayesian RL agents
by Rohin Shah — 2020-08-26T17:20:04.960Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

SECTIONS

HIGHLIGHTS

TECHNICAL AI ALIGNMENT

ITERATED AMPLIFICATION

MESA OPTIMIZATION

AGENT FOUNDATIONS

FORECASTING

MISCELLANEOUS (ALIGNMENT)

OTHER PROGRESS IN AI

REINFORCEMENT LEARNING

NEWS

HIGHLIGHTS

The Alignment Problem for Bayesian History-Based Reinforcement Learners (Tom Everitt et al) (summarized by Rohin): After forgetting its existence for quite a while, I've finally read through this technical report (which won first place in round 2 of the AI alignment prize (AN #3)). It analyzes the alignment problem from an AIXI-like perspective, that is, by theoretical analysis of powerful Bayesian RL agents in an online POMDP setting.

In this setup, we have a POMDP environment, in which the environment has some underlying state, but the agent only gets observations of the state and must take actions in order to maximize rewards. The authors consider three main setups: 1) rewards are computed by a preprogrammed reward function, 2) rewards are provided by a human in the loop, and 3) rewards are provided by a reward predictor which is trained interactively from human-generated data.

For each setup, they consider the various objects present in the formalism, and ask how these objects could be corrupted, misspecified, or misleading. This methodology allows them to identify several potential issues, which I won't get into as I expect most readers are familiar with them. (Examples include wireheading and threatening to harm the human unless they provide maximal reward.)

They also propose several tools that can be used to help solve misalignment. In order to prevent reward function corruption, we can have the agent simulate the future trajectory, and evaluate this future trajectory with the current reward, removing the incentive to corrupt the reward function. (This was later developed into current-RF optimization (AN #71).)

Self-corruption awareness refers to whether or not the agent is aware that its policy can be modified. A self-corruption unaware agent is one that behaves as though it's current policy function will never be changed, effectively ignoring the possi...
Excusing a Failure to Adjust
by Ronny Fernandez — 2020-08-26T13:51:54.132Z

Crossposted from Figuring Figuring.

Here is a handle for a mistake I make all the time that works really well for me… that is the handle works really well for me. I make this mistake fairly often, but of course I notice it in other people more often.

Imagine that your credence in a belief is represented by a physical slider. When you learn a new piece of evidence or argument or whatever, there are two kinds of mental motions you can react with. You can adjust the slider up or down, or you can explain why you don’t have to adjust the slider at all.

Sometimes the right thing to do really is to not adjust the slider. Sometimes people will think that you should adjust the slider when really you shouldn’t for reasons that are difficult to explain. But this is far far rarer than cases where you should adjust the slider, but instead you focus on coming up with reasons to not adjust the slider. On some days, it seems like 90% of human cognition is coming up with reasons to not adjust the slider.

And of course, for beliefs that you like, you will come up with excuses for refusing to adjust the slider down way more often than you will come up with excuses for refusing to adjust the slider up. And vice versa for beliefs that you dislike.

Probably, you would be better off banning that mental motion altogether. Nearly every new piece of evidence should cause some adjustment, even if it is very small. Most conversations should be about how much to adjust, not about whether to adjust. But banning that mental motion is difficult, and before you can, you must learn to notice the motion, and learn how it feels different from other motions.

This is the best tool I have found for noticing so far. I ask myself: “Am I trying to explain why I don’t need to adjust the slider?” Something different might work better for you, but I highly recommend trying at all to notice the difference.
nostalgebraist: Recursive Goodhart's Law
by Kaj_Sotala — 2020-08-26T11:07:46.690Z
Link: https://nostalgebraist.tumblr.com/post/186105132274/theres-this-funny-thing-about-goodharts-law

Would kind of like to excerpt the whole post, but that feels impolite, so I'll just quote the first four paragraphs and then suggest reading the whole thing:

There’s this funny thing about Goodhart’s Law, where it’s easy to say “being affected by Goodhart’s Law is bad” and “it’s better to behave in ways that aren’t as subject to Goodhart’s Law,” but it can be very hard to explain why these things are true to someone who doesn’t already agree.

Why? Because any such explanation is going to involve some step where you say, “see, if you do that, the results are worse.” But this requires some standard by which we can judge results … and any such standard, when examined closely enough, has Goodhart problems of its own.

There are times when you can’t convince someone without a formal example or something that amounts to one, something where you can say “see, Alice’s cautious heuristic strategy wins her $X while Bob’s strategy of computing the global optimum under his world model only wins him the smaller $Y, which is objectively worse!”

But if you’ve gotten to this point, you’ve conceded that there’s some function whose global optimum is the one true target. It’s hard to talk about Goodhart at all without something like this in the background – how can “the metric fails to capture the true target” be the problem unless there is some true target?
On Suddenly Not Being Able to Work
by Jacob Falkovich — 2020-08-25T22:14:45.747Z

Cross-posted, as always, from Putanumonit.

Since I started Putanumonit five years ago I’ve rarely ever gone two weeks without posting anything. I’ve now gone a full month.

It’s not that I have nothing to write about. I have several half-finished drafts, Twitter threads that can be expanded into posts, marked-up books for review, and more. This post is none of the above, it’s about the reason why I have not been writing. And chiefly, it’s a way to just get back to writing by any means possible, to make this a habit again.

If this sounds like boring post that you don’t want to read, you are probably correct. You can instead watch my video interview with Chase Harris about COVID, Rationality, social reality, and more.

The reason I haven’t blogged in a month is that I haven’t been able to do my job. My work laptop is plugged in, my colleagues are online, my deadlines are approaching fast, and yet I have consistently failed to put in the required (and not very large) number of hours it would take me to catch up on my work. I am not sure why this has been happening, and I have observed this happening as if from the outside with some bemusement.

My brain keeps “work” and “blogging” very near each other on the mental shelf. Both are obligations of a sort. Both are done on the same screen with the same keyboard sitting on the same chair. And so, since my rising panic at falling behind on work is ever-present, whenever I sit down to write I think I should do work instead — it pays my rent and has specific people that depend on it, as opposed to an abstract readership. So whenever I try to blog I switch to trying to work. Then I fail and end up doing neither.

This experience of not being able to do work is very strange. On the face of it, nothing is different in August from March and April when I was working from home quite productively on similar projects. I am seeing more friends now and spending more time outdoors but these usually happen in the evening, when I have already spent the day mysteriously not working.

I don’t particularly dread my work or feel stressed about it. It’s as if the job is shrouded in fog — it’s hard to get started and when I do I’m easily distracted by other things which capture my attention for unusually long periods of time. I have also noticed in the last two weeks that my stomach acts up whenever I log in to the work laptop, even several times a...
I Started a Sports and Gambling Substack
by Zvi — 2020-08-25T21:30:00.876Z

The substack is here.

The latest post is here and contains predictions for tomorrow’s baseball games. I also posted predictions for yesterday’s games here, and here you can find my offensive team ratings. Pitcher ratings will be posted soon.

I wanted to try out the substack system and software as a potential landing point, and as per my discussion regarding the forced switch to the new WordPress editor. I also wanted to avoid conflating the sports modeling and gambling talk with my rationality blog, as I expect them to be relevant to the interests of different groups of people.

If you have meaningful feedback for what the model has to say, or are finding these posts interesting or useful, let me know and also hopefully subscribe to the substack. If not, I’ll likely periodically mention it is there, but will do my best to keep the two efforts distinct.