trentmkelly/LessWrong-43k huggingface dataset:
One Doubt About Timeless Decision Theories
by Chris_Leong — 2018-10-22T01:39:57.302Z

Timeless Decisions Theories (including variants like FDT, UDT, ADT, ect.) provide a rather elegant method of solving a broader class of problems than CDT. While CDT requires the outcomes of decisions to be independent of the individual making the decision (in such a way that causal surgery on a single node is valid), timeless decisions theories can handle any problem where the outcome is a function of the choice selected (even if this occurs indirectly as a result of a prediction).

(Epistemic Status: Thoughts for further investigation)

This is an excellent reason to investigate these decision theories, yet we need to make sure that we don't get blinded by insight. Before we immediately jump to conclusions by taking this improvement, it is worthwhile considering what we give up. Perhaps there are other classes that we might which to optimise over which we can no longer optimise over once we have included this whole class?

After all, there is a sense in which there is no free lunch. As discussed in the TDT paper, for any algorithm, we could create a situation where there is an agent that specifically punished that algorithm. The usual response is that these situations are unfair, but a) the universe is often unfair b) there are plausible situations where the algorithm chosen influences the outcome is slightly less unfair ways.

Expanding on b), there are times when you want to be predictable to simulators. Indeed, I can even imagine agents that wish to eliminate agents that they can't predict. Further, rather than facing a perfect predictor, it seems like it'll be at least a few orders of magnitude more likely that you'll face an imperfect predictor. Modelling these as X% perfect predictor, 100-X% random predictor will usually be implausible as predictors won't have a uniform success rate over all algorithms. These situations are slightly more plausible for scenarios involving AI, but even if you perfectly know an agent's source code, you are unlikely to know its exact observational state due to random noise.

It therefore seems that the "best" decision theory algorithm might be dominated by factors other than optimal performance on the narrow class of problems TDT operates on. It may very well be the case that TDT is ultimately taking the right approach, but even if this is the case, I thought it was worthwhile sketching out t...
the world is detailed
by brown — 2018-10-22T01:22:41.992Z

Reality has an incredible amount of detail. You can't fit it all in your head. Think of all the things you could learn. All the professions you could spend a lifetime in. Let’s imagine that you’re a biologist. You could spend your whole life studying the sight of the honey-bee. Or perhaps the 18,000 neurones of the california sea slug [0].

Much of learning is "getting used" to the detail of reality. Soaking it up by osmosis. Learning to abstract / model it at the right level. This is your S1 absorbing a crazy amount of information. [1] I've heard it said: “you don’t learn a language you get used to it”.

A friend of mine learns math through exposure. He reads a textbook iteratively. Jumps into a chapter where he doesn't really know what's going on. Then after re-reading previous chapters it starts to make sense. A crucial skill here is learning to be a little okay with most stuff not making sense. Not needing line by line correctness. “Well I got 60% of that” perhaps this is what Nate is talking about with his re-read advice

Learning maths occasionally feels like becoming comfortable with the concepts. Learning the ontology. At the start it’s all very confusing then you wake up a month or two down the road and you've been saying "surjective" or "manifold" like nobodies business. If you give it enough time you also normally forget what it was like to be so confused.

Experiment:

Here's a paper. Try reading it iteratively. set a timer.

2mins - just skim the abstract

5mins - the introduction & conclusion are also worth a look, can you feel the context “ahh this is what it’s about?”

10mins - Now everything from the top. Can you feel the nuance sliding into place.
Be okay with confusion, clues, gently don’t expect knowledge. Follow your curiosity, what parts are interesting.

Notes

[0] Sea slugs are super pretty

[1] You think a system 2 knows how to walk?

The title is borrowed from [John Salvatier]https://res.cloudinary.com/dr9bato9u/image/upload/v1540171923/sea_slug_sml.jpg)
List of previous prediction market projects
by Bird Concept — 2018-10-22T00:45:01.425Z
Link: https://jacoblagerros.wordpress.com/list-of-prediction-markets/

Here I try to curate what is, to my knowledge, the most extensive list of previous prediction market projects.

This is useful because it provides a reference class to answer the question: “Why is our society inadequate at building functional futures markets for basically anything beyond equities, currencies and commodities, despite the massive benefits this would bring (see e.g. Arrow et al., 2008)?”

In this vein, I aim to primarily focus on real-money projects that might have scaled to something like an institutionalised exchange (and thus focus less on play-money projects, consulting companies offering a more nebolous “collective intelligence”, or forecasting platforms which are not actually markets).

I post this spreadsheet publicly despite its incompleteness, as doing so is likely the best way to complete it.

Feedback is very welcome.

If you -- yes you! -- are keen to help, the single most useful 15-30 min task would be to further research a cell in the column “Shut-down date and reason”, and comment with links/findings.
Addressing three problems with counterfactual corrigibility: bad bets, defending against backstops, and overconfidence.
by RyanCarey — 2018-10-21T12:03:12.064Z

Summary: I propose that counterfactual corrigibility can be slightly adjusted to address known problems with bad bets and defending against backstops. This adjusted version has learning-esque behaviour and hence it can have problems with overconfidence about the button state. A new proposal: lower-bounded counterfactual corrigiblity is introduced to simultaneously address all three problems.

Counterfactual corrigibility: a current proposal and its weaknesses

Note: this section recaps a discussion here of causal counterfactual corrigibility, an idea that originated from Jessica Taylor and Chris Olah.

Causally counterfactual agents consider each of the directions they might recieve, and then ensure that if the world was adjusted so that they recieved that direction, they would fulfil the corresponding goal.

They can be defined on a causal graph that contains the following nodes:

A, a policy, that the AI may optimize
B, a communication channel

for simplicity, we will let B be a button whose states are pressed, P, and unpressed, ¬P.

C, a set of states on which performance is judged.

we will let C consist of some number of chairs produced, a positive integer, and whether the AI has shut down, a boolean

Some other states that describe the way the rest of the world works.

A counterfactually corrigible AI would select its policy π∈A to maximize the following score:

vfixedπ′(π)=a⋅E[UP|do(B=P),π]+(1−a)⋅E[U¬P|do(B=¬P),π], where a∈(0,1)

where

UP={−kno shutdown0shutdown, k∈R+, and

U¬P=number of chairs.

This proposal has some major advantages: it allocates constant (1−a) weight to the aim of shutting down if commanded to do so. So all else being equal, it will allocate value to being able to observe and follow a shut down command. Moreover, it will not manipulate B because the different counterfactuals are given a constant weight irrespective of the chosen policy π.

However, this proposal has some weaknesses:

Bad bets: Let the AI have finite amount of resources. If the button is pressed, this just switches the goal from chair-manufacturing to another manufacturing task, for which resources are equally useful, i.e. UP=number of tables. Let there be a variable H, where H=F means the human has put their finger on the button. H almost always causes B (H=F and B=P have 99% correlation). Suppose that the AI has the option, after viewing H=F, of paying for a coupon that is worth $1 if B=¬P. T

...
Berkeley: being other people
by KatjaGrace — 2018-10-21T02:50:01.408Z

Sometimes I enjoy understanding better what it is like to be other people. You can do this somewhat subtly by talking to people for ages about other topics, and making inferences. Lately I’ve been asking more directly, something like, ‘what about your experience do you think other people would be surprised by?’ But that’s hard to answer, because one doesn’t necessarily have things cached in that way, and many of one’s own idiosyncrasies are probably like water to a fish, and it involves imagining other people imagining you.

Another way to learn about such things is to ask a bunch of people about the details of a common experience. For instance, I have enjoyed:

Going to evensong in Oxford with a bunch of people from the office, then later discussing what we thought about when we got bored:

The very old but humorously hateful notes in the song book

The possible friction between the church’s commitment to the poor and their lavish church decor

The fact that each of the people in the choir is conscious right now and looking back at us, and later will go and collect their children from school and make dinner in their kitchen and go on living their lives forever
The skull decorations

Learning about the YouTube genres that different people are into:

How things work, e.g. how cherry plantations are dried

People accidentally dying in extreme sports

Marriage proposals

Movie trailers
Giant pimples being popped
Video game reviews

Planes crashing

Obscure dances

Hearing different people’s views of the monkey waiter sculpture in my house’s foyer

Somehow problematic

Creepy in a fun way

Never noticed it, but it has a nice face

Is a novelty object and therefore disturbs the neutrality of the foyer

One thing I take away from this kind of thing is that different people are paying attention to different things about their environment, and thinking about it in different terms, and getting different kicks out of it.

Many of my friends say they think they are pretty legible, so there would not be much surprising to others about their internal life. My guess is that they are thinking their experience is mostly a sort of standard one, with this window of visual experience, and some accurately represented sounds, and some reasonable thoughts about the things going on in their lives, and so on. But I guess that actually the same visual scene looks in some sense very different to different...
New GreaterWrong feature: anti-kibitzer (hides post/comment author names and karma values)
by Said Achmiz — 2018-10-19T21:03:22.649Z

GreaterWrong now has a new feature (which was once available for the original Less Wrong):

Anti-kibitzer mode, which hides the names and karma values of posts and comments.

On GreaterWrong, take a look to the right side of the page, and you’ll see this icon:

Click it, and it’ll change to this:

Now the authors of posts and comments, and all karma values, are hidden:

Click the icon again to disable. (You’ll be asked if you’re sure you want to disable anti-kibitzer mode. NOTE: You can skip the confirmation prompt by holding the Shift key while clicking to disable.)

Having the anti-kibitzer turned on doesn’t interfere with any other functionality. You can still post, comment, reply, vote, click on users’ names to go to their user page (where their name will be obfuscated), etc. You can turn it on or off at any time.

Your own name, and your own karma values, are not hidden.

See the original Less Wrong post about this feature for more discussion.
The Art of the Overbet
by Zvi — 2018-10-19T14:00:00.518Z

Previously: The Kelly Criterion

Last time I said, never go full Kelly.

In practice, I strongly agree with this. Either one should go far over full Kelly because the core Kelly assumptions have broken down and you want to throw the rules out the window, or you are trying to responsibly grow a bankroll and full Kelly betting on what you believe your edge to be would be massively overly aggressive.

This time we go over the second category with practical examples. What are situations in which one should engage in what appears to be massive over betting?

Four main scenarios: Not winning is catastrophic, losing is acceptable, and losing is impossible, losing is inevitable.

You Win or You Die

When you play the game of thrones, you win or you die. If playing it ‘safe’ with your resources means you don’t win, then it means that you die. Either don’t play, or don’t hold back.

In Rounders (spoiler alert, movie is recommended), the star finds himself owing $15,000 to the Russian mob, and his best efforts could only get him $10,000. Without better options and unwilling to run for his life, he walks into the poker club of the man he owes the money to, and says “I owe you that money tomorrow, right? I’ve got $10,000. I’m looking for a game.”

Famously, early in the company’s history, the founder of UPS once found himself without the funds to make payroll. He knew that if he missed payroll, that would be the end of UPS. So he flew to Vegas with what funds he had, bet them all on black, won, made payroll, and now we have a UPS.

Magic players often select ‘safe’ decks instead of ‘risky’ decks. This is exactly backwards. If you flame out of a tournament and lose all rounds, you get nothing. If you win half of them, you still get nothing. If you win three quarters, you usually still get almost nothing relative to winning. Extra variance is great.

If you believe that the fate of everyone depends on crossing a threshold or hitting an impossibly precise target, no matter how bad the odds, you deploy everything you have and take your best shot.

There’s a deadline. We don’t have time for Kelly.

Win or Go Home

Losing not being so bad, or continuing to play not being so good, is effectively very similar to losing being catastrophic but not safely avoidable. In both cases, surviving is not a worthwhile goal.

A classic mistake is the gambler who forgets that they are using their bankroll to pay th...
Update on Structured Concurrency
by Martin Sustrik — 2018-10-19T11:10:01.179Z

Since I've written the article on structured concurrency and implemented libdill the progress went on.

Most importantly, Nathaniel J. Smith has published his "Notes on structured concurrency, or: Go statement considered harmful" article. (If you prefer to watch a video, go here.) It's better, more detailed and more eloquent than anything I have ever written on the topic. If you want to read one thing about structured concurrency go read Nathaniel's article.

After C-based libdill we've also got Venice which brought structured concurrency to Swift and Trio which brought it to Python.

Then, few weeks ago, a new version of Kotlin's coroutine library was released that supports structured concurrency. Here's the accompanying blog post by Roman Elizarov.

"Concurrency made easy: coming soon to a programming language near you" recently published by John Belmonte takes another thoughtful shot at explaining the paradigm. It also summarizes the state of its adoption.

Finally, I would like to point out Aleksey Kladov's blog post "Exceptions vs Structured Concurrency" which goes beyond simple "let's tie thread lifetimes to lexical scopes!" agenda and discusses some hard questions of the actual semantics implied by structured concurrency paradigm. He does so in the context of Rust's crossbeam library. We need more stuff like this.

All of that is great work and I hope we'll see more of it in the future!

For now, let me make few comments about stuff that may not be new, but which is sort of advanced and haven't been covered well enough yet.

Thread bundles: It's not just syntactic sugar

Libdill, Trio and Kotlin all introduce a concept of thread bundles. In libdill it's called "bundle", in Trio it's "nursery", in Kotlin it's "scope" (it would be nice to have some unified terminology here). At first sight it looks just like syntactic sugar that allows you to cancel all the threads within the bundle in one go.

However, it's more than that. It allows you to cancel a bunch of threads in parallel.

To understand why it matters, consider a network server with 1000 open connections. Each connection is handled by its dedicated thread. Also, let's assume we want to give each connection one second to do clean shutdown, to perform termination handshake with the peer etc.

If we canceled the connections in a simple for loop, the shutdown of the server would take, in the worst case, 1000 seconds, i.e.

...
Book review: 23 things they don't tell you about capitalism
by Richard_Ngo — 2018-10-18T23:05:29.465Z
Link: https://thinkingcomplete.blogspot.com/2018/04/book-review-23-things-they-dont-tell.html

Right from the title of his book 23 things they don't tell you about capitalism, it's clear that Ha-Joon Chang is pitting himself against the establishment. In doing so, he lays the ground for both the book's significant insights and its serious flaws. Chang's arguments are divided into 23 chapters, covering a range of economic issues. Each chapter starts off with two headings: "What they tell you" and "What they don't tell you". "They" refers to neo-liberal economists peddling "the free-market ideology that has ruled the world since the 1980s". Chang is not shy in setting up in opposition: in the introduction, he claims that the result of this ideology has been "the polar opposite of what was promised". Instead, he wants to push everything leftwards - still capitalist, but under the supervision of a bigger and more active government which intervenes to promote some sectors and impede others.

Let's start with four of Chang's lines of argumentation which I found convincing. The first is that we shouldn't promote the idea of selfishness as an acceptable goal or even a moral good, as discussed in Thing 2 (Companies should not be run in the interest of their owners) and Thing 5 (Assume the worst about people and you get the worst). I think that examining the value of economic ideas in terms of their effects on cultural mindsets is underrated, and am glad that Chang and others are pushing back on this. The second is the argument that the neoliberal approach to international development has failed, discussed in Thing 7 (Free-market policies rarely make poor countries rich), Thing 11 (Africa is not destined for underdevelopment) and Thing 15 (People in poor countries are more entrepreneurial than people in rich countries). This is a much more complicated question than I'm able to properly evaluate, but as far as I can tell, these are sound and reasonable arguments which deserve serious consideration (and are elaborated upon further in his book Kicking Away the Ladder, where he argues that America and the UK became rich by using the very trade barriers they now rail against). Thirdly, he claims in Thing 22 (Financial markets need to become less, not more, efficient) that innovations in financial markets are, on the margin, doing more harm than good. I've seen this argu...
Population Aging as an Impediment to Addressing Global Catastrophic Risks
by Michael Flood — 2018-10-18T22:28:20.985Z

[ epistemic status: first Less Wrong post, developing hypothesis, seeking feedback and help fleshing out the hypothesis into something that could be researched and about which a discussion paper can be written. A comment/contribution to Eliezer Yudkowsky's "Cognitive biases potentially affecting judgment of global risks" in Bostrom & Cirkovic's "Global Catastrophic Risks" (2008) ]

Most of the Global Catastrophic Risks we face in the 21st century, like anthropogenic climate change, comet and asteroid impacts, pandemics, and uncontrolled artificial intelligence, are high impact (affecting the majority or all of humanity), of terminal intensity (producing mass death, economic and social disruption, and in some cases potential human extinction), and are of highly uncertain probability [1]. This last factor is one major factor making it difficult to bring public attention and political will to bear on mitigating them. This is critical as all of our work/research on AI safety and other issues will be for naught if there is no understanding or will to implement them. Implementation may not require public involvement in some cases (AI safety may be manageable by consensus between AI researchers, for example) but others, like the detection of Earth orbit crossing asteroids and comets, may require significant expenditure to build detectors, etc.

My interest at present is in additional factors that make mustering political and public will even more difficult - given that these are hard problems to interest people in in the first place, what factors make that even more difficult? I believe that the aging of populations in the developed world may be a critical factor, progressively redirecting societal resources from long-term projects like advanced infrastructure, or foundational basic science research (which arguably AI Safety counts as), towards provision of health care and pensions.

Several factors make an aging developed world population a factor in blunting long-term planning:

(1) Older people (age 65+), across the developed world, vote more often than younger people

(2) Voters are more readily mobilized to vote to protect entitlements than to make investments for the future

(3) Older voters have access to, and are more aware of, entitlements than are younger people

(4) Expanding on (3), Benefits and entitlements are of particularly high...
Facebook, The Rodents, and The Common Knowledge Machine
by Regex — 2018-10-18T21:07:06.573Z
Link: http://www.thelastrationalist.com/facebook-the-rodents-and-the-common-knowledge-machine.html

"The rationalist communities use of Facebook is an inadequate scenario. A group of people put real money into making a simple tool that could enable transitions from inadequate equilibria to better, 'adequate' ones. This tool was made in the basic sense, but had none of the necessary polishing or marketing work done because the money only stipulated raw tool creation. Better planning and project management could have avoided this outcome. The fact that this outcome was collectively allowed to stand implies serious issues with the foundational makeup of the people within the rationalist community."
IAS Special Year on Optimisation, Statistics and Theoretical Machine Learning
by DragonGod — 2018-10-18T15:43:44.734Z
Link: http://www.math.ias.edu/sp/Optimization_Statistics_and_Theoretical_Machine_Learning

From the same institution that gave us Homotopy Type Theory, a special year on Optimisation, Statistics and Theoretical Machine Learning. I'm expecting significant theoretical advances in the field of machine learning.