trentmkelly/LessWrong-43k huggingface dataset:
deluks917 on Online Weirdos
by Jacob Falkovich — 2018-11-24T17:03:20.474Z

Cross-posted from Putanumonit.

This is part 1 of an interview with deluks917. Stay tuned for part 2 in which we talk about Buddhism and peanut butter, and part 3 which covers archetypes, superego poisoning, and Moana.

Since I have a face for radio and a voice for blogging, our conversation is transcribed for your pleasure and edited for readability. This is my second transcribed interview; suggestions for other people to chat with are welcome!

Jacob: I met you through the New York rationalist meetup, which now often takes place in your apartment. How did you even become part of this world?

deluks917: I had sporadically attended the New York meetups in the past, and then I was away in Pennsylvania for three years. I had been reading SlateStarCodex and commented under a couple of different names there. Later I made the SlateStarCodex Discord, and it took off around the time I came back to New York. The stars aligned. There was a room in a rationalist house that needed to be filled and no one else was going to take it.

For me, rationality filled a hole that had always existed in my life. Being nerdy and a systematizing thinker I was always reading psychology books and the like, but nothing seemed quite right until I read the Sequences.

I tried reading the Sequences a couple of times, starting seriously about eight years ago. I found them pretty confusing. But then I came back later and they made more sense. It took some time for things to marinate and for my mood to change.

I have always been part of nerdy communities, either online or IRL. I feel like my intellectual interests change pretty regularly and I happened to come back to rationality at the right time. By 2013 or so ideas about AI suddenly seemed more profound. I don’t know if I was wise or foolish not to be convinced about AI risk in 2010, maybe I was just too young. But by 2013 I was pretty convinced.

You created the SSC Discord and you do a lot of moderation of online communities. What does that involve day to day? And why do you do it?

When I first started it, it took a lot of my time. I wanted to get things on the right track. It has progressively taken less and less time. One thing that does that still takes time is the rationality feed: a daily digest of recommended rationality articles. In addition to vetting the articles, I include a short review or summary quote. Originally I started the feed to en...
Upcoming: Open Questions
by Raemon — 2018-11-24T01:39:33.385Z

(note: everything in this post is still subject to change)

I wanted to give people some sense of our next priorities (and why you've seen fewer public-facing features lately). The LessWrong team's current project is Open Questions (or "Q&A" for short)

This will be a new format for posts, and probably a new section of the site. Users can post questions, or respond to them with detailed answers, similar to sites such as Quora, StackExchange or MathOverflow.

Many different types of questions will be fine (ranging from noob questions, to open problems in AI alignment, to just trying to understand a given phenomena).

The aspiration here is to aim for rigor in our question-answers (a la a place like MathOverflow), but to encourage questions in somewhat murkier domains, where the answer may not be clear cut, and "progress" on a question may look more like refactoring into a different question, or breaking it up into small chunks and hacking away at the edges.

Goals for Open Questions

1. Develop more "demand driven" content on LessWrong. Right now, people basically come up with ideas for posts based on their own curiosity or interest, but they don't have much sense of how much other people will care. With Open Questions, you know that at least one person actually cares about understanding something (and if their question is highly upvoted, many other people do as well).

2. Tweak the site to focus more explicitly on intellectual progress. Right now, people show up on LessWrong with a vague desire to learn, but not a clear goal. This means that the default discussions often end up feeling more like "nerds hanging out on the internet" than "nerds gaining skills, learning or solving interesting problems."

Q&A will aim to:

Add a new common, default activity that's a bit more goal directed than typical LW discussion.
Help us notice when there are longstanding important, confusing questions unresolved. And, more excitingly, help us notice when we've made significant progress on them.

Ultimately, we'd like to Q&A section to not just be useful for answering one-of questions but for laying out actual research agendas. (Possibly with a higher-level organization that groups questions together, similar to how sequences group posts).

3. Create a lower-barrier-to-entry action for new users.

We intend ...
A Dragon Confronts the Terasem Movement
by Alephywr — 2018-11-24T01:31:52.288Z
Link: https://dancefighterredux.wordpress.com/2018/09/02/a-dragon-confronts-the-terasem-movement/

I wrote a previous blog takedown of Malatora, which was somewhat shooting fish in a barrel. In the spirit of moving on to harder targets, I now turn my meager attentions and talents to the subject of the Terasem Movement. This is a marginally more difficult movement to vivisect, but ultimately there are many problems with it, although I must say that I am on the whole much more sympathetic to the Terasem Movement than to Malatora, even though that isn’t a difficult hurdle to clear. Let’s get right down to it:

Addressing Terasem

General Impressions

There are many different components to the Terasem movement. There’s a kind of dopey new-agey religiosity that combines less-than-fully researched physics knowledge with some Carl Sagan tropes, some general scientistic philosophical incompetence, and throws in a dash of instrumentalist reinterpretation of tradition for good measure. Taken together the overall result is not great: Terasem lacks the ethos or justification of real science while simultaneously stripping religion of its force through its instrumentalization. This is not a strong foundation for any sort of normative system, and most of the subsequent problems of Terasem stem from a mixture of intentional and unintentional sophistry that isn’t quite enabled by this, but certainly isn’t helped by it; it is one of the flaws which the sophistry most tries to brush under the carpet. It’s best to describe Terasem as “Science-adjacent”. What this means isn’t so much that it’s antithetical or exclusionary to science, and indeed many of its members seem very well-good credentialed. But it’s not science, at all, and the use and misuse of technological terminology makes my bullshit detector explode. It’s possible I am just not educated enough to understand certain terms and concepts, so I will leave this alone for the most part and operate as if everyone is engaged in good faith.

Besides the religious component, there are also various philosophical leitmotifs and idée fixes which are regrettable, most especially in the frequent appearance of contradiction. I understand that a large movement can’t be entirely consistent, but the differences that are glossed over are more substantive than what is actually explicated, and I will show a few examples of this later. I also get the impression that many of these people believe that simply defining enough terms wit...
What if people simply forecasted your future choices?
by ozziegooen — 2018-11-23T10:52:25.471Z

tldr: If you could have a team of smart forecasters predicting your future decisions & actions, they would likely improve them in accordance with your epistemology. This is a very broad method that's less ideal than more reductionist approaches for specific things, but possibly simpler to implement and likelier to be accepted by decision makers with complex motivations.

Background

The standard way of finding questions to forecast involves a lot of work. As Zvi noted, questions should be very well-defined, and coming up with interesting yet specific questions takes considerable consideration.

One overarching question is how predictions can be used to drive decision making. One recommendation (one version called "Decision Markets") often comes down to estimating future parameters, conditional on each of a set of choices. Another option is to have expert evaluators probabilistically evaluate each option, and have predictors predict their evaluations (Prediction-Augmented Evaluations.)

Proposal

One prediction proposal I suggest is to have predictors simply predict the future actions & decisions of agents. I temporarily call this an "action prediction system." The evaluation process (the choosing process) would need to happen anyway, and the question becomes very simple. This may seem too basic to be useful, but I think it may be a lot better than at least I initially expected.

Say I'm trying to decide what laptop I should purchase. I could have some predictors predicting which one I'll decide on. In the beginning, the prediction aggregation shows that I have an 90% chance of choosing one option. While I really would like to be the kind of person who purchases a Lenovo with Linux, I'll probably wind up buying another Macbook. The predictors may realize that I typically check Amazon reviews and the Wirecutter for research, and they have a decent idea of what I'll find when I eventually do.

It's not clear to me how to best focus predictors on specific uncertain actions I may take. It seems like I would want to ask them mostly about specific decisions I am uncertain of.

One important aspect is that I should have a line of communication to the predictors. This means that some clever ones may eventually catch on to practices such as the following:

A forecaster-sales strategy

Find good decision options that have been overlooked

Make forecasts or bets on them suc

...
Oversight of Unsafe Systems via Dynamic Safety Envelopes 
by Davidmanheim — 2018-11-23T08:37:30.401Z

Idea

I had an idea for short-term, non-superhuman AI safety that I recently wrote up and ̶w̶i̶l̶l̶ ̶b̶e̶ ̶p̶o̶s̶t̶i̶n̶g̶ have now posted on Arxiv. This post serves to introduce the idea, and request feedback from a more safety-oriented group than those that I would otherwise present the ideas to.

In short, the paper tries to adapt a paradigm that Mobileye has presented for autonomous vehicle safety to a much more general setting. The paradigm is to have a "safety envelope" that is dictated by a separate algorithm than the policy algorithm for driving, setting speed- and distance- limits for the vehicle based on the position of vehicles around it.

For self-driving cares, this works well because there is a physics based model of the system that can be used to find an algorithmic envelope. In arbitrary other systems, it works less well, because we don't have good fundamental models for what safe behavior means. For example, in financial markets there are "circuit breakers" that function as an opportunity for the system to take a break when something unexpected happens. The values for the circuit breakers are set via a simple heuristic that doesn't relate to the dynamics of the system in question. I propose taking a middle path - dynamically learning a safety envelope.

In building separate models for safety and for policy, I think the system can address a different problem being discussed in military and other AI contexts, which is that "Human-in-the-Loop" is impossible for normal ML systems, since it slows the reaction time down to the level of human reactions. The proposed paradigm of a safety-envelope learning system can be meaningfully controlled by humans, because the adaptive time needed for the system can be slower than the policy system that makes the lower level decisions.

Quick Q&A

1) How do we build heuristic safety envelopes in practice?

This depends on the system in question. I would be very interested in identifying domains where this class of solution could be implemented, either in toy models, or in full systems.

2) Why is this better than a system that optimizes for safety?

The issues with balancing optimization for goals versus optimization for safety can lead to perverse effects. If the system optimizing for safety is segregated, and the policy-engine is not given access to it, this should not occur.

This...
On MIRI's new research directions
by Rob Bensinger — 2018-11-22T23:42:06.521Z
Link: https://intelligence.org/2018-update-our-new-research-directions/

Nate Soares has written up a post that discusses MIRI's new research directions: a mix of reasons why we're pursuing them, reasons why we're pursuing them the way we are, and high-level comparisons to Agent Foundations.

Read the full (long!) post here.
LW Update 2018-11-22 – Abridged Comments
by Raemon — 2018-11-22T22:11:10.960Z

Haven't posted an update in a while. We've been working on a mixture of bug fixes, small features to support the new AI Alignment Forum, as well as some longer-term, larger features that aren't quite ready yet.

Abridged Comments

The biggest change that should be visible to most users is our new Comment Abridging system. For posts with lots of comments, by default it now truncates them to make it easier to skim the thread. Clicking anywhere on a comment thread will expand all the comments below that point (so it's fairly easy to switch from "skim the thread" to "reading it in detail.")

We're fine tuning the exact details, but the current setup is:

Comments are abridged-by-default if the post has 25 or more comments. (See the thread from Realism About Rationality for a good example)
If a comment has more than 20 karma, it's given a higher character count in the abridging. (i.e. instead of cutting it down to 300 characters, it's cut down to 1000 characters). The idea is that if a newcomer arrives at a large thread, they can quickly get an overview of the discussion, with some sense of which parts of the discussion were most interesting, rather than automatically having their attention directed to whichever discussion was responding to the top-ranked top-level comment.
For right now, you can expand a given subthread by clicking on a comment node. We're thinking about ways to make it intuitive to expand all comments but hadn't quite solved the UI for that. (We're thinking of making it so that typing "command-F" automatically expands everything, so that if you're searching for a particular comment via a snippet of text the abridging quickly gets out of your way.)
Approval-directed agents
by paulfchristiano — 2018-11-22T21:15:28.956Z

Note: This is the first post from part two: basic intuitions of the sequence on iterated amplification. The second part of the sequence outlines the basic intuitions that motivate iterated amplification. I think that these intuitions may be more important than the scheme itself, but they are considerably more informal.

Research in AI is steadily progressing towards more flexible, powerful, and autonomous goal-directed behavior. This progress is likely to have significant economic and humanitarian benefits: it helps make automation faster, cheaper, and more effective, and it allows us to automate deciding what to do.

Many researchers expect goal-directed machines to predominate, and so have considered the long-term implications of this kind of automation. Some of these implications are worrying: if sophisticated artificial agents pursue their own objectives and are as smart as we are, then the future may be shaped as much by their goals as by ours.

Most thinking about “AI safety” has focused on the possibility of goal-directed machines, and asked how we might ensure that their goals are agreeable to humans. But there are other possibilities.

In this post I will flesh out one alternative to goal-directed behavior. I think this idea is particularly important from the perspective of AI safety.

Approval-directed agents

Consider a human Hugh, and an agent Arthur who uses the following procedure to choose each action:

Estimate the expected rating Hugh would give each action if he considered it at length. Take the action with the highest expected rating.

I’ll call this “approval-directed” behavior throughout this post, in contrast with goal-directed behavior. In this context I’ll call Hugh an “overseer.”

Arthur’s actions are rated more highly than those produced by any alternative procedure. That’s comforting, but it doesn’t mean that Arthur is optimal. An optimal agent may make decisions that have consequences Hugh would approve of, even if Hugh can’t anticipate those consequences himself. For example, if Arthur is playing chess he should make moves that are actually good—not moves that Hugh thinks are good.

The quality of approval-directed decisions is limited by the minimum of Arthur’s ability and Hugh’s ability: Arthur makes a decision only if it looks good to both Arthur and Hugh. So why would Hugh be interested in this proposal, rather than doing things himself?

Hugh doe

...
Believing others' priors
by rk — 2018-11-22T20:44:15.303Z

Meet the Bayesians

In one way of looking at Bayesian reasoners, there are a bunch of possible worlds and a bunch of people, who start out with some guesses about what possible world we're in. Everyone knows everyone else's initial guesses. As evidence comes in, agents change their guesses about which world they're in via Bayesian updating.

The Bayesians can share information just by sharing how their beliefs have changed.

"Bob initially thought that last Monday would be sunny with probability 0.8, but now he thinks it was sunny with probability 0.9, so he must have has seen evidence that he judges as 4/9ths as likely if it wasn't sunny than if it was"

If they have the same priors, they'll converge to the same beliefs. But if they don't, it seems they can agree to disagree. This is a bit frustrating, because we don't want people to ignore our very convincing evidence just because they've gotten away with having a stupid weird prior.

What can we say about which priors are permissible? Robin Hanson offers an argument that we must either (a) believe our prior was created by a special process that correlated it with the truth more than everyone else's or (b) our prior must be the same as everyone else's.

Meet the pre-Bayesians

How does that argument go? Roughly, Hanson describes a slightly more nuanced set of reasoners: the pre-Bayesians. The pre-Bayesians are not only uncertain about what world they're in, but also about what everyone's priors are.

These uncertainties can be tangled together (the joint distribution doesn't have to factorise into their beliefs about everyone's priors and their beliefs about worlds). Facts about the world can change their opinions about what prior assignments people have.

Hanson then imposes a pre-rationality condition: if you find out what priors everyone has, you should agree with your prior about how likely different worlds are. In other words, you should trust your prior in the future. Once you have this condition, it seems that it's impossible to both (a) believe that some other people's priors were generated in a way that makes them as likely to be good as yours and (b) have different priors from those people.

Let's dig into the sort of things this pre-rationality condition commits you to.

Consider the class of worlds where you are generated by a machine that randomly generates a prior and sticks it in your head. The pre-ration

...
Speculative Evopsych, Ep. 1
by Optimization Process — 2018-11-22T19:00:04.676Z

(cw death, religion, suicide, evolutionary psychology, shameless tongue-in-cheek meta-contrarianism)

I have a passing interest in biology, so I recently bought some fruit flies to run experiments on. I did two things to them. First, I bred them for intelligence. The details are kinda boring, so let’s fast-forward: after a few tens of millions of generations, they were respectably intelligent, with language and culture and technology so on.

In parallel with that, and more interestingly, whenever a fly was about to die of injury, I immediately plucked it out of the box, healed it, and put it in a different box (“Box Two”), a magnificent paradise where it blissfully lived out the rest of its days. Evolutionarily, of course, relocation was equivalent to death, and so the flies evolved to treat them the same: you could still make somebody stop affecting your world by stabbing them, and their kin would still grieve and seek revenge – the only difference was the lack of a corpse.

It didn’t really matter that the two boxes were separated only by a pane of glass, and that the flies in Box One could clearly see their “deceased” fellows living fantastic lives in Box Two. They “knew” on an abstract, intellectual level that getting “fatally” wounded wouldn’t actually make them stop having conscious experiences like death would. But evolution doesn’t care about that distinction; so it doesn’t select for organisms that care about that distinction; so the flies generally disregarded Box Two.

A small subculture in Box One claimed that “if anybody actually believed in Box Two and all its wonders, they’d stab themself through the heart in order to get there faster. Everybody’s literally-mortal fear of relocation proves that they don’t truly believe in Box Two, they only – at best – believe they believe.”

Strangely, nobody found this argument convincing.
If You Want to Win, Stop Conceding
by Davis_Kingsley — 2018-11-22T18:10:31.439Z

Author's note: This is the first in what I suppose might be a series of posts with respect to things I learned from playing traditional games competitively that I think might have broader applications. I write this as a private individual, not on behalf of CFAR or any other organization.

Traditional games - card games, board games, miniatures, etc. are a lot of fun, and I've played several of them at quite a competitive level. [1]

The #1 piece of advice that I can give if you want to get better at these games - a piece of advice that applies across essentially every game or sport I've played and a lot of "real world" stuff as well - is "if you want to win, stop conceding."

On the surface that doesn't sound super deep or interesting, but there's more to it than the obvious meaning - not all concedes are formal resignations, and indeed the ones that aren't are often more important.

Some time ago I read a book - either "The Inner Game of Tennis" or "Bonds that Make Us Free" or maybe both - that taught me that very many people concede games well before they need to be over, either because they incorrectly estimate their chances or because they make a mental motion away from trying to win and towards trying to make excuses for losing.

Here are some examples of what excuse-making thoughts might sound like:

"The dice are against me, there's nothing I can do."
"I didn't get a good night's sleep or eat breakfast this morning, otherwise I would be winning - I just can't focus."
"This guy's bad but he got lucky." [2]
"This guy's way better than me, I shouldn't even be matched up against him." [3]
"I don't know how she even got this far ahead, there must be some bug."
"This is pointless, why play it out?"
"This is just for fun anyway, I've basically lost, may as well wind things down."

Once you have moved from trying to win and towards trying to excuse losing, you have more or less already lost. Sometimes an opponent might snatch defeat from the jaws of victory, but that's rare. Most of the time, assuming you've lost is the same as actually losing, it just takes longer.

If you want to get better, stop it. Force yourself to keep playing and keep looking for outs. Learn to recognize these mental patterns and suppress them. Don't concede games unless there's nothing you can do anymore. Fight until the bitter end.

That's a real 'if', because if you don't want to get better I certainly don...