trentmkelly/LessWrong-43k huggingface dataset:
A Rationality Condition for CDT Is That It Equal EDT (Part 2)
by abramdemski — 2018-10-09T05:41:25.282Z

The previous post sketched an application of Jessica's COEDT framework to get rid of one of the assumptions of my argument for CDT=EDT. Looking at the remaining assumptions of my argument, the hardest one to swallow was implementability: the idea that when the agent implements a mixed strategy, its randomization is successfully controlling for any factors other than those involved in the decision to use that particular mixed strategy. Stated in bayes-net terms, the action has no parents other than the decision.

I stated that the justification of the assumption had to do with the learnability of the causal connections. I then went on to discuss some issues in learning counterfactuals, but not in a way which directly addressed the implementability assumption.

The present post discusses aspects of learning which are more relevant to the implementability assumption. Actually, though, these considerations are only arguments for implementability in that they're arguments for CDT=EDT. None of the arguments here are watertight.

Whereas the previous post was largely a response to COEDT, this post is more exclusively talking about CDT=EDT.

How Should We Learn Counterfactuals?

When Are Counterfactuals Reasonable?

How do we evaluate a proposed way to take logical counterfactuals? One reason progress in this area has been so slow is that it has been difficult to even state desirable properties of logical counterfactuals in a mathematically concrete way, aside from strong intuitions about how specific examples should go.

However, there is one desirable property which is very clear: counterfacting on what's true should result in what's true. You might get a strange alternative system of mathematics if you ask what things would be like if 2+2=3, but counterfacting on 2+2=4 just gets us mathematics as we know it.

When we consider situations where an agent fully understands what decision problem it's in (so we are assuming that its map corresponds perfectly to the territory), this means that counterfacting on the action which it really takes in that situation, the counterfactual should tell it the true consequences of that action. However, it is less clear how to apply the principle for learning agents.

In the purely Bayesian case, one might argue that the prior is the equivalent of the true circumstance or decision problem; the best an agent can do is to r...
Alignment Newsletter #27
by Rohin Shah — 2018-10-09T01:10:01.827Z

Dan Hendrycks has now joined, and will likely write summaries primarily on adversarial examples and robustness. As with Richard, his summaries are marked as such; I'm reviewing some of them now but expect to review less over time.

Highlights

80K podcast with Paul Christiano (Paul Christiano and Rob Wiblin): This is a mammoth 4-hour interview that covers a lot of ground. I'll try to state the main points without the supporting arguments in roughly chronological order, listen to the podcast for more.

- The problem of AI safety is that we don't know how to build AI that does what we want it to do. It arises primarily because each actor faces a tradeoff between AI systems being maximally effective at its task, and being robustly beneficial.

- AI safety has had much more attention in the last few years.

- Everyone agrees that we don't know how to build AI that does what we want, but disagrees on how hard the problem is, or how it should be framed.

- The best arguments against working on alignment are opportunity cost (eg. working on biosecurity instead) and that the problem might be very easy or impossible, but even then it seems like work would be valuable for getting information about how hard the problem actually is.

- It's not very important for the best AI safety team to work with the best ML team for the purpose of pursuing alignment research, but it is important for actually building powerful aligned AI.

- The variance in outcomes from AGI come primarily from uncertainty in how hard the technical problem is, how people behave about AGI, and then how good we are at technical safety research. The last one is easiest to push on.

- It seems useful to build organizations that can make commitments that are credible to outsiders. This would allow the top AI actors to jointly commit that they meet a particular bar for safety, though this would also require monitoring and enforcing to be effective, which is hard to do without leaking information.

- We should expect slow takeoff, as Paul defines it. (I'm ignoring a lot of detail here.)

- We should focus on short timelines because we have more leverage over them, but the analogous argument for focusing on fast takeoff is not as compelling.

- Paul places 15% probability on human labor being obselete in 10 years, and 35% on 20 years, but doesn't think he has done enough analysis that people shoul...
Things I Learned From Working With A Marketing Advisor
by sarahconstantin — 2018-10-09T00:10:01.320Z

Epistemic Status: Opinions stated without justification

I’ve been getting a bunch of advice and help at LRI from a marketing/strategy expert, and it’s been an education. She’s been great to work with — she kicks my ass, in a good way. Basically, she takes my writing, rips it apart, and helps me put it back together again, optimized to make the organization look better. Every kind of writing, from professional emails to website copy to grant proposals, gets a makeover. I’m thinking of the experience as something of an introduction to the conventions of business/promotional communication, which are very different from the kinds of writing norms I’m used to.

Here are some of the general patterns I’ve been learning about, stated in my own words (and maybe mangled a little in translation).

Discretization

“People hate reading,” she tells me.

Seriously? You’re going to rip up my nice, fluent, carefully-written essay explaining my rationale and replace it with a table?

Yes. Yes we are.

She’s not wrong, though. I’ve had the experience of meeting with executives after sending them a two-page document, worrying that I should have written something more comprehensive, and finding they didn’t even read the two-pager. I learn best through text, but clearly not everyone does. So promotional content needs to make allowances for the skimmers, the glancers, the reading-avoidant.

Hence: tables. Headers. Bolding key phrases. Bullet points. Pictures and graphs. Logos. And, of course, slide decks.

Layout matters. If you cross your eyes until the page turns blurry and don’t read anything, how does it look? Is it a wall of text? If so, you need to break it up.

The principle of discretization is things should be broken up into separate, distinctive, consistently labeled parts.

What things? Everything.

Your website has parts. Your five-year plan has parts. Your value proposition has parts.

LRI doesn’t have a “product”, but in companies that sell a product, your product has parts called “features.” Even when the “product” is sort of an abstract, general thing like “we produce written reports”, in order to make them legible as products, you have to have a list of distinct parts that each report contains.

Once you have parts, you need to get obsessive about matching and parallelism. Each part needs to have one, and only one, name, and you have to use the same name everywhere. If your organizat...
Stop buttons and causal graphs
by Donald Hobson — 2018-10-08T18:28:01.254Z

Edit: This work is known to be obsolete. The design will not make correct side bets on whether the button will be pressed.

Suppose that all problems relating to epistemic rationality have been solved. We create an AI with an epistemic module and an instrumental module. The epistemic module produces a causal graph like this one. (well a probability distribution over such graphs, but suppose only one has significant probability for now.)

Here blue rectangles are the actions the AI can take in the world, red ellipses represent real world events. The green diamond represents utility. Mathematically, these nodes are stochastic functions that take in several random variables inputs, and return an output random variable (RV). The rectangles are action nodes, so they just represent a function of the AI’s action. Each node is the situation at a single instant, if the AI has multiple opportunities like this one at successive times, the graph needs to be repeated. The AI’s instrumental rationality can work by argmaxAct(E(Utility|A))

Now lets add a stop button, carrying on from the broomstick in this example. https://intelligence.org/2016/12/28/ai-alignment-why-its-hard-and-where-to-start/

The AI has 3 actions available. It can

1)Press its own stop button. After this, it can’t do anything.

2)Break the stop button so the human can’t press it, then fill the cauldron

3)Fill cauldron

Unormal(act)={1 if cauldron full0 if cauldron empty

If the AI has a utility Unormal, then it breaks the stop button, so the human can’t stop it, then fill the bucket.

Ususpend(act)={1 if suspended0 otherwise

V={Ususpendif Button PressedUnormalif Button Not Pressed

When we change the utility to V this AI presses the stop button. The referenced work proposed adding another term to the utility function.

θ=maxactE[Unormal∣act]–maxactE[Ususpend∣act]

Vindifferent={Ususpend+θif Button PressedUnormalif Button Not Pressed

This is equivalent to the AI choosing argmaxAct(E(Unormal|A,Button Not Pressed))

This could be done in the causal graph by deleting all the inputs to the “Stop Button” node and setting that nodes value to definitely not pressed. This works, the AI chooses option 3) and fills the cauldron.

However suppose the situation was different. Here the “Circuit Breaker” is broken, and the AI has to fix it before the stop button will work. The AI now has a new option.

4)Fix...
A compendium of conundrums
by Richard_Ngo — 2018-10-08T14:20:01.178Z
Link: http://thinkingcomplete.blogspot.com/2018/09/a-compendium-of-conundrums.html

Logic puzzles

None of the puzzles below have trick answers - they can all be solved using logic and a bit of maths. Whenever a group of people need to achieve a task, assume they're allowed to confer and come up with a strategy beforehand. They're listed roughly in order of difficulty. Let me know of any other good ones you find!

Two ropes

I have two ropes which each, if lighted at one end, takes 1 hour to burn all the way to the other end. However, they burn at variable rates (e.g. the first might take 55 minutes to burn 1/4 of the way, then 5 minutes to burn all the rest; the second might be the opposite). How do I use them to time 45 minutes?

25 horses

I have 25 horses, and am trying to find the 3 fastest. I have no timer, but can race 5 at a time against each other; I know that a faster horse will always beat a slower horse. How many races do I need to find the 3 fastest, in order?

Monty hall problem (explanation taken from here)

The set of Monty Hall's game show Let's Make a Deal has three closed doors. Behind one of these doors is a car; behind the other two are goats. The contestant does not know where the car is, but Monty Hall does. The contestant picks a door and Monty opens one of the remaining doors, one he knows doesn't hide the car. If the contestant has already chosen the correct door, Monty is equally likely to open either of the two remaining doors. After Monty has shown a goat behind the door that he opens, the contestant is always given the option to switch doors. Is it advantageous to do so, or disadvantageous, or does it make no difference?

Four-way duel

A, B, C and D are in a duel. In turn (starting with A) they each choose one person to shoot at, until all but one have been eliminated. They hit their chosen target 0%, 33%, 66% and 100% of the time, respectively. A goes first, and of course misses. It's now B's turn. Who should B aim at, to maximise their probability of winning?

Duck in pond

A duck is in a circular pond with a menacing cat outside. The cat runs four times as fast as the duck can swim, and always runs around the edge of the pond in whichever direction will bring it closest to the duck, but cannot enter the water. As soon as the duck reaches the shore it can fly away, unless the cat is already right there. Can the duck escape?

Non-transitive dice

Say that a die A beats another die B if, when both rol...
Subjunctive Tenses Unnecessary for Rationalists?
by Sailor Vulcan — 2018-10-07T15:28:30.602Z

Do any Spanish speaking rationalists use subjunctive tenses? I've had the sense for a while that it's necessary for when you're expressing unfocused uncertainty or a strong emotional attachment to a belief or opinion, but that most of the time you can get by just fine with indicative tenses only. You can express uncertainty as a factual statement about how certain you are of a claim's accuracy. You can express strong emotions by making a factual statement about how you feel about something.

Then again, maybe it takes more words to do it that way. But surely emotions can be gleaned well enough from context and tone without using subjunctive tenses. Other languages such as English don't have any subjunctive tenses at all and that doesn't seem to cause any communication difficulties. (Although I could be wrong since English is my first langauge.)

Rationalists have better tools for focusing uncertainty and being more aware of one's emotions and learning to avoid investing one's emotions in their beliefs too much because there's always a chance they might be wrong. Basically I'm wondering if that causes Spanish speaking rationalists to reduce or eliminate their use of subjunctive tenses.

Thoughts?

Edit: someone pointed out that English does have subjunctive tenses and I just didn't notice it. Now I'm wondering if the use of subjunctive tenses is less clear cut than a quick google search can explain. I thought it was just for expressing emotions and uncertainty (which you can technically still do in indicative). But it might also be necessary for politeness or speaking courteously to others. ("Would you like something to drink?" is subjunctive right? And "Do you want something to drink" is indicative?)
Differential Progress and a Personal Commitment
by DragonGod — 2018-10-07T13:42:53.239Z

Regarding the problem of differential progress in AI capability research vs AI Alignment research.

I intend to work in Artificial Intelligence, but I would want to pursue capability research over alignment research, and due to the nature of incentives in the two fields so would most people. This means that safety research would lag behind capability research which is a lose lose situation for everyone as expecting the people who first develop Human Level Machine Intelligence (HLMI) to hold off on deployment until safety research catches up is optimistic at best and outright naive at worst.

To do my own part in mitigating this problem, I've decided to commit to donate 10% of my income towards AI Safety, and if I decide to work full time in Safety I would reduce this to 5%, whereas if I instead choose to pursue capability research I would increase this to 20% (modulo acceptable standard of living and quality of life). This incentivises me to contribute to Safety research (or at least not worsen the problem), and if I do end up exacerbating the problem, I'll pay for it accordingly.

This is my own personal solution, and I make no claim that others should do something similar (even though I would prefer that they do). I'm currently personally poor (I don't work and have an allowance of around $50+ a month (PayPal didn't let me donate cents, or at least I didn't see how to), but I'm dependent on my parents so I have an acceptable standard of living and quality of life). It is more important to me that I donate at all to AI Safety research, than that I donate to the most effective AI Safety research organisation, and to prevent myself from procastinating under the guise of finding the most effective organisation, I just decided to donate to MIRI. Making this commitment public is so that I'm less likely to weasel out of it, so I should be creating these posts around once a month, and if 60 days go by without me creating a post about my donation towards AI Safety, feel free to send me a friendly reminder. ;)
Open Source Issue Roundup
by Raemon — 2018-10-06T20:09:32.257Z

There are a few particular items in our Github issue tracker, that I thought would make for good first issues for people who haven't coded for LessWrong yet (or have only done so a bit).

(In general, you can find recommended issues by filtering by the "good first issue" tag)

Sharing a draft with someone should cause that post to appear in their drafts (on their user profile)
Downvoted posts should still appear on the author's Profile page
Messaging a person shouldn't create a new conversation if you already have an untitled conversation with that person (this may be slightly trickier as a first-issue but still pretty manageable)
Thinking of the days that are no more
by Richard_Ngo — 2018-10-06T17:00:01.208Z
Link: http://thinkingcomplete.blogspot.com/2017/12/thinking-of-days-that-are-no-more.html

In a previous post, I talked about some of the biases which skew the evaluations of our memories carried out by our "remembering selves". One domain in which these biases are particularly prevalent is romantic relationships. The most emotionally charged period of a relationship is usually the acrimony which accompanies its demise; the peak-end effect ensures that this negative affect is one of the main things we remember. Then there's duration neglect: our memories discount long periods of uneventful happiness compared with sudden changes. There's also a great deal of cognitive dissonance involved in reflecting on past relationships: we don't want to think that we were the reason a happy relationship failed, so it's easiest to conclude that it probably wasn't happy, and that this unhappiness was the other person's fault!

Lastly, there's a comparison effect, where we constantly hold in our minds certain ideals to which past relationships haven't measured up - in particular, the ideal of a lifelong commitment. There are certainly many practical reasons to favour long-term romantic commitments, for example the ability to raise children in a more stable environment. However, I think it is a mistake to view relationships as successful if they last "until death do us part", and unsuccessful if they don't (a mistake whose prevalence is due in no small part to the biases I described above). Rather, relationships can be positive experiences in the same way that holidays are: fun while they last, and memorable afterwards, without any need for expectations of permanence.
Of course there are many reasons to want a permanent relationship - to spend the rest of your life committed to another person. For a majority of people (probably including myself), those considerations will be overwhelming. But I'd be surprised if there were not plenty of people more suited to a series of medium-term relationships, each lasting perhaps half a dozen or a dozen years. Given the heights that divorce rates have soared to, it seems like many are already ending up in this scenario accidentally.
You might object that almost everyone could manage long-term relationships - because almost everyone managed it "back in the good old days" - and it's just modern life which is giving people shorter attention spans. Perhaps that is true. But under the heading of "moder

...
[link] "Paper strips" memory technique
by lincolnquirk — 2018-10-06T12:11:35.248Z

I was skimming reddit and someone named Yakari posted in response to “what are your best studying techniques?” They described a studying method where they overlay strips of paper vertically on top of their notes as they study them, periodically adding more strips to obscure more and more of the notes.

I haven’t tried it but it seems pretty genius — I expect it would be effective and easy
to implement, and I hadn’t heard of it before so figured you folks might not have either.

https://www.reddit.com/r/AskReddit/comments/9lskg2/serious_straight_a_students_of_reddit_what_is/e799ia2/
The Valley of Bad Theory
by johnswentworth — 2018-10-06T03:06:03.532Z

An interesting experiment

: researchers set up a wheel on a ramp with adjustable weights. Participants in the experiment then adjust the weights to try and make the wheel roll down the ramp as quickly as possible. The participants go one after the other, each with a limited number of attempts, each passing their data and/or theories to the next person in line, with the goal of maximizing the speed at the end. As information accumulates “from generation to generation”, wheel speeds improve.

This produced not just one but two interesting results.

First, after each participant’s turn, the researchers asked the participant to predict how fast various configurations would roll. Even though wheel speed increased from person to person, as data accumulated, their ability to predict how different configurations behave did not increase. In other words, performance was improving, but understanding was not.

Second, participants in some groups were allowed to pass along both data and theories to their successors, while participants in other groups were only allowed to pass along data. Turns out, the data-only groups performed better. Why? The authors answer:

... participants who inherited an inertia- or energy- related theory showed skewed understanding patterns. Inheriting an inertia-related theory increased their understanding of inertia, but decreased their understanding of energy…

And:

... participants’ understanding may also result from different exploration patterns. For instance, participants who received an inertia-related theory mainly produced balanced wheels (Fig. 3F), which could have prevented them from observing the effect of varying the position of the wheel’s center of mass.

So, two lessons:

Iterative optimization does not result in understanding, even if the optimization is successful.
Passing along theories can actually make both understanding and performance worse.

So… we should iteratively optimize and forget about theorizing? Fox not hedgehog, and all that?

Well… not necessarily. We’re talking here about a wheel, with weights on it, rolling down a ramp. Mathematically, this system just isn’t all that complicated. Anyone with an undergrad-level understanding of mechanics can just crank the math, in all of its glory. Take no shortcuts, double-check any approximations, do it right. It’d be tedious, but certainly not intractable. And then… then you’d understand the...