trentmkelly/LessWrong-43k huggingface dataset:
Who wants to be a Millionaire?
by Bucky — 2019-02-01T14:02:52.794Z

tldr; Using Toy models, the Kelly criterion, prospect theory, Bayes and amateur psychology in an unnecessarily detailed analysis of Who wants to be a Millionaire. I strongly suspect there will be errors but hopefully not too many.

Motivating example

I was watching Who wants to be a Millionaire? a couple of nights ago for the first time in about 20 years after its recent recommissioning in the UK.

One contestant’s general knowledge got her to £250,000, 2 correct answer away from £1,000,000. She had £125,000 guaranteed and still had 2 lifelines (ask the audience and phone a friend).

Question: The Norwegian explorer Roald Amundsen reached the South Pole on 14th December of which year?

A: 1891

B: 1901

C: 1911

D: 1921

She thought about/discussed this at length (there was no time limit and the total time spent on the question was about 20 minutes!). She knew that Amundsen beat Scott to the south pole and was confident that Scott was Victorian, which ruled out C & D. She pointed out that if it was 1911 then the 100 year anniversary would have been 2011 and she felt she would have remembered there being something about it in the news.

This didn’t help her choose between A & B so she asked the audience (she was confident none of her friends would know). The results were (from memory, maybe slightly off):

A: 28%

B: 48%

C: 24%

D: 0%

What would you do?

The result that stood out to me was the 24% who said C. After everything that she said about how confident she was that it isn’t 1911, who are these people voting C? It turns out they’re the people who knew the right answer.

Unfortunately, the contestant went with the majority, said B, and left with only £125k. Not too bad really, even if the name Roald Amundsen is haunting her and popping up everywhere she goes.

I’ll admit that even though I suspected that the answer was C based purely on the ask the audience result, I don’t think I would have been confident enough to go for it based only on that.

If I knew that b was the right answer, would I have been surprised that it got 48% of the vote? No, that would make sense.

If I knew that b was the wrong answer, would I have been surprised that it got 48% of the vote? Maybe a little, it is quite high, but not out of the question, especially this late in the game.

If I knew that c was the right answer, would I have been surprised that it got 24% of the vote? No, that would make sense; it’s ... ,
What is Wrong?
by Inyuki — 2019-02-01T12:02:13.023Z

I've always looked at LessWrong as a community that aims to reduce error of reasoning. However, the word "wrong" always seemed to have connotations with ethics and logic, less so goal pursuit. Something being "right" or "wrong" is generally thought of as a state of a logical proposition with respect to some logic axioms and ontological assumptions, rather than a pragmatic one.
However, it may be true that the axioms of logic that one believes are a result of one's interests, the observations about the world. For example, if one is interested in binary-tree-like understanding, one chooses to accept the law of excluded middle. If one is interested in understanding the universe through simulation, then one may choose to accept the axioms of constructive logic. If one is interested in disproving obviously true statements, one chooses Trump logic, and so on. It is pragmatic...
So, what do we do if I we to *explore* rather than to adhere to any predefined logic axioms? In general, if one has a goal Y, one searches for logic axioms X, that would help the reasoning to achieve Y. Therefore, with respect to an agent with a goal Y, the "wrong" is any X that does not minimize one's distance to Y, and being "Less Wrong" implies *not just* reducing cognitive or reasoning errors, but *generally* "optimizing" -- not just in the domain of logical functions, or ethical functions, but in general.
The answer as to what specific domain we have to optimize to be less wrong in general, has been elusive to me, but it seems that creation of new more capable media to transmit, hold, preserve, that let evolve and flourish all of our systems, minds and cultures, is the one domain with respect to which we should consider what is wrong or right.
So, when judging something (X) to be right or wrong, we should take a look, how does this affects world's

total information content

(Y).

Is AI wrong?

AI is a way to compress information by creating powerful models. Once model is built, information may be thrown away. When a social network like G+ acquires all the information that needed about the participants, then the company like Google learns it, and prunes it (closes the service). After all, if it figures out how you produce the content, you may not be valuable as information source anymore.

It may be wrong to concentrat...
Drexler on AI Risk
by PeterMcCluskey — 2019-02-01T05:11:01.008Z
Link: http://www.bayesianinvestor.com/blog/index.php/2019/01/30/drexler-on-ai-risk/

Eric Drexler has published a book-length paper on AI
risk, describing an approach that
he calls Comprehensive AI Services (CAIS).

His primary goal seems to be reframing AI risk discussions to use a
rather different paradigm than the one that Nick Bostrom and Eliezer
Yudkowsky have been promoting. (There isn't yet any paradigm that's
widely accepted, so this isn't a
Kuhnian paradigm shift;
it's better characterized as an amorphous field that is struggling to
establish its first paradigm). Dueling paradigms seems to be the best
that the AI safety field can manage to achieve for now.

I'll start by mentioning some important claims that Drexler doesn't
dispute:

an intelligence
explosion
might happen somewhat suddenly, in the fairly near future;

it's hard to reliably align an AI's values with human values;

recursive self-improvement, as imagined by Bostrom / Yudkowsky,
would pose significant dangers.

Drexler likely disagrees about some of the claims made by Bostrom /
Yudkowsky on those points, but he shares enough of their concerns about
them that those disagreements don't explain why Drexler approaches AI
safety differently. (Drexler is more cautious than most writers about
making any predictions concerning these three claims).

CAIS isn't a full solution to AI risks. Instead, it's better thought of
as an attempt to reduce the risk of world conquest by the first AGI that
reaches some threshold, preserve existing
corrigibility
somewhat past human-level AI, and postpone need for a permanent solution
until we have more intelligence.

Stop Anthropomorphising Intelligence!

What I see as the most important distinction between the CAIS paradigm
and the Bostrom / Yudkowsky paradigm is Drexler's objection to having
advanced AI be a unified, general-purpose agent.

Intelligence doesn't require a broad mind-like utility function.
Mindspace is a small subset of the space of intelligence.

Instead, Drexler suggests composing broad AI systems out of many,
diverse, narrower-purpose components. Normal software engineering
produces components with goals that are limited to a specific output.
Drexler claims there's no need to add world-oriented goals that would
cause a system to care about large parts of spacetime.

Systems built out of components with narrow goals don't need to develop
much broader goals. Existing trends in AI research suggest that
better-than-human intelligence can be ac

...
Why is this utilitarian calculus wrong? Or is it?
by EconomicModel — 2019-01-31T23:57:55.780Z

Suppose that I value a widget at $30. Suppose that the widget costs the widget-manufacturer $20 to produce, but, due to monopoly power on their part, they can charge $100 per widget.

The economic calculus for this problem is as follows. $30 (widget valuation) - $100 (widget price) = -$70 to me; $100 (widget price) - $20 (widget cost) = $80 to widget producers. $80 - $70 = +$10 total value. Ordinarily, this wouldn't imply that utilitarians are required to spend all their money on widgets because for a function to convert dollars to utils u($), u'($)>0, u''($)<0 and widget-producers usually have higher $ then widget consumers.

But suppose the widget monopolist is a poor worker commune. The profits go directly to the workers who, on average, have lower $ then I do. It seems like buying widgets would be more moral then, say, donating $80 to the same group of poor people ($80 - $80 = $0) because the widget purchase slightly compensates me for the donation in a way that is greater then the cost of the recipient to produce the widget.

And yet, I feel even less moral compunction to buy widgets then I do to donate $80 to GiveDirectly. Is this just an arbitrary, unjustifiable, subconscious desire to shove economic transactions into a separate domain from charitable donations or is there actually some mistake in the utilitarian logic here? If there isn't a mistake in the logic, is this something that the Open Philanthropy Project should be looking at?

[Question inspired by a similar question at the end of chapter 7 of Steven Landsburg's The Armchair Economist]
Small hope for less bias and more practability
by ArthurLidia — 2019-01-31T22:09:29.357Z

Unpractical approach of rationality demonstrated by the story of Bertrand Russell. Those who use the creative fire to achieve goals like: finding the truth. One who has superiority because he is rational and others are NPC, is not the right approach. It could be that the cause of unpractical approach in rationality. Of course if we all used the 2% of our existence (the whole feeling machine, physical machine, subjective experience, personal projects and consciousness ). The rational side would be more of that. But that wouldn't be a gain, but a loss. Like losing your feelings. Rationality never brings balance. As Logicomix (the best book truth adventure) may put it

”happy ending” for whom? Kurt Gödel starved himself. Bertrand Russell and Hilbert psychotic sons. And Cantor turning crazy.

These people later influenced Alan Turing (foundation of computer science, the cliche ending to the book). Perhaps rational communities, just like Logic served well the mathematics into not dispersing , and mathematics is the language of physics who supports science.

Pointed before by Paul Crowley. One can see that weakness that lesswrong was interest in psychology WEIRD samples Psychology is still in its alchemy like state, so is the era before the systematic bias.

In 2010, a group of researchers reported a systemic bias in psychology studies towards WEIRD ("western, educated, industrialized, rich and democratic") subjects.[221] Although only 1/8 people worldwide fall into the WEIRD classification, the researchers claimed that 60–90% of psychology studies are performed on WEIRD subjects. The article gave examples of results that differ significantly between WEIRD subjects and tribal cultures, including the Müller-Lyer illusion.

which is three years before lesswrong started to decay which was unfortunate. Perhaps the new era of better research in psychology (after 2010 )would give rise to “Less Wrong” cognitive biases.
Reliability amplification

by paulfchristiano — 2019-01-31T21:12:18.591Z

In a recent post I talked about capability amplification, a putative procedure that turns a large number of fast weak agents into a slower, stronger agent.

If we do this in a naive way, it will decrease reliability. For example, if…

Our weak policy fails with probability 1%.
In order to implement a strong policy we combine 10 decisions made by weak agents.
If any of these 10 decisions is bad, then so is the combination.

…then the combination will be bad with 10% probability.

Although the combination can be more powerful than any individual decision, in this case it is much less reliable. If we repeat policy amplification several times, our failure probability could quickly approach 1, even if it started out being exponentially small.

Complementary to capability amplification is reliability amplification: given a policy H that usually works, can we implement a policy H that works significantly more reliably?

To be slightly less imprecise (but still quite crude):

Given a distribution A over policies that ε-close to a benign policy for some ε ≪ 1, can we implement a distribution A⁺ over policies which is δ-close to a benign policy of similar capability, for some δ ≪ ε?

When I say that A is ε-close to a benign policy, I mean that there is a benign distribution over policies which has total variation distance at most ε from A. We can view this is as a simple mixture between a single aligned policy that is chosen with probability (1−ε) and an adversarial policy that is chosen with probability ε.

I will move freely between talking about “policies” and “distributions over policies;” we can turn a distribution into a single policy by averaging.

The notion of “implement” here is exactly the same as for capability amplification: we can use any number of implementations of A and whatever other resources are at our disposal, and must compute A⁺’s responses to a sequence of observations, within some time limit such as a day.

Reliability amplification via deliberation

I think there is one very natural way to do reliability amplification — if some variant of this idea doesn’t work, I’m not sure what else to try.

As a warm-up, let’s consider the setting where the agents want to make a single decision rather than a sequence of decisions — we’ll return to the sequential case soon.

We sample three agents A, B, and C from the given distribution. We ask them to talk for a while and then each pro...
The role of epistemic vs. aleatory uncertainty in quantifying AI-Xrisk
by David Scott Krueger (formerly: capybaralet) — 2019-01-31T06:13:35.321Z

(Optional) Background: what are epistemic/aleatory uncertainty?

Epistemic uncertainty is uncertainty about which model of a phenomenon is correct. It can be reduced by learning more about how things work. An example is distinguishing between a fair coin and a coin that lands heads 75% of the time; these correspond to two different models of reality, and you may have uncertainty over which of these models is correct.

Aleatory uncertainty can be thought of as "intrinsic" randomness, and as such is irreducible. An example is the randomness in the outcome of a fair coin flip.

In the context of machine learning, aleatoric randomness can be thought of as irreducible under the modelling assumptions you've made. It may be that there is no such thing as intrinsic randomness, and everything is deterministic, if you have the right model and enough information about the state of the world. But if you're restricting yourself to a simple class of models, there will still be many things that appear random (i.e. unpredictable) to your model.

Here's the paper that introduced me to these terms: https://arxiv.org/abs/1703.04977

Relevance for modelling AI-Xrisk

I've previously claimed something like "If running a single copy of a given AI system (let's call it SketchyBot) for 1 month has a 5% chance of destroying the world, then running it for 5 years has a 1 - .95**60 ~= ~95% chance of destroying the world". A similar argument applied to running many copies of SketchyBot in parallel. But I'm suddenly surprised that nobody has called me out on this (that I recall), because this reasoning is valid only if this 5% risk is an expression of only aleatoric uncertainty.

In fact, this "5% chance" is best understood as combining epistemic and aleatory uncertainty (by integrating over all possible models, according to their subjective probability).

Significantly, epistemic uncertainty doesn't have this compounding effect! For example, you could two models of how the world could work, one where we are lucky (L), and SketchyBot is completely safe, and another where we are unlucky (U), and running SketchyBot even for 1 day destroys the world (immediately). If you believe we have a 5% chance of being in world U and a 95% chance of being in world L, then you can roll the dice and run SketchyBot and not incur more than...
Applied Rationality podcast - feedback?
by Bae's Theorem — 2019-01-31T01:46:36.703Z

My goal is to help non-rationalists become rationalists, and provide exercises for existing rationalists to hone their skills, in the form of audio/video episodes.

'Rationally Speaking' and 'The Bayesian Conspiracy' are lovely, but they don't serve the function I have in mind.

Since I am an amature rationalist, and possibly the least intelligent person in this community, it would be ideal for someone else to run this (preferrably CFAR themselves, or at least someone who has attended).
If there is an existing project like this, please point me to it so I can support it.
If not, I at least have the drive to get the ball rolling.

I would probably start by using the 'Hammertime' sequence as material, and breaking it down a bit so the average layperson can understand it easier. This would require very little knowledge on my part, and give me time to maybe find a replacement host, cohost, or a queue of interviewees to assist in future episodes.

My question: is there any feedback you can give on this idea?
Wireheading is in the eye of the beholder
by Stuart_Armstrong — 2019-01-30T18:23:07.143Z

tl;dr: there is no natural category called "wireheading", only wireheading relative to some desired ideal goal.

Suppose that we have a built an AI, and have invited a human H to help test it. The human H is supposed to press a button B if the AI seems to be behaving well. The AI's reward is entirely determined by whether H presses B or not.

So the AI manipulates or tricks H into pressing B. A clear case of the AI wireheading itself.

Or is it? Suppose H was a meddlesome government inspector that we wanted to keep away from our research. Then we want H to press B, so we can get them our of our hair. In this case, the AI is behaving entirely in accordance with our preferences. There is no wireheading involved.

Same software, doing the same behaviour, and yet the first is wireheading and the second isn't. What gives?

Well, initially, it seemed that pressing the button was a proxy goal for our true goal, so manipulating H to press it was wireheading, since that wasn't what we intended. But in the second case, the proxy goal is the true goal, so maximising that proxy is not wireheading, it's efficiency. So it seems that the definition of wireheading is only relative to what we actually want to accomplish.

In other domains

I similarly have the feeling that wireheading-style failures in value-learning, low impact, and corrigibility, also depend on a specification of our values and preferences - or at least a partial specification. The more I dig into these areas, the more I'm convinced they require partial value specification in order to work - they are not fully value-agnostic.
Masculine Virtues
by Jacob Falkovich — 2019-01-30T16:03:56.000Z

Cross-posted from Putanumonit (where there's already a good discussion going).

Boys Will Be Boys

Have you seen the Gillette ad? Everyone’s seen the Gillette ad. And after my last post on masculinity, everyone’s been asking me what I think of the Gillette ad.

Well, I used to shave with Gillette and I’ve dumped them… back in 2014 when I realized that Dollar Shave Club sells basically the same razors for $1 each.

And the ad? Eh, it’s fine.

Cringeworthy at times, but fine.

Gillette is a division of a consumer products company selling bathroom items. No one is forced to watch their ads or use their razors. Clay Routledge put it brilliantly: we are living in an era of woke capitalism in which companies pretend to care about social justice to sell products to people who pretend to hate capitalism. Woke capitalism is silly but it gives Gillette customers what they want, which all you can expect of a corporation.

In contrast, APA is a professional organization of health care providers, writing guidelines for practicing therapists who deal with vulnerable men who come to them for help. The standards are quite different.

The content is quite different also.

Here is a list of things APA considers “harmful”, under the umbrella term of “traditional masculinity”:

Stoicism.
Competitiveness.
Aggression.
Dominance.
Anti-femininity.
Achievement.
Adventure and risk.
Violence.
Providing for loved ones (if you’re a black man).

Here’s a list of things the Gillette ad is against:

A mob chasing a teenager.
Texting someone “FREAK!!!”
Old TV shows.
Catcalling and butt-grabbing.
Patronizing your employees.
Six-year-olds fighting.
Chanting “boys will be boys” in unison.
Sexual assault and sexual harassment.

What do the two lists have in common? Violence, which is never the answer, is the only answer. Find the traditional man closest to you and ask them how many things on Gillette’s list they approve of; it’s not going to be many. “Traditional” men tend to complain they it’s no longer OK to hold doors open for women or take their kids hunting, not that in good ol’ days you could bully people over text or grope ladies on the street.

Here are the things Gillette is in favor of:

Terry Crews.
Accountability.
Demonstratively protecting women from other men.
Fatherhood.
Using your superior strength to break up fights between smaller males.
Teaching all of the above to your son.

Those are remarkably traditional male traits ...
Deconfusing Logical Counterfactuals
by Chris_Leong — 2019-01-30T15:13:41.436Z

This post aims to provide a strong philosophical foundation for logical counterfactuals, while sketching out an informal scheme that will hopefully be formalised further in the future. I believe that providing such a philosophical foundation is important for the same reasons that Sam listed in Motivating a Semantics of Logical Counterfactuals.

Introductory Material

Miri's Functional Decision Theory (FDT) paper defines subjunctive dependence to refer to the situation where two decision process involve the same calculation. This can result from causation, such as if I calculate an equation, then tell you the answer and we both implement a decision based on this sum. It can also occur non-causally, such as a prediction being linked to a decision as per Newcomb's problem. The problem of logical counterfactuals can be characterised as figuring out what processes subjunctively depend on other processes, so that we can apply FDT.
In the Co-operation Game, I argued that logical counterfactuals are more about your knowledge of the state of the world than the world itself. Suppose there are two people who can choose A or B. Suppose that a predictor knows that both people will choose A conditional on them being told one of the following two facts a) the other person will choose A b) the other person will choose the same as you. Then whether your decision is modelled to subjunctively depend on the other person depends on which of the two facts you are told. Going further than the original post, one might be told a) and the other b), so that the first sees themselves as not subjunctively linked to the second, while the second sees themselves as subjunctively linked to the first.
"Your knowledge of the state of the world" can be explicated as being about internally consistent counterfactuals, which we'll label Raw Counterfactuals. When there are multiple raw counterfactuals consistent with your state of knowledge, you can pick the one with the highest utility.
However, there will also be cases where only a single counterfactual is consistent with your state of knowledge, which results in a rather trivial problem. Consider for example Transparent Newcomb's Problem, where a perfect predictor places the million in a transparent box if and only if it predicts that you will one-box if it does. If you see the million, you know that you must have one-box

...