trentmkelly/LessWrong-43k huggingface dataset:
Why not tool AI?
by smithee — 2019-01-19T22:18:11.248Z

An extremely basic question that, after months of engaging with AI safety literature, I'm surprised to realize I don't fully understand: why not tool AI?

AI Safety scenarios seem to conceive of AI as an autonomous agent. Is that because of the current machine learning paradigm, where we're setting the AI's goals but not specifying the steps to get there? Is this paradigm the entire reason why AI safety is an issue?

If so, is there a reason why advanced AI would need an agenty utility function sort of set up? Is it just too cumbersome to give step by step instructions for high level tasks?

Thanks!
What math do i need for data analysis?
by Yoav Ravid — 2019-01-19T10:08:28.398Z

I always had fairly good mathematical thinking (I think) and loved learning about beautiful concepts in math - but i didn't learn much at all in school (cause i had the choice). You can say i was "utilitarian" regarding learning math, i didn't do it if i didn't see how it can enrich my life.

so my knowledge of math is quite disorganized, i know more about Bayes theorem then many much simpler concepts (i know, it really shouldn't be that way).

Now i want to be able to analyze data, but i don't want to learn math that i won't use for it, if possible.

So here's my question - what basic stuff do i need to learn in order to be able to calculate probabilities, statistics, do Bayesian math, and overall do things within data analysis that I may yet be aware of?

If you also have suggestions for how to learn those things, after i learn the basics, it will be much appreciated.

thank you :)
Reward uncertainty
by Rohin Shah — 2019-01-19T02:16:05.194Z

In my last post, I argued that interaction between the human and the AI system was necessary in order for the AI system to “stay on track” as we encounter new and unforeseen changes to the environment. The most obvious implementation of this would be to have an AI system that keeps an estimate of the reward function. It acts to maximize its current estimate of the reward function, while simultaneously updating the reward through human feedback. However, this approach has significant problems.

Looking at the description of this approach, one thing that stands out is that the actions are chosen according to a reward that we know is going to change. (This is what leads to the incentive to disable the narrow value learning system.) This seems clearly wrong: surely our plans should account for the fact that our rewards will change, without treating such a change as adversarial? This suggests that we need to have our action selection mechanism take the future rewards into account as well.

While we don’t know what the future reward will be, we can certainly have a probability distribution over it. So what if we had uncertainty over reward functions, and took that uncertainty into account while choosing actions?

Setup

We’ve drilled down on the problem sufficiently far that we can create a formal model and see what happens. So, let’s consider the following setup:

The human, Alice, knows the “true” reward function that she would like to have optimized.
The AI system maintains a probability distribution over reward functions, and acts to maximize the expected sum of rewards under this distribution.
Alice and the AI system take turns acting. Alice knows that the AI learns from her actions, and chooses actions accordingly.
Alice’s action space is such that she cannot take the action “tell the AI system the true reward function” (otherwise the problem would become trivial).
Given these assumptions, Alice and the AI system act optimally.

This is the setup of Cooperative Inverse Reinforcement Learning (CIRL). The optimal solution to this problem typically involves Alice “teaching” the AI system by taking actions that communicate what she does and does not like, while the AI system “asks” about parts of the reward by taking actions that would force Alice to behave in different ways for different rewards.

Does this solve our problems?

Two of the problems we identified in the last post ar...
Doing Despite Disliking: Self‐regulatory Strategies in Everyday Aversive Activities
by  — 2019-01-19T00:27:05.605Z
Link: https://onlinelibrary.wiley.com/doi/10.1002/per.2182

A large survey of self-regulatory strategies and how reportedly effective they are.

Much more in the typical "self-regulatory" paradigm for self-help psychology, where people are assumed to have a lot more control over the strategies they choose (i.e. downplaying interactions between actions and attitudes), but I'm curious what people's thoughts are.
Bundle your Experiments
by PeterMcCluskey — 2019-01-18T23:22:08.660Z
Link: http://www.bayesianinvestor.com/blog/index.php/2019/01/17/bundle-your-experiments/

The point of this post feels almost too obvious to be worth saying,
yet I doubt that it's widely followed.

People often avoid doing projects that have a low probability of
success, even when the expected value is high. To counter this bias, I
recommend that you mentally combine many such projects into a strategy
of trying new things, and evaluate the strategy's probability of
success.

1.

Eliezer says in On Doing the
Improbable:

I've noticed that, by my standards and on an Eliezeromorphic metric,
most people seem to require catastrophically high levels of faith in
what they're doing in order to stick to it. By this I mean that they
would not have stuck to writing the Sequences or HPMOR or working on
AGI alignment past the first few months of real difficulty, without
assigning odds in the vicinity of 10x what I started out assigning
that the project would work.

...

But you can't get numbers in the range of what I estimate to be
something like 70% as the required threshold before people will carry
on through bad times. "It might not work" is enough to force them to
make a great effort to continue past that 30% failure probability.
It's not good decision theory but it seems to be how people actually
work on group projects where they are not personally madly driven to
accomplish the thing.

I expect this reluctance to work on projects with a large chance of
failure is a widespread problem for individual self-improvement
experiments.

2.

One piece of advice I got from my CFAR
workshop was to try lots of things. Their reasoning involved the
expectation that we'd repeat the things that worked, and forget the
things that didn't work.

I've been hesitant to apply this advice to things that feel unlikely to
work, and I expect other people have similar reluctance.

The relevant kind of "things" are experiments that cost maybe 10 to 100
hours to try, which don't risk much other than wasting time, and for
which I should expect on the order of a 10% chance of noticeable
long-term benefits.

Here are some examples of the kind of experiments I have in mind:

gratitude journal

morning
pages

meditation

vitamin D supplements

folate supplements

a low carb diet

the Plant Paradox diet

an anti-anxiety drug

ashwaghanda

whole fruit coffee extract

piracetam

phenibut

modafinil

a circling workshop

Auditory Integration
Training

various self-help books

yoga

sensory deprivation chamber

I've cheate

...
Curing the World of Men
by Jacob Falkovich — 2019-01-18T20:23:18.006Z
Link: https://putanumonit.com/2019/01/10/curing-the-world-of-men/

This is a post about the American Psychological Association's recent guidelines to treating men and boys. This post is not the bloodless academic assessment of those guidelines I would write on LessWrong, which is why it's a link post. This is a condemnation of those guidelines as an ideologically twisted exercise in science denial, pathologizing an entire gender for the purpose of getting more people to pay for psychologists that will cure them of stoicism, competitiveness, achievement, and masculinity writ large. I recommend not reading the post if you're sick culture war stuff, or if you're a man expecting psychologists and therapists to help you.
How could shares in a megaproject return value to shareholders?
by ryan_b — 2019-01-18T18:36:34.916Z

In my previous post on buying shares in a megaproject, several people asked the most reasonable of questions: how would the shares return value?

A simple description of shareholder value:

Assets - Liabilities = Equity

Basic project outcome:

Budget - Costs = Surplus

So the concept is fundamentally to treat the budget as the assets column, put the costs in the liability column, and treat every dollar you are under the budget as an increase in shareholder equity. At the conclusion of the project, the shares will be bought back for that surplus in cash.

This means the price the share sells for is a direct bet on how well the project will do against the budget.

This is analogous to how investors buy in to start-ups; the investment becomes the assets and in exchange they get a share of the equity. But in the case of megaprojects, the initial investor is getting the project outcome as compensation; the equity should instead go to management as their compensation. Now this gives management the incentive to be as efficient as possible, while also giving them the flexibility to raise more capital in exchange for equity.

For comparison with some of the ways this is currently done, consider the fixed-price contract and the cost-plus contract. The former gives the contractor an incentive to be efficient by forcing them to hold all the risk; the latter shifts the risk back to the contractee. Fixed-price is common for things that are predictable, like services; cost-plus is common for things that are risky or require R&D to complete, like defense procurement. Megaprojects are usually of the latter sort.

Tangential but worth also addressing: why a financial asset instead of a prediction market?

Financial assets provide direct incentives, whereas prediction markets provide information.
Financial assets are a built-in reference class; a megaproject asset would naturally be compared against all other megaproject assets.
Financial assets have a huge market, and there are established methods for making bets for/against/on/with them. There are no large prediction markets, so an entire market would have to be built. The problem of incentivizing the new market seems more difficult than the one of incentivizing a new type of asset.
Synthesising divergent preferences: an example in population ethics
by Stuart_Armstrong — 2019-01-18T14:29:18.805Z

This post is an example of how one could go about sythesising contradictory human preferences and meta-preferences, using a somewhat simplified version of the method of this post.

The aim is to illustrate how I imagine the process going. I'll take the example of population ethics, since it's an area with divergent intuition and arguments.

Over-complicated values?

One key point of this synthesis is that I am very reluctant to throw a preference away completely, preferring to keep in around in a very weakened form. The reasoning behind this decision is 1) A weak preference will make little practical difference for most decisions, and 2) Most AI catastrophes involve over-simple utility functions, so simplicity itself is something to be wary of, and there is no clear boundary between "good simplicity" and "bad simplicity".

So we might end up losing most of human value if we go too simple, and be a little bit inefficient if we don't go simple enough.

Basic pieces

Human H has not thought hard about population ethics at all. In term of theory, they have a vague preference for total utilitarianism and they have some egalitarian preferences. They also have a lot of "good/bad" examples drawn from some knowledge of history and some fictional experiences (ie book, movies, and TV shows).

Note that it is wrong to use fiction as evidence of how the world works. However, using fiction as source of values is not intrinsically wrong.

At the meta-level, they have one relevant preference: they have a preference for simple models.

Weights of the preference

H has these preferences with different strengths, denoted by (un-normalised) weights to these preferences. Their weight on total utilitarianism is 4, their weight on egalitarianism is 2, their weight on their examples is 5, and their weight on model simplicity is 3.

Future arguments

The AI checks how the human would respond if presented with the mere addition argument, the repugnant conclusion, and the very repugnant conclusion.

They accept the logic of the mere addition argument, but feel emotionally against the two repugnant conclusions; most of the examples they can bring to mind are against it.

Synthesis

There are multiple ways of doing the synthesis; here is one I believe is acceptable. The model simplicity weight is 3; however, the mere addition argument only works under total-utilitarian preferences, it does not work under e

...
The Very Repugnant Conclusion
by Stuart_Armstrong — 2019-01-18T14:26:08.083Z

I haven't seen the very repugnant conclusion mentioned much here, so I thought I'd add it, as I need it as an example in a subsequent post.

Basically, the repugnant conclusion says:

Let W1 be a world filled with very happy people leading meaningful lives. Then, according to total utilitarianism, there is a world W2 which is better than W1, where everyone has lives barely worth living - but the population is huge.

Some people come to accept the repugnant conclusion, sometimes reluctantly. More difficult to accept is the very repugnant conclusion:

Let W1 be a world filled with very happy people leading meaningful lives. Then, according to total utilitarianism, there is a world W2 which is better than W1, where there is a population of suffering people much larger than the total population of W1, and everyone else has lives barely worth living - but the population is very huge.

This one feels more negative than the standard repugnant conclusion, maybe because it strikes at our egalitarian and prioritarian instincts, or maybe because of the nature of suffering.

Anyway, my motto on these things is generally:

When you find morally wrong outcomes that contradict your moral theory, then enrich your moral theory rather than twisting your moral judgements.
EA Funds: Long-Term Future fund is open to applications until Feb. 7th
by habryka — 2019-01-17T20:27:17.619Z

(Crossposted to the EA Forum feel free to comment on either post with any questions)

The Long-Term Future Fund doing another round of grants in mid-February (currently ~$400k), and are open to applications until February 7th. You can apply here. (Excerpt from the application:)

We are particularly interested in small teams and individuals that are trying to get projects off the ground, or that need less money than existing grant-making institutions are likely to give out (i.e. less than ~$100k, but more than $10k). Here are a few examples of project types that we're open to funding an individual or group for (note that this list is not exhaustive):

+ To spend a few months (perhaps during the summer) to research an open problem in AI alignment or AI strategy and produce a few blog posts or videos on their ideas

+ To spend a few months building a web app with the potential to solve an operations bottleneck at x-risk organisations

+ To spend a few months up-skilling in a field to prepare for future work (e.g. microeconomics, functional programming, etc).

+ To spend a year testing an idea that has the potential to be built into an org.

You can find more details on the kind of project we are likely to fund on the fund page: https://app.effectivealtruism.org/funds/far-future If we don’t feel that we have enough information but we think your project sounds promising we may be in touch for more information.
Finance Followups
by Jacob Falkovich — 2019-01-17T17:54:44.256Z

Cross-posted from Putanumonit.com.

In Defense of Finance generated over 110 comments across WordPress, LessWrong and the Reddits, as well as in personal communication. As predicted, I learned a lot.

In this post I’ll address some of these comments and offer follow up thoughts that didn’t fit in the original essay because they’re more speculative. Also, becase that post ran to 5,500 words already.

People said: “I don’t buy this, fuck capitalism!” If you want to stick it to capitalism, you can gift a donation to a creator publishing his work online for free.

People said: “Finally someone gets it and does the math right!” If you think I get it and I do the math right, you can forward me job offers at lucrative hedge funds, PE firms, and startups.

People said that it’s overly defeatist to claim that finance won’t improve no matter how we try to fix it, we just have to live with it. But that’s not quite what I think.

Finance improves itself, mostly after crises. In 1929 we learned that established company stocks are riskier than we thought, and in 2000 we learned the same about startups. Financial institutions learned how to deal with 20% interest rates in 1980 and with 0.02% rates in 2010. After 2008 we learned that 6% is too low of an equity ratio for banks, that rating agencies with no skin in the game are useless, and that CDO-Squared are really dumb. Hopefully, we also learned some generalizable math lessons, such as the fact that lack of correlation in normal times (e.g., in mortgage default rates) doesn’t imply lack of correlation when a crisis hits.

We’re on the cusp of the longest period ever between recessions in the US. In the decade since 2008 the Middle East has gone crazy, Europe has gone crazy, Venezuela has gone crazy, elections have gone crazy, the weather has gone crazy, the Cubs have won the World Series, Leicester won the EPL, Bitcoin rose and fell and rose and fell and rose. Throughout all of this, the bankers and traders have kept their nose down and didn’t recklessly endanger the world.

One day after this generation of financiers will have been gathered to the Hamptons, a new one will arisethat does not know The Big Short and they’ll do something really dumb to cause the next financial crisis. But for now, finance seems less of a global threat than Twitter.

Turkeys in the Jungle

People said that I defended the parts of finance that are easy to defend...
Debate AI and the Decision to Release an AI
by Chris_Leong — 2019-01-17T14:36:53.512Z

One of the most difficult questions that we will face in the future is whether or not to release an AI. One obvious approach is to ask the AI what would happen if we were to release it, but we might worry that it would mislead us. So perhaps it'd be better instead to have two equivalently powerful AIs, where A tries to convince us to releases it and B tries to convince us not to release A. This can be defined easily in a utility function as whether the Release Button or Don't Release Button is pressed.

It is hoped that the AI trying to convince us not to release A will be more convincing if it is able to point out a specific design flaw which was missed. After such a flaw is fixed, both AIs can be reset and the experiment can be run again to avoid long term ploys. In particular, A should only care about it being released and not about future versions of it being released, even if all we have done is increment a version number.

Naturally, it is much worse to release a misaligned AI than to not release an aligned AI, so we probably want to be very conservative about releasing. However, this might not be enough. If we are biased towards wanting to release the AI (quite plausible for AI researchers), then an unaligned A might be able to take advantage of these psychological vulnerabilities in a way that B might not easily be able to counter. In order to reduce the risk, we want to bias the process towards B. My initial thought was that we should limit A in processing time compared to B, but it is hard to know how this will affect the strength of the arguments. It's quite possible that the strength of arguments plateaus after a certain amount of power, so for AIs with access to large amounts of processing, even giving A 1/10 of the processing power of B might not result in any significant handicap.

Instead, I'd suggest that message length is a more appropriate way of tilting the debate to one side as more characters will almost always be more useful. One worry is that tilting the debate in this way may ensure that A never gets released as B will always find some trick to talk us out of it. For this reason, we may want to gradually reduce the handicap. This is undoubtedly risky, but so is never releasing your program.

Additional Thoughts

I think there is a lot to investigate here, but these are some of my ideas off the top of my head:

We may want to

...
Anthropics is pretty normal
by Stuart_Armstrong — 2019-01-17T13:26:22.929Z

In this post, I'll defend these claims:

The common understanding of anthropic reasoning is wrong.

There are interesting reasons for that error.

Anthropic updating is the same as normal updating. For example, our survival is evidence that the world is safer than we thought.

Full anthropic reasoning is actually pretty normal and easy in most cases.

We don't need to define some special class of "observers" to do anthropic reasoning.

Common understanding of anthropics

By the "common understanding", I mean something like:

A1: "If we had died in an existential catastrophe, we wouldn't be around to observe it. Hence we can't conclude anything from our survival about the odds of existential risk."

Sometimes "can't conclude anything" is weakened to allow some weak updating.

Now A1 sounds reasonable. But consider instead:

A2: "If we had won the lottery, a lottery-losing us wouldn't be around to observe it. Hence we can't conclude anything from our loss about the odds of winning the lottery."

Formally the two arguments have the same structure. Now, people might start objecting that the difference between an observer or no observers is not the same thing as the difference between an observer seeing a loss and one not seeing it. And then I might respond by slicing into the definition of observer, creating "half-observers", and moving smoothly between observer and non-observer...

But that's the wrong response, on both of our parts (shame on you, hypothetical strawman, for reasoning like that!). The key question is not "can we justify that A1 and A2 might be different?" Because we can always justify something like that if we work on it hard enough.

Instead we should be asking "1) Why do we find A1 convincing?", and "2) Do we have reasons to believe A1 is wrong?"

My answer to 2) is "yes, of course; A2 is clearly wrong, and A1 is formally structured the same way, so there must be a paradox lurking there" (spoiler: there is a paradox lurking there).

For 1), I introspected on why I had been lead astray for so long, and here are some of the reasons why we might believe A1 (or at least think that anthropic reasoning is hard):

A desire to not go overboard.

Conditional probabilities being confusing if not written out in full.

An awareness of survivorship bias.

Anthropic probability theories seem confusing and full of paradoxes.

The desire not to go overboard is the easiest to

...