trentmkelly/LessWrong-43k huggingface dataset:
RFC: Mental phenomena in AGI alignment
by Gordon Seidoh Worley — 2018-07-05T20:52:00.267Z

I'm working on writing a paper about an idea I previously outlined for addressing false positives in AI alignment research. This is the second completed draft of one of the subsections arguing for the adoption of a particular, necessary hinge proposition to reason about aligned AGI (first subjection here). I appreciate feedback on this subsection especially regarding if you agree with the line of reasoning and if you think I've ignored anything important that should be addressed here. Thanks!

Since AGI alignment is necessarily alignment of AGI, alignment schemes can depend on the dispositions of AGI, and one disposition AGI has is to subjective experience and mental phenomena (Adeofe, 1997), (Nagel, 1974). Whether or not we expect AGI to realize this disposition matters because it influences the types of alignment schemes that can be considered since an AGI without a mental aspect can only be influenced by modifying its algorithms and manipulating its behavior whereas an AGI with a mind can be influenced by engaging with its perceptions and understanding of the world (Dreyfus, 1978). In other words we might say mindless AGI can be aligned only by algorithmic and behavioral methods whereas mindful AGI can also be aligned by philosophical methods that work on its epistemology, ontology, and axiology (Brentano, 1995). It's unclear what we should expect about the mentality of future AGI, though, because we are presently uncertainty about mental phenomena in general (cf. the work of Chalmers and Searle for modern, popular, and opposing views on the topic), so we are forced to speculate about mental phenomena in AGI when we reason about alignment (Chalmers, 1996), (Searle, 1984).

Note, though, that this uncertainty may not be fundamental (Dennett, 1991). For example, if materialist or functionalist attempts to explain mental phenomena prove adequate, perhaps because they lead to the development of conscious AGI, then we may agree on what mental phenomena are and how they work (Oizumi, Albantakis, and Tononi, 2014). If they don't, though, we'll likely be left with metaphysical uncertainty around mental phenomena that's rooted in the epistemic limitations of perception (Hussrl, 2014). Regardless of how uncertainty about mental phenomena might later be resolved, it currently creates a need for pragmatically making assumptions about it in ...
Debt is an Anti-investment
by Jacob Falkovich — 2018-07-05T20:00:38.267Z

Cross-posted from Putanumonit.

Since I wrote Get Rich Slowly I’ve received a steady stream of questions regarding personal finance. The most common of those is: should I prioritize investing or paying off my debts?

Get Rich Slowly wasn’t meant to break any new ground, just summarize some of the best advice online in a clear way for my readers. So, I thought I could just look up the existing best advice on debt vs. investing. I did, it sucks.

A lot of places tell you to invest if the return you’re expecting is higher than the interest on your debt, but that completely ignores risk. Taking a loan at 20% to invest in a highly speculative venture with expected returns of 20.1% isn’t smart investing, it’s a reckless gamble that’s likely to leave you bankrupt. The Balance, one of the most popular personal finance websites, mentions the importance of risk-adjustment but is too lazy to do the math explicitly. It also recommends maxing out your Roth IRA (expected return 6-7% with a fair bit of risk) before paying off credit cards (20%+ interest rate), which is utterly insane.

Risk adjustment is difficult and subjective, but there’s no escape from putting a number on it ourselves.

Anti-investment

I like to think of debt as an anti-investment. Let’s say you have a $10,000 loan which charges 5% interest, and you also have $10,000 invested with a risk-free after-tax return of 5%. The two would cancel each other out – your cash flow is the same as if you had neither one, namely zero. So, if you pay off the loan, you can think of it as gaining a risk-free investment with the same after-tax return as the loan’s interest rate.

What if instead of paying off the loan you invest the money? Then instead of the risk-free investment, you gain a different one. For example, if you invest the money in an S&P 500 US stock index, you will gain an investment that should return 7% (or 6% after paying capital gains tax), albeit with quite a bit of risk.

Thus, the question of paying off the debt becomes a comparison between a risk-free investment and your best alternative investment option. The question becomes: what risk-free rate of return is equivalent to your best available investment? If you pay a higher interest on your debt than that, you should pay it off. If the rate you’re paying is lower, you should invest. In our example, if you like holding a risky 6%-return stock fund about as mu...
On Authority
by quanticle — 2018-07-05T02:37:28.793Z
Link: https://www.interfluidity.com/v2/6970.html

Stephen Randy Waldman (Interfluidity) writes about the process by which authority is created. His definition of authority applies to information in a social context. Namely, information that is authoritative is information which we take to be true by our actions, regardless of whether that information is true or not.

The method by which human societies cause coordination to scale is by producing authoritative information. While a certain amount of coordination is possible via brute coercion, voluntary coordination is far cheaper and more scalable. In order to engender voluntary compliance, one needs authoritative information upon which to coordinate.

Small societies can easily produce enough authoritative information by choosing one of their members as a designated source of authoritative information, an authority. However, the problems of producing authoritative information grow non-linearly with scale. As a result, much of the social infrastructure we have today (courts, the press, the financial system) is dedicated to the production and (social) validation of authoritative information.

The construction of authority is necessary in order to constrain society enough to produce useful outcomes. The methods by which we construct this authority is among the most important social and technological problems we face.
Making a Difference Tempore: Insights from 'Reinforcement Learning: An Introduction'
by TurnTrout — 2018-07-05T00:34:59.249Z

The safety of artifi cial intelligence applications involving reinforcement learning is a topic that deserves careful attention.

Foreword

Let's get down to business.

Reinforcement Learning

1: Introduction

2: Multi-armed Bandits

Bandit basics, including nonstationarity, the value of optimism for incentivizing exploration, and upper-confidence-bound action selection.

Some explore/exploit results are relevant to daily life - I highly recommend reading Algorithms to Live By: The Computer Science of Human Decisions.

3: Finite Markov Decision Processes

The framework.

4: Dynamic Programming

Policy evaluation, policy improvement, policy iteration, value iteration, generalized policy iteration. What a nomenclative nightmare.

5: Monte Carlo Methods

Prediction, control, and importance sampling.

Importance Sampling

After gathering data with our behavior policy b, we then want to approximate the value function for the target policy π. In off-policy methods, the policy we use to gather the data is different from the one whose value vπ we're trying to learn; in other words, the distribution of states we sample is different. This gives us a skewed picture of vπ, so we must overcome this bias.

If b can take all of the actions that π can (i.e., ∀a,s:π(a|s)>0⟹b(a|s)>0), we can overcome by adjusting the return Gt of taking a series of actions At,…,AT−1 using the importance-sampling ratio ρt:T−1:=∏T−1k=tπ(Ak|Sk)b(Ak|Sk). This cleanly recovers vπ by the definition of expectation:

ρt:T−1Eb[Gt|St=s]=(T−1∏k=tπ(Ak|Sk)b(Ak|Sk))Eb[Gt|St=s]=Eπ[Gt|St=s]=vπ(s).

Then after observing some set of returns (where {Gt}t∈T(s) are the relevant returns for state s), we define the state's value as the average adjusted observed return

V(s):=∑t∈T(s)ρt:T(t)−1Gt|T(s)|.

However, the ρt:T(t)−1's can be arbitrarily large (suppose π(A|S)=.5 and b(A|S)=1×10−10; .51×10−10=.5×1010), so the variance of this estimator can get pretty big. To get an estimator whose variance converges to 0, try

V(s):=∑t∈T(s)ρt:T(t)−1Gt∑t∈T(s)ρt:T(t)−1.

Death and Discounting

Instead of viewing the discount factor γ as measuring how much we care about the future, think of it as encoding the probability we don't terminate on any given step. That is, we expect with 1−γ probability to die on the next turn, so we discount rewards accordingly.

This intuition hugs pre-theoretic understanding much more closely... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Some Remarks on the Nature of Political Conflict
by Paperclip Minimizer — 2018-07-04T12:31:07.840Z

[Part of the debate around: Conflict Vs. Mistake]

[Criticizing articles like: In defence of conflict theory, Conservatives As Moral Mutants (part of me feels like the link is self-trigger-warning, but I guess I will just warn you that this is not a clever attention-grabbing title, the link means exactly what it says and argues it at some length)]

[Related to: Knowing About Biases Can Hurt People, Would Your Real Preferences Please Stand Up?, The Cowpox of Doubt, Guided By The Beauty Of Our Weapons]

[Epistemic effort: I thought of this argument and was so pleased by my own cleverness that I decided to post it.]

[Note: I have a nagging feeling I’ve spent a thousand words spelling out something completely obvious. Still, I hope there’s value in actually spelling it out.]

There has been a flurry of discussion around the nature of political conflict in the rationality movement for the last five months, sparked by a blog post by Scott Alexander on his blog Slate Star Codex making a dichotomy between mistake theorists who think their political opponents are mistaken on factual policy questions and conflict theorists who think their political opponents are innately evil. There have been a lot of good articles on the subject on every side and on both the object-level and the meta-level (well, on both the meta-level and the meta-meta-level), but also many bad ones resting on mistakes (I know, I am showing my side here).

One class of pro-conflict-theory arguments that bother me a lot goes like this:

Mistake theory can't be the correct worldview because, for example, it's historically documented that tobacco companies hired scientists to spread misinformation about whether smoking causes cancer instead of thinking about it in a rational way.

Other historical case studies used include the rise of liberal democracy, the abolition of slavery, giving women the right to vote, the end of segregation, etc.

A scientific theory that is often used in this kind of argument is Jonathan Haidt's work on political psychology. Jonathan Haidt, with his co-conspirator Jesse Graham, created moral foundations theory, according to which morality is divided into five foundations:

Care: cherishing and protecting others; opposite of harm

Fairness or proportionality: rendering justice according to shared rules; opposite of cheating

Loyalty or ingroup: standing with your group, family, nation; oppos

...
The Learning-Theoretic AI Alignment Research Agenda
by Vanessa Kosoy — 2018-07-04T09:53:31.000Z

In this essay I will try to explain the overall structure and motivation of my AI alignment research agenda. The discussion is informal and no new theorems are proved here. The main features of my research agenda, as I explain them here, are

Viewing AI alignment theory as part of a general abstract theory of intelligence

Using desiderata and axiomatic definitions as starting points, rather than specific algorithms and constructions

Formulating alignment problems in the language of learning theory

Evaluating solutions by their formal mathematical properties, ultimately aiming at a quantitative theory of risk assessment

Relying on the mathematical intuition derived from learning theory to pave the way to solving philosophical questions

Philosophy

In this section I explain the key principles and assumptions that motivate my research agenda.

The importance of rigor

I believe that the solution to AI alignment must rely on a rigorous mathematical theory. The algorithms that comprise the solution must be justified by formal mathematical properties. All mathematical assumptions should be either proved or at least backed by considerable evidence, like the prominent conjectures of computational complexity theory. This needs to be the case because:

We might be facing one-shot success or failure. This means we will have little empirical backing for our assumptions.

To the extent we have or will have empirical evidence about AI, without a rigorous underlying theory it is very hard to know how scalable and transferable the conclusions are.

The enormity of the stakes demands designing a solution which is as reliable as possible, limited only by the time constraints imposed by competing unaligned projects.

That said, I do expect the ultimate solution to have aspects that are not entirely rigorous, specifically:

The quantitative risk analysis will probably rely on some parameters that will be very hard to determine from first principles, because of the involvement of humans and our physical universe in the equation. These parameters might be estimated through (i) study of the evolution of intelligence (ii) study of human brains, (iii) experiments with weak AI and its interaction with humans (iv) our understanding of physics. Nevertheless, we should demand the solution to be highly reliable even given cautious err

...
The Prediction Problem: A Variant on Newcomb's
by Chris_Leong — 2018-07-04T07:40:21.872Z

Update: If you are interested in understanding my thoughts on Newcomb's problem, I would recommend starting with Why 1-boxing doesn't imply backwards causation and then Deconfusing Logical Counterfactuals. The later doesn't quite represent my most recent views, but I still think it's worth reading.

I no longer endorse the Prediction Problem as being informative about which decision theory is better, but rather only as a useful intuition pump for why you should care about meta-theoretic uncertainty.

When trying to understand a problem, it is often helpful to reduce it to something simpler. Even if the problem seems as simple as possible, it may still be able to be simplified further. This post will demystify Newcomb's Problem by reducing it to the Prediction Problem, which works as follows:

Step 1: An AI called Alpha scans you and makes a prediction about what action you are going to take. A thousand people have already played this game and Alpha was correct every time.
Step 2: You pull either the left or the right lever. Neither does anything.
Step 3: Alpha gives you $1 million if it predicted that you'd pull the left lever, otherwise it gives you $1000.

The empirical answer seems to be that you ought to pull the left lever. On the other hand, some strictly following Causal Decision Theory ought to be indifferent to the two solutions. After all, the reasoning goes, Alpha has already made their prediction nothing you do now can change this.

At this point someone who thinks they are smarter than they actually are might decide that pulling the left lever may have an upside, but doesn't have a downside, so you may as well pull it and then go about their lives without thinking about this problem any more. This is the way to win if you were actually thrust into such a situation, but a losing strategy if your goal is to actually understand decision theory. I've argued before that practise problems don't need to be realistic, it's also fine if they are trivial. If we can answer why exactly you ought to pull the left lever, then we should also be able to justify one-boxing for Newcomb's Problem and also Timeless Decision Theory.

"Decision Theory" is misleading

The name "decision theory" seems to suggest a focus on making an optimal decision, which then causes the optimal outcome. For the Prediction Pro...
Intertheoretic utility comparison
by Stuart_Armstrong — 2018-07-03T13:44:18.498Z

This is presenting some old work on combining different possible utility functions, that is worth revealing to the world.

I've written before about the problem of reaching an agreement between agents with different utility functions. The problem re-appears if you yourself are uncertain between two different moral theories.

For example, suppose you gave 99% credence to average utilitarianism and 1% credence to total utilitarianism. In an otherwise empty universe, you can create one person with 2 utility, or a thousand with 1 utility.

If we naively computed the expected utility of both actions, we would get 0.99(2)+0.01(2)=2 for the first choice, and 0.99(1)+0.01(1000)=10.99 for the second. It therefore seems that total utilitarianism wins by default, even though it is very unlikely (for you).

But the situation can be worse. Suppose that there is a third option, which created ten thousand people with each 0.000001 utility. And you have 99% credence on average utilitarianism, (1−10−100)% credence on total utilitarianism, and (10−100)% credence on exponential utilitarianism, where the average utility is multiplied by two to the power of the population. In this case the third option - and the incredibly unlikely exponential utilitarianism - win out massively.

Normalising utilities

To prevent the large-population-loving utilities from winning out by default, it's clear we need to normalise the utilities in some way before adding them together, similarly to how you normalise the utilities of opposing agents.

I'll distinguish two methods here: individual normalisations, and collective normalisations. For individual normalisations, if you have credences of pi for utilities ui∈U, then ui is normalised into ^ui using some procedure that is independent of pi, pj, and uj for j≠i. Then the normalised utilities are added to give your total utility function of:

u=∑ipi^ui.

In collective normalisations, the normalisation of ui into ^ui is allowed to depend upon the other utilities and the credences. All Pareto outcomes for the utilities are equivalent (modulo resolving ties) with maximising such a u.

The Nash Bargaining Equilibrium and the Kalai-Smorodinsky Bargaining Solution are both collective normalisations; the Mutual Worth Bargaining Solution is an individual normalisation iff the choice of the default point is individual (but doing that violates the spirit of what...
What will we do with the free energy?
by ChristianKl — 2018-07-03T10:50:08.650Z

Epistemological status: Superficial personal speculation .

This posts follows What could be done with RNA and DNA sequencing that's 1000x cheaper than it's now? in thinking about technical challenges we will face in the next decades. It's an attempt by my to play a bit with futurism.

Both solar and wind energy have exponential efficiency improvements:

A majority of Indian new coal power now costs more then new wind and solar power. This means that we can expect a lot of new wind and solar power installations to be build.

Last year, in 2017, in Germany there were 146 hours in which energy prices were negative. This means that during those hours wholesale customers got payed to consume energy.

Unfortunately, batteries are expensive and while it might be economical to load a battery in the day and release the energy at night it won't be economical to load the battery in summer and release it in winter.

As a result batteries don't work as energy sink for summer days with too much energy production.

Maybe, we can construct our wind turbines in a way that we can dumb excess energy into them when there isn't wind? It would however be more desirable to find a way to actually spend that free energy productively.

Most machines that we currently build are optimized to run as much as possible and thus don't provide good energy sinks. A machine that produces spoons is most economical when it runs 24/7.

This means that there's an economic opportunity to build machines that can work as energy sinks for days with negative energy costs.

It's likely a good idea to build data-centers in a way that they are not only able to run in a low energy mode but also in a mode where they burn as much energy as possible. As a result it might b worthwhile for some cloud provider to have in a year specific times where the compute is radically cheaper than on other days. Maybe that compute will go into crypto-mining but it we can also think about other uses for it.

The data-centers likely won't be enough of a data sink and we need new ideas about what to do with other excess energy.

Do you have any energy about how to build good energy sinks that might use energy usefully and that are economical if they are only used for a few weeks/days a year?
Letting Go II: Understanding is Key
by johnswentworth — 2018-07-03T04:08:44.638Z

Context: The previous post gave a long list of examples of just-let-go design: a problem-solving approach/aesthetic based on giving up direct control over the system. In this post, we’ll talk about the first of several reasons why we'd want use just-let-go design, as a strategy in itself.

Compared to railroading-style direct control, just-let-go solutions generally require a deeper understanding of the problem.

If we want to build a platform to support all kinds of different programs, or a sandbox to support many different kinds of players, then we need to understand our players/programmers and their goals in some detail. By comparison, railroading players requires little understanding of the players themselves.

The economic examples make the difference even sharper. Anyone can pass a law which makes it illegal for car-sellers to lie about their car. Building a system in which sellers are no longer incentivized to lie about their car requires more thought.

The car-seller example brings up another point. Suppose we do make it illegal for car-sellers to lie about their car. What will happen? Well, people will probably just work around it. Without modifying the incentives, we haven’t fixed the root cause, and the problem probably won’t go away. The point: direct control solutions don’t actually remove the need for understanding the problem. Using direct control without understanding the problem will usually just break things.

Looking back over the examples makes this clear. A game railroads the players too much? The players get angry and leave. An app railroads the users too much? The users get bored and drop off. Price controls? Classic example of unintended side effects. Poorly thought out incentives? Another classic example of unintended side effects. Even when writing code, programmers are advised to trust the compiler for basic optimization, rather than try to do it themselves.

Just-let-go solutions, on the other hand, are hard to even think about without understanding the problem. We want to lower prices without instituting price controls? We’re immediately forced to think about the root causes of high prices. Now, that doesn’t mean our understanding will be correct. We could still be wrong. But at the very least, we are definitely forced to think about the problem.

That’s one major advantage of just-let-go solutions: they don’t give us a choice in whether...