trentmkelly/LessWrong-43k huggingface dataset:
Biological humans and the rising tide of AI
by cousin_it — 2018-01-29T16:04:54.749Z

The Hanson-Yudkowsky AI-Foom Debate focused on whether AI progress is winner-take-all. But even if it isn't, humans might still fare badly.

Suppose Robin is right. Instead of one basement project going foom, AI progresses slowly as many organizations share ideas with each other, leading to peaceful economic growth worldwide - a rising tide of AI. (I'm including uploads in that.)

With time, keeping biological humans alive will become a less and less profitable use of resources compared to other uses. Robin says humans can still thrive by owning a lot of resources, as long as property rights prevent AIs from taking resources by force.

But how long is that? Recall the displacement of nomadic civilizations by farming ones (which happened by force, not farmers buying land from nomads) or enclosure in England (which also happened by force). When potential gains in efficiency become large enough, property rights get trampled.

Robin argues that it won't happen, because it would lead to a slippery slope of AIs fighting each other for resources. But the potential gains from that are smaller, like a landowner trying to use enclosure on another landowner. And most such gains can be achieved by AIs sharing improvements, which is impossible with humans. So AIs won't be worried about that slippery slope, and will happily take our resources by force.

Maybe humans owning resources could upload themselves and live off rent, instead of staying biological? But even uploaded humans might be very inefficient users of resources (e.g. due to having too many neurons) compared to optimized AIs, so the result is the same.

Instead of hoping that institutions like property rights will protect us, we should assume that everything about the future, including institutions, will be determined by the values of AIs. To achieve our values, working on AI alignment is necessary, whether we face a "basement foom" or "rising tide" scenario.
"Taking AI Risk Seriously" (thoughts by Critch)
by Raemon — 2018-01-29T09:27:03.941Z

Note: Serious discussion of end-of-the-world and what to do given limited info. Scrupulosity triggers, etc.

Epistemic Status: The first half of this post is summarizing my own views. I think I phrase each sentence about as strongly as I feel it. (When I include a caveat, please take the caveat seriously)

Much of this blogpost is summarizing opinions of Andrew Critch, founder of the Berkeley Existential Risk Initiative, who said them confidently. Some of them I feel comfortable defending explicitly, others I think are important to think seriously about but don’t necessarily endorse myself.

Critch is pretty busy these days and probably won’t have time to clarify things, so I’m trying to err on the side of presenting his opinions cautiously.

Table of Contents:

Core Claims
My Rough AGI Timelines
Conversations with Critch

Hierarchies
Deep Thinking
Turning Money into Time
Things worth learning
Planning, Thinking, Feedback, Doing

A Final Note on Marathons

Summary of Claims

Between 2016 and 2018, my AI timelines have gone from "I dunno, anywhere from 25-100 years in the future?" to "ten years seems plausible, and twenty years seems quite possible, and thirty years seems quite likely."

My exact thoughts are still imprecise, but I feel confident that:

Claim 1: Whatever your estimates two years ago for AGI timelines, they should probably be shorter and more explicit this year.

Claim 2: Relatedly, if you’ve been waiting for concrete things to happen for you to get worried enough to take AGI x-risk seriously, that time has come. Whatever your timelines currently are, they should probably be influencing your decisions in ways more specific than periodically saying “Well, this sounds concerning.”

Claim 3: Donating money is helpful (I endorse Zvi’s endorsement of MIRI and I think Lark’s survey of the overall organizational landscape is great), but honestly, we really need people who are invested in becoming useful for making sure the future is okay.

What this might mean depends on where you are. It took me 5 years to transition into being the sort of person able to consider this seriously. It was very important, for the first couple of those years, for me to be able to think about the questions without pressure or a sense of obligation.

I still don’t see any of this as an obligation - just as the obviously-right-...
Write
by alkjash — 2018-01-29T02:11:38.919Z
Link: https://radimentary.wordpress.com/2018/01/28/write/

This post is Part 5 of the sequence on Babble. After writing Hammers and Nails, I figured out that my favorite Hammer is writing.

Write about everything. Write it immediately. Edit afterwards.

In this post, I tell two stories about the magic of putting words on paper. Make your own conclusions about the brain-enhancing effects of writing.

Then, I share my recursive blogging process which avoids the ickiness of planning and gets directly to the thick of things. The recursion makes it easy to expand sections without breaking into structureless rambles. My process is mostly inspired by Nonfiction Writing Advice.

Finally, I explain why disclaimers suck.

Meaning Injection

In middle school, I carried a single enormous binder to every class, filled with colored tabs and lined paper. In that binder, I took notes religiously, trying various techniques of organization and highlighting.

I had no good notion of why I took notes. After all, I never looked at the notes I took - textbooks were always easier on the eyes than my chicken scrawl. I did notice, however, that I remembered things better when I wrote them down, so for a time my plan was to simply take the notes and forget about them in my binder.

I eventually noticed what was actually working, and discarded the superstition. I left my binder to gather dust and started bringing lined paper to class to take the notes. When the bell tolled, I dropped the notes in the recycling before leaving the classroom. I wrote down only the key words and spent the rest of class staring at those scattered words, injecting them with the meaning. Imagine lonely crimson ink-drops falling into a glass of water, bursting into delicious, velvety tendrils at contact. Plop, Plop. That's how meaning injection feels.

Several iterations later, my study strategy is perfectly streamlined.

I no longer take notes in class. The night before each test, I skim the textbook with a blank sheet of paper and write down all the important terms - without definitions; the sheet of paper serves as a kind of corporeal mind palace. I read through the list once, injecting the words with meaning. Plop, Plop. And that would be enough.

Research Notes

Fast forward to 2013 and transport yourself to my first summer research program. Every Monday, we give a brief board about that week's progress. I mull ideas on paper over the week before TeXing them up Sunday night.

A...
Replace Stereotypes With Experiences
by  — 2018-01-29T00:07:15.056Z

[A small, short reminder for a substitution mistake our (read: my) brain(s) sometime(s) make.]

Related to: Learning from Past Experiences

Here’s something that sometimes happens to me:

Someone will ask me what flavor of ice cream I want. I look at the choices, and I see vanilla, chocolate, and mango. I look at mango and think “Huh, mango. That sounds pretty cool. All fruity and exotic, that sounds exciting.” So, having won myself over with the “mango-ness” of the said option, I’ll choose mango.

Once I actually start eating, though, I remember past times I had ice cream, and I recall that actually I really like vanilla; I often choose vanilla as a flavor, and I never regret it. Somehow, a part of me had relevant memories about ice cream flavors, and they just didn’t show up when I was considering the options.

Here’s another example:

Someone suggests that I do some stuff in physics. “Huh, physics. That sounds pretty cool. Stars, theories, grand intricate mysteries of the universe,” I think to myself. There’s a genuine feeling of excitement associated with all of those mental images.

Once I actually get down to it, I remember past times when I did physics work and it was super tedious. And, sitting down, I realize that the actual work in physics is completely different than the feelings I had associated with “doing physics” when I first thought of the option.

In other words, your mental prototype/stereotype of X might not actually match your past experience of X (meaning the two might not be well-connected in your mind). In other cases, you might conflate any valence you have attached to an object Y, with Z, where Z is merely an action that involves Y in some capacity.

More potential examples:

The experience of reading a book might not match your opinions on whether books are good or bad.
Actually doing math is quite different from any abstract feelings you have about mathematics, the field as a whole.
The sensations when you are cleaning up your room might feel different than how you feel about cleaning up your room, or how you feel when you look at your not-yet-cleaned room.

One specific thing I’m trying to get at here is that, when considering questions of preference, I’m giving lots of weight to the valence and immediate associations I have to the options, instead of relevant past experience which could better inform my decision.

For example, it’s much better to ask you...
All Mathematicians are Trollable: Divergence of Naturalistic Logical Updates
by abramdemski — 2018-01-28T14:50:25.000Z

The post on naturalistic logical updates left open the question of whether the probability distribution converges as we condition on more logical information. Here, I show that this cannot always be the case: for any computable probability distribution with naturalistic logical updates, we can show it proofs in an order which will prevent convergence. In fact, at any time, we can drive the probability of x up or down as much as we like, for a wide variety of sentences x.

As an aid to intuition, I describe the theorem informally as "all mathematicians are trollable". I was once told that there was an "all mathematicians go to Bayesian hell" theorem, based on the fact that a computable probability distribution must suffer arbitrarily large log-loss when trying to model mathematics. The idea here is similar. We are representing the belief state of a mathematician with a computable probability distribution, and trying to manipulate that belief state by proving carefully-selected theorems to the mathematician.

(ETA: A way around this result has now been found, giving us an untrollable mathematician.)

□p represents "p is provable", but we consider a class of proofs which the mathematician believes are sound; this justifies characterizing the mathematician's beliefs with the provability logic GLS (as in naturalistic logical updates) rather than GL, so the mathematician's beliefs will obey the principle □p→p. For concreteness, we can suppose that the mathematician accepts proofs in PA.

Definition. A probability assignment over a language L is a function from sentences of that language to values in the interval [0,1].

This is, of course, an extremely weak definition. We will generally also want coherence with respect to some logic. Where a logic A is a set of axioms and inference rules over L, we define coherence as usual:

Definition. A probability assignment P over L is coherent with respect to A if and only if:

⊢a implies P(a)=1 and a⊢⊥ implies P(a)=0,

whenever a⊢b, we have P(a)≤P(b), and

P(a)+P(b)=P(a∨b)+P(a∧b).

This is equivalent to a number of other common definitions of coherence. For a computable probability assignment, coherence with PA won't be possible. Requirement (1) alone is impossible: the sentences which can be proven from PA and those which can be refuted are recursively inseparable sets.

We can, however, insist that the probability assignment is coheren

...
Conflict Theorists pretend to be Mistake Theorists
by Chris_Leong — 2018-01-28T12:06:31.307Z

Scott's recent post on Conflict Theory vs Mistake Theory is probably his most important post in quite a while. As Less Wrongers tend to be Mistake Theorists, they tend to try to see others through the lens of Mistake Theory. And a lot of value can be derived from this. Even when someone seems to be merely pursuing their own interests, there usually are at least some arguments as to why they are actually acting for the common good. Yet some people are Conflict Theorists and denying this reality is simply not rational.

Before I continue any further, I need to make a distinction between Total War Conflict Theory and Proportional War Conflict Theory. Those who practise Total War have very weak inhibitions against using underhand or vicious tactics against their enemies. I don't mean they have no limits - many people set a limit at serious physical violence - but they certanly don't wait for their enemies to shoot first. On the other hand, those who practise Proportional War still have a strong respect for ethical standards of behaviour. They recognise that they are in a war and that winning a war has a cost, but they try to hold onto as much morality as they can, knowing how easy it is to slowly slide down the slippery slope. In those post, when I talk about Conflict Theorists, I'll actually be using it as shorthand for Total War Conflict Theorists. I just wanted to clarify this before I started.

Anyway, one key point that Scott didn't mention is that many Conflict Theorists will try as hard as possible to convince everyone that they are Mistake Theorists. You see, if they admit that they are merely ideological warriors for their side and that they aren't really open to discussion or debate, many people would choose not to talk to them or to listen to them. So Conflict Theorists will pretend to debate in good faith, but if they aren't winning they will simply switch to attacking your character instead. Of course, they don't want to admit that they are Conflict Theorists, so the standard pattern for doing that is to present you as an unreasonable, ideological warrior who is in denial about obvious truths. The benefits of this strategy are to deny the enemy views airtime, to indicate to allies that considering these views would be a betrayal and to bully moderates into being silent.

Another common strategy is the use of ideolog...
The different types (not sizes!) of infinity
by Stuart_Armstrong — 2018-01-28T11:14:00.000Z

In a recent conversation, a smart and mathy friend of mine revealed they didn't understand Cantor's diagonal argument. Further questionning revealed that she was using the wrong concept of infinity. I thought I'd pass on my explanations from there.

What is infinity? Does one plus infinity make sense? What about one over infinity? Well, it all depends what concept of infinity you're using. Roughly speaking, infinity happens when you take a finite concept and add a "and then that goes on for ever and ever" at the end. There are four major examples:

If you take the concept of "size" and push that to infinity, you get the infinite cardinals.
If you take the concept of "ordered set" and push that to infinity, you get the infinite ordinals.
If you take the concept of "increasing function" and push that to infinity, you get the poles and limits of continuous functions - a major part of real analysis.
And if you take normal algebra and push that to infinity, you get the hyperreals.

To add to the confusion, there are normally only two symbols for infinity to go around - ω and ∞ - even though there are at least four very different concepts (honourable mention also goes to the infinities of complex analysis and algebraic geometry, which are like more limited versions of the real analysis one).

So, does ω+1 make sense (as something different from ω)? It does, for the ordinals and hyperreals only. What about ω-1, similarly? That only makes sense for the hyperreals. And -ω? That works for the hyperreals and real analysis. 1/ω? This is well defined for the hyperreals (where it is not equal to 0), and is arguably well-defined for real analysis, where it would be equal to 0.

Is ω times 2 the same as ω? "Yes", say real analysis and cardinals; "No", say ordinals and hyperreals. How about there being many different infinities of different sizes? That makes sense for all of these, except real analysis.

So when talking about "infinity", be careful that you're using the right concept of infinity, and don't mix one with another, or import the intuitions of one area into an arena where it doesn't make sense.

***

EDIT: See also Scott Garrabrant's comment:

Starting with the natural numbers:

Cardinals: How many are there?
Ordinals: What comes next?
Limits: Where are they going?
Hyperreals: Wh

...
A LessWrong Crypto Autopsy
by Scott Alexander — 2018-01-28T09:01:49.799Z

Wei Dai, one of the first people Satoshi Nakamoto contacted about Bitcoin, was a frequent Less Wrong contributor. So was Hal Finney, the first person besides Satoshi to make a Bitcoin transaction.

The first mention of Bitcoin on Less Wrong, a post called Making Money With Bitcoin, was in early 2011 - when it was worth 91 cents. Gwern predicted that it could someday be worth "upwards of $10,000 a bitcoin". He also quoted Moldbug, who advised that:

If Bitcoin becomes the new global monetary system, one bitcoin purchased today (for 90 cents, last time I checked) will make you a very wealthy individual...Even if the probability of Bitcoin succeeding is epsilon, a million to one, it's still worthwhile for anyone to buy at least a few bitcoins now...I would not put it at a million to one, though, so I recommend that you go out and buy a few bitcoins if you have the technical chops. My financial advice is to not buy more than ten, which should be F-U money if Bitcoin wins.

A few people brought up some other points, like that if it ever became popular people might create a bunch of other cryptocurrencies, or that if there was too much controversy the Bitcoin economy might have to fork. The thread got a hundred or so comments before dying down.

But Bitcoin kept getting mentioned on Less Wrong over the next few years. It's hard to select highlights, but one of them is surely Ander's Why You Should Consider Buying Bitcoin Right Now If You Have High Risk Tolerance from January 2015. Again, people made basically the correct points and the correct predictions, and the thread got about a hundred comments before dying down.

I mention all this because of an idea, with a long history in this movement, that "rationalists should win". They should be able to use their training in critical thinking to recognize more opportunities, make better choices, and end up with more of whatever they want. So far it's been controversial to what degree we've lived up to that hope, or to what degree it's even realistic.

Well, suppose God had decided, out of some sympathy for our project, to make winning as easy as possible for rationalists. He might have created the biggest investment opportunity of the century, and made it visible only to libertarian programmers willing to dabble in crazy ideas. And then He might have made sure that all of the earli...
Pareto improvements are rarer than they seem
by philh — 2018-01-27T22:23:24.206Z
Link: http://reasonableapproximation.net/2018/01/27/pareto-improvements-rare.html

this is surely not an original insight, but I haven't seen it before

A Pareto improvement is where you make one party better off and no parties worse off.

Suppose Adam has a rare baseball card. He assigns no intrinsic value to baseball cards. Adam likes Beth, and somewhat values her happiness. Beth collects baseball cards, and would happily pay $100 for Adam's card.

If Adam just gives Beth his baseball card, is that a Pareto improvement? Naively, yes: he loses the card that he doesn't care about, and gains her happiness; she gains the card. Both are better off.

But I claim not, because if Adam has the card, he can sell it to Beth for $100. He would much prefer doing that over just giving her the card. But if Beth has the card, he can't do that. He assigns no intrinsic value to the card, but he can still value it as a trading chip.

Now suppose Adam has the baseball card but Beth also has a copy of that card. Then Beth has less desire for Adam's card, so this situation also isn't a Pareto improvement over the original. By giving something to Beth, we've made Adam's situation worse, even though Adam likes Beth and values her happiness .

And I think situations like this are common. The ability to give someone something they want, is a form of power; and power is instrumentally useful. And the less someone wants, the less able you are to give them something they want¹.

For a closer-to-reality example, the reddit comment that sparked this post said:

bringing Platform 3 back into use at Liverpool Street Underground Station was denied because the platform would not be accessible. Neither of the platforms currently in use for that line is accessible, so allowing Platform 3 to be used would be a Pareto improvement

The model here is that there are two parties, people who can access the platforms at Liverpool St and those who can't. If Platform 3 is brought back into use, the first group gains something and the second group loses nothing.

But I think that if Platform 3 is brought back into use, the second group loses some power. They lose the power to say "we'll let you bring back Platform 3 if you give us...". Maybe Platform 3 can be made accessible for $1 million. Then they can say "we'll let you bring it back if you make it accessible", but they can't do that if it's already back in use.

An...
Against Instrumental Convergence
by zulupineapple — 2018-01-27T13:17:19.389Z

Instrumental convergence is the idea that every sufficiently intelligent agent would exhibit behaviors such as self preservation or acquiring resources. This is a natural result for maximizers of simple utility functions. However I claim that it is based on a false premise that agents must have goals.

What are goals?

For agents constructed as utility maximizers, the goal coincides with utility maximization. There is no doubt that for most utility functions, a utility maximizer would exhibit instrumental convergence. However, I claim that most minds in the mindspace are not utility maximizers in the usual sense.

It may be true that for every agent there is a utility function that it maximizes, in the spirit of VNM utility. However these utility functions do not coincide with goals in the sense that instrumental convergence requires. These functions are merely encodings of the agent's decision algorithm and are no less complex than the agent itself. No simple conclusions can be made from their existence.

Humans exhibit goals in the usual sense and arguably have VNM utility functions. Human goals often involve maximizing some quantity, e.g. money. However explicit human goals represent only a small fraction of their total utility computation. Presumably, the goals explain the extent to which some humans exhibit instrumental convergence, and the rest of the utility function explains why we haven't yet tiled the universe with money.

What about non-human non-utility-maximizer agents? Certainly, some of them can still exhibit instrumental convergence, but I will argue that this is rare.

What is the average mind in the mindspace?

The "mindspace" refers to some hypothetical set of functions or algorithms, possibly selected to meet some arbitrary definition of intelligence. I claim that most minds in the mindspace do not share any properties that we have not selected for. In fact, most minds in the mindspace do nothing even remotely useful. Even if we explicitly selected a random mind from a useful subset of the mindspace, the mind would most likely do nothing more than the bare minimum we required. For example, if we search for minds that are able to run a paperclip factory, we will find minds that run paperclip factories well enough to pass our test, but not any better. Intelligence is defined by the ability to solve problems, not by the degree of ag...