trentmkelly/LessWrong-43k huggingface dataset:
Hull: An alternative to shell that I'll never have time to implement
by Martin Sustrik — 2019-04-28T07:40:00.528Z

Here's an idea. It's kind of crazy but it's also fun to think about.

Imagine a language which has every piece of its state stored in the filesystem.

Yes, I know it sounds weird but do read on!

Datatypes

Let's say we do this:

a = 1

What will happen under the covers is that file "a" will be created that contains nothing but "1".

Lists could be represented as files where each line corresponds to an item.

b = [1, 2, 3]

The above would simply create file "b" with the following content:

1 2 3

Filesystem also provides hash tables out of the box. They are called directories.

c = { foo: 1, bar: { quux: 3 } }

Which would create a directory "c" with file "foo" (containing "1") and directory "bar" containing file "quux" which would in turn contain "2".

Fun idea: For map indexing use operator / instead of []. So, to access "quux" you would write "c/bar/quux" which happens to be the path to the file in question.

Call stack

Of course, state of the program is not just about variables. We would have to represent the call stack in some way.

Easy to do! For each stack frame create a directory called "nextframe". The call stack would then look like a sequence of embedded directories:

main |-- a # contains 1 |-- b # contains 1\n2\n3 `-- nextframe |-- c | |-- foo # contains 1 | `-- bar | `-- quux # contains 2 `-- nextframe `-- a # contains 2

In such a model, scoping is easy. If you want to access "b" in the innermost frame, just walk up the directory tree until you find a file (or directory) named "b".

Also, overrides work as expected. If you ask for "a" in the innermost frame, you'll get "2", not "1".

Instruction pointer

Now you are probably wondering how would one persist the instruction pointer. The language needs to know which command is the next to execute after all and not making this information persistent defeats the entire point of the language.

Let's make it work this way:

When program is started, a copy of the source file is created. We'll call it a "todo" file.

Then the first line of the "todo" file is executed. Then it is removed from the file.

We'll do the same with the second and every subsequent line.

That way the "todo" file always contains a list of commands that are yet to be executed.

But hey, you cry, if there's a loop in the source code we can't just delete a command within the loop! We will still need it in the next iteration!

Easy to fix! L

...
Aging research and population ethics
by emanuele ascani — 2019-04-28T07:21:17.888Z

This is the second of a series of posts in which I'm trying to build a framework to evaluate aging research. Previous post:

A general framework for evaluating aging research. Part 1: reasoning with Longevity Escape Velocity.

Summary

The first part of this post will explore the potential sensitivity of the impact of aging research on two different views of population ethics. Under the person-affecting view of population ethics, in which creating new lives has neutral moral value, it seems that aging research is really valuable. Under the impersonal view, in which creating new lives has a positive moral value, it could be less clear. By looking at demographic trends, and analyzing the motivations for why people have children, it turns out that saving people by hastening the arrival of LEV wouldn't prevent births and could actually increase the average fertility rate of the world. This leads to a counterintuitive result: Aging research could be even more valuable under the impersonal view of population ethics.

In the second part of the post, I'll explore how to reason about moral weights, which could also increase the impact of making LEV come closer if longer lives are valued more than shorter lives for reasons other than QALYs. There are various arguments for why one should prefer some kind of age-discounting or its contrary, but the answer ultimately depends on what a 1000-year-old life and mind looks like and how it is different from the life and mind of a shorter-lived person. Therefore, taking a neutral stance is suggested unless it is believed that the future, if there will be one, is more likely to be better than the present. In that case, the lives of people saved through LEV should count for more.

Population ethics

At first glance, the impact of aging research seems to greatly change depending on if you adopt the impersonal view of population ethics or the person-affecting perspective. In the impersonal view, creating new lives is regarded as good. Assuming that there isn't suffering at the end of life and people get replaced immediately, this view holds no ethical difference between making people live longer and replacing them with new people. Under the person-affecting perspective, however, creating new lives is not valued: only already existing people are valued, and thus how bad it is to die depends on the amount of well-being lost....
[AN #54] Boxing a finite-horizon AI system to keep it unambitious
by Rohin Shah — 2019-04-28T05:20:01.179Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter.

The newsletter now has exactly 1,000 subscribers! It's a perfect time to take the 3-minute

survey

if you haven't already -- just think of how you'll be making the newsletter better for all 1,000 subscribers! Not to mention the readers on

Twitter

and the

Alignment Forum

.

Highlights

Asymptotically Benign AGI

(Michael Cohen): I'm a bit out of my depth with this summary, but let's give it a shot anyway. The setting: we are not worried about how much compute we use (except that it should be finite), and we would like to build a powerful AI system that can help us with tasks but does not try to influence the world. We'll assume that we can construct a box from which no signals can pass through, except by pressing a specific button that opens a door.

First, the simple version of BoMAI (Boxed Myopic AI). We'll put the AI system and the operator in the box, and the operator and the AI system can talk via text message, and the operator can enter rewards. Each episode has a maximum length (hence myopic), and if the operator ends the episode early, all future rewards are set to zero. BoMAI maximizes episodic reward in a manner similar to

AIXI

. It has a distribution (initially a speed prior) over all possible time-bounded Turing Machines as possible models that predict observations and rewards. BoMAI uses the maximum a posteriori (MAP) Turing Machine to predict future observations and rewards given actions, searches over all possible sequences of actions for the best one, and executes the first action of that sequence. (See

this comment

and its parents for the reason to use the MAP model.)

Intuitively, BoMAI has no incentive to affect anything outside the box: information can only leave the box if the episode ends, but if that happens, reward is guaranteed to be exactly zero. It might deceive the operator into giving it high reward, but any such deception will only incidentally affect the outside world, it hasn't been optimized to cause any such effect. However, this is all assuming that the MAP world model is accurate, which may not be true.

Obviously at initialization our world model will not be accurate: we need to explore. Rather than using something like epsilon-explor...
What are some good examples of incorrigibility?
by RyanCarey — 2019-04-28T00:22:45.459Z

The idea of corrigibility is roughly that an AI should be aware that it may have faults, and therefore allow and facilitate human operators to correct these faults. I'm especially interested in scenarios where the AI system controls a particular input channel that is supposed to be used to control it, such as a shutdown button, a switch used to alter its mode of operation, or another device used to control its motivation. I'm especially interested in examples that would be compelling within the machine learning discipline, so I'd favour experimental accounts over hypotheticals.
Counterspells
by Virgil Kurkjian — 2019-04-27T23:37:09.353Z

Back in the early days of the internet, when New Atheism was king, people would collect and catalogue various logical fallacies. Then, when getting into an argument on some forum, they would identify and call out all the logical fallacies being used by their opponent; Ad Hominem! Appeal to Emotion! Tu Quoque! Post hoc, ergo propter hoc!

These days we tend to know better. Appealing to these lists of logical fallacies and their Latin names doesn't help you, and it doesn't help the conversation. The problem is that to the person you're talking to, throwing around the names of these fallacies just sounds like an Argument from Authority—

Wait! I mean, it just sounds like you think you're smarter or better educated than they are, which isn't relevant to the argument. If you really had something to contribute, you would explain the problem with their reasoning, rather than just throwing out a Latin term they've never heard of. Even if they have heard of it, you should still take the time to tell them why you think it applies to what they said.

Logical fallacies are all sound complaints. But when disagreeing with someone, they deserve to hear the arguments those complaints are based on. More than that, you're not going to convince someone of your position unless you actually try to convince them.

Fortunately, logical fallacies are such bad arguments that most of them can be rebutted in a few simple sentences. And the form of each fallacy tends to be so consistent that these rebuttals can be highly formulaic.

In accordance with the Rationalist tradition that requires everything to have a nerdy sci-fi or fantasy name, I am calling these formulaic rebuttals Counterspells.

Like I mentioned, I know that by now everyone knows how to avoid using these fallacies themselves. Most of us can also recognize when other people are using these bad arguments against us. Somehow the next step doesn't come as easily.

You might think that an intelligent person, recognizing a bad argument they've seen a hundred times before, would be able to provide a knock-down counterargument. But for me at least, I still regularly crash and burn in my attempts to extemporize rebuttals. I see a lot of you all make the same problem.

Some writers have proposed some great Counterspells, but no one has collected them. To begin with, I'm going to review some exc...
10 Good Things about Antifragile: a positivist book review
by SebastianG  — 2019-04-27T18:22:16.665Z

Although Taleb is "comically arrogant and kind of childish about his disagreements with others," here are 10 things I really appreciated about Antifragile.

1) Sometimes I overstrategize how to formulate my thoughts and I get stuck never voicing them, and then articulating them badly anyway. Part of the cause of this overstrategizing comes from fear of burning bridges and fear of being outcast. Taleb frequently does not formulate his thoughts well, but one cannot say he is not bold in stating his ideas and what he thinks of other people (“frauds” and “Fragilistas”). While his style is not a virtue itself, it’s worth taking his thigh-slapping puff-chested example for what it’s worth – a bit of encouragement. Taleb provides real life examples of this too. A Lebanese widow during the civil war had her windows shot out in a fire fight. The next day she went outside and confronted the enemy soldiers. Tiny, dressed in black, guns pointing at her, she nonetheless yelled at these 20 year old boys that they were messing up the neighborhood, behaving improperly and shamefully. Taleb says it was clear that although they had guns, their faces betrayed embarrassment. This incident of a widow’s heedless self-respect reminds me also of the few resistors of arrest and interrogation Solzhenitzen mentions in Gulag Archipelago (get the unabridged edition BTW, the abridged is missing important elements of the literary experiment). Why were they able to resist arrest and interrogation? They exposed the immorality of the event fearlessly, thus embarrassing oppressors.

2) If you have sour grapes about a job, academic position, or intellectual position, Antifragile might be right for you. Taleb’s litany of Anglican country priests who made impressive scientific discoveries betrays what he sees as the institutional myth. You as a scholar and researcher do not need a big institution to do your work. You can build it on your own. Intellectual entrepreneurship is essentially what Less Wrong and the Rationality community is. Maybe you can’t get into an organization that you want to be part of. There is nothing stopping you from founding your own. Start it as a hobbyist, just as the priests which made so many scientific discoveries did.

3) One thing Antifragile helped me see is the high amount of optionality in my current job, a job which like everyone I sometimes dislike. Noneth...
What's the best approach to curating a newsfeed to maximize useful contrasting POV?
by Ben Goldhaber — 2019-04-26T17:29:30.806Z

In Superforecasters, Tetlock describes one superforecaster as having built a collection of automated scripts and sources to get useful information from a variety of different sources and different perspectives. This seems very useful and I'd like to emulate it.

I'm looking for a generalizable approach where, given a specific topic (ex. AI) I can curate sources (ex. Twitter accounts) that maximizes diversity of points (ex. different perspectives on AI Alignment risk). I'd like to avoid the bias where I only use sources that are well known and popular, which seems particularly likely to happen in cases where I am not familiar with the field

I have a few different tactics in mind, but I haven't yet settled on a cohesive strategy, and am interested in suggestions, in particular if you've done this before.
Speaking for myself (re: how the LW2.0 team communicates)
by Ruby — 2019-04-25T22:39:11.934Z

Posts made by members of the LessWrong 2.0 team are typically made from the perspective of the individual - even when they are writing in their capacity as LessWrong team members. My (Ruby’s) model of the team’s reason for this is that even if there are collective decisions, there are no collective models. Not real models.

When the team agrees to do something, it is only because enough of the individual team members individually have models which indicate it is the right thing to do. Our models might be roughly the same at a high-level such that you can describe a “common denominator” model, but this isn’t an actual model held by an actual person. I think such “common denominator” group models are undesirable for at least the following reasons:

Pressure to form consensus reduces the diversity of models, sometimes going from multiple models per person to only a single model for the group. This can then result in overconfidence in the surviving model.
There might be no group model. The group might have agreed on a decision, but they never reached consensus on the reasons for it.
It is costly to describe group models. Either someone has to draft the model, get feedback, make revisions, repeat, until eventually it is “good enough” or someone describes a model putatively held by the group, but which is not actually representative of the group's thinking.
In fact, no individual might endorse the group model as being their own.
The person describing the group model doesn’t necessarily understand things they’re including which came from others.
In averaging multiple models, detail is lost and you no longer have a model which can usefully generate predictions.
No individual owns the model, making it hard for any one person to elucidate, defend, or be held accountable for it.
Reluctance to speak on the behalf of others means that very little gets said.

Crucially, group models which get shared externally are very often not the models which were used to make decisions. If you want to understand a decision, you want the actual model which generated it.

Given the goal of sharing our actual true thinking with the outside world (rather than nicely curated PR announcements), the LessWrong team has the rough policy that we speak from our personal point of view and avoid committing to an impersonal, authoritative, “official view of LessWrong.”

I suspect (and I believe the team generally...
Asymmetric Justice
by Zvi — 2019-04-25T16:00:01.106Z

Related and required reading in life (ANOIEAEIB): The Copenhagen Interpretation of Ethics

Epistemic Status: Trying to be minimally judgmental

Spoiler Alert: Contains minor mostly harmless spoiler for The Good Place, which is the best show currently on television.

The Copenhagen Interpretation of Ethics (in parallel with the similarly named one in physics) is as follows:

The Copenhagen Interpretation of Ethics says that when you observe or interact with a problem in any way, you can be blamed for it. At the very least, you are to blame for not doing more. Even if you don’t make the problem worse, even if you make it slightly better, the ethical burden of the problem falls on you as soon as you observe it. In particular, if you interact with a problem and benefit from it, you are a complete monster. I don’t subscribe to this school of thought, but it seems pretty popular.

I don’t say this often, but seriously, read the whole thing.

I do not subscribe to this interpretation.

I believe that the majority of people effectively endorse this interpretation. I do not think they endorse it consciously or explicitly. But they act as if it is true.

Another aspect of this same phenomenon is how most people view justice.

Almost everyone agrees justice is a sacred value. That it is good and super important. Justice is one of the few universally agreed upon goals of government. Justice is one of the eight virtues of the avatar. Justice is up there with truth and the American way. No justice, no peace.

But what is justice? Or rather, to avoid going too deeply into an infinitely complex philosophical debate millenniums or eons old, how do most people instinctively model justice in broad terms?

In a conversation last night, this was offered to me (I am probably paraphrasing due to bad memory, but it’s functionally what was said), and seems common: Justice is giving appropriate punishment to those who have taken bad action.

I asked whether, in this person’s model, the actions needed to be bad in order to be relevant to justice. This prompted pondering, after which the reply was that yes, that was how their model worked.

I then asked whether rewarding a good action counted as justice, or failing to do so counted as injustice, using the example of saving someone’s life going unrewarded.

We can consider three point-based justice systems.

In the asymmetric sys...
What would Femtotechnology, attotechnology, yoctotechnology and planck level technology be theoretically capable of ?
by Orange Apple Juice — 2019-04-25T07:45:39.189Z

In paul davies book "super force" there's the following except

" Quantum physics associates a scale of energy (equivalently mass) with a
scale of length. Measure-for-measure the diameter of a nucleus (about
10 -12 cm) corresponds roughly to the mass of the pion. As smaller
lengths are probed, so the energies involved start to climb. To explore the
quark-filled interior of a proton it is necessary to go up to energies at least
ten times greater than the proton mass. Much further up the scale of
energy is the unification mass, at about 1014 proton masses. If one had
command of such huge mass-energy (which we don't) it would be
possible to probe the world of the X particles, at which the very distinction
between quarks and leptons fades away.
How much energy do we need to 'get inside' the seven-sphere and
explore the other dimensions of space? According to the Kaluza-Klein
theory it is necessary to go beyond even the unification scale, to an energy
equivalent to 1019 proton masses. Only at this unimaginable energy would
the extra dimensions manifest themselves directly.
The huge value of 1019 proton masses is known as the 'Planck scale',
because it was originally discovered by Max Planck. the inventor of the
quantum theory. At the Planck energy. all four forces of nature would be
completely merged into a single superforce, and all ten dimensions of
space would exist on an equal footing. If we could concentrate enough
energy to take us up to the Planck scale, the full dimensionality of space
would be exposed in all its splendour.
Letting imagination have free rein, it is possible to envisage mankind
one day gaining control over the superforce. To achieve this would enable us to manipulate the greatest power in the universe, for the superforce is
ultimately responsible for generating all forces and all physical structures.
It is the fountain-head of all existence. With the superforce unleashed, we
could change the structure of space and time, tie our own knots in
nothingness, and build matter to order. Controlling the superforce would
enable us to construct and transmute particles at will, thus generating
exotic forms of matter. We might even be able to manipulate the dimensionality of space itself, creating bizarre artificial worlds with unimaginable
properties. Truly we should be lords of the universe."

How powerful would the ability to change the structure of th

...
Strategic implications of AIs' ability to coordinate at low cost, for example by merging
by Wei Dai — 2019-04-25T05:08:21.736Z

It seems likely to me that AIs will be able to coordinate with each other much more easily (i.e., at lower cost and greater scale) than humans currently can, for example by merging into coherent unified agents by combining their utility functions. This has been discussed at least since 2009, but I'm not sure its implications have been widely recognized. In this post I talk about two such implications that occurred to me relatively recently.

I was recently reminded of this quote from Robin Hanson's Prefer Law To Values:

The later era when robots are vastly more capable than people should be much like the case of choosing a nation in which to retire. In this case we don’t expect to have much in the way of skills to offer, so we mostly care that they are law-abiding enough to respect our property rights. If they use the same law to keep the peace among themselves as they use to keep the peace with us, we could have a long and prosperous future in whatever weird world they conjure. In such a vast rich universe our “retirement income” should buy a comfortable if not central place for humans to watch it all in wonder.

Robin argued that this implies we should work to make it more likely that our current institutions like laws will survive into the AI era. But (aside from the problem that we're most likely still incurring astronomical waste even if many humans survive "in retirement"), assuming that AIs will have the ability to coordinate amongst themselves by doing something like merging their utility functions, there will be no reason to use laws (much less "the same laws") to keep peace among themselves. So the first implication is that to the extent that AIs are likely to have this ability, working in the direction Robin suggested would likely be futile.

The second implication is that AI safety/alignment approaches that aim to preserve an AI's competitiveness must also preserve its ability to coordinate with other AIs, since that is likely an important part of its competitiveness. For example, making an AI corrigible in the sense of allowing a human to shut it (and its successors/subagents) down or change how it functions would seemingly make it impossible for this AI to merge with another AI that is not corrigible, or not corrigible in the same way. (I've mentioned this a number of times in previous comments, as a reason why I'm pessimistic about specific appro

...