trentmkelly/LessWrong-43k huggingface dataset:
Jan Bloch's Impossible War
by Slimepriestess — 2020-02-17T16:14:46.033Z
Link: https://hivewired.wordpress.com/2020/02/17/jan-blochs-impossible-war/

Epistemic Status: Endorsed
Content Warning: Neuropsychological Infohazard, Evocation Infohazard, World War I
Recommended Prior Reading: Blueprint for Armageddon Part I
Part of the Series: Truth

“History doesn’t repeat itself but it often rhymes”

In any real look into the past, you realize pretty quickly that things don’t have neat beginnings or simple origins in the vast majority of cases. Historical events are the result of billiard ball interactions among a chaotic conflux of actors and forces, themselves all built out of past impacts and collisions stretching back into the mists of antiquity.

Thus when trying to tell the origin story of the modern rationality community, it can be very tempting to just keep extrapolating backwards. How far back should we look? Do we need to rehash Plato’s Cave and Cogito Ergo Sum? Francis Bacon is credited as the grandfather of science, so maybe we should start with him?

For the moment at least I’m writing blog posts not thousand page textbooks, and my goal here isn’t to rehash the entire history of scientific and philosophical thought (I’d like to keep this blog post under three thousand words). If you want the entire history of scientific thought, Cosmos is a great place to start and has some pretty spiffy graphics.

But unlike history, every story and every blog post have to start somewhere, and I think the best place to start for our purposes is with polish banker and railway financier Jan Gotlib Bloch.

Bloch was born a Polish Jew in Tsarist Russia in the 1800s, and would later convert to Calvinism to protect himself from antisemitism within the Tsarist government. Bloch worked as a banker and would go on to finance the building of rail lines in Russia, as well as penning a lengthy treatise on the management and operation of said rail lines in 1875, for which he:

was awarded a medal of the first class at the geographical exhibition of Paris, and was heartily endorsed by the Imperial Russian Geographical Society.

But it was Bloch’s later work that would be remembered for. In 1870, The Northern German Confederation would go to war with the Second French Empire. Fueled by fears of the growing power of a rapidly unifying and industrializing Germany, France declared war and invaded in August of 1870.

The war was only six months long. By September, Napoleon III was captured and the French Imperial Army had b

...
Subagents and impact measures: summary tables
by Stuart_Armstrong — 2020-02-17T14:09:32.029Z

These tables will summarise the results of this whole sequence, checking whether subagents can neutralise the impact penalty.

First of all, given a subagent, here are the results for various impact penalties and baselines, and various "value difference summary functions" f:

Impact PenaltyPenalty neutralised- sw inaction?Penalty neutralised- inaction?Non-indexicalYesNoIndexical, f decreasingYesNoIndexical, f increasingYesMostly

Another way of phrasing "f decreasing": it penalises too little power, not too much. Conversely, "f increasing" penalises too much power, not too little. Thus, unfortunately:

Subagents do allow an agent to get stronger than the indexical impact penalty would allow.

Subagents don't allow an agent to get weaker than the indexical impact penalty would allow.

Examples

This table presents, for three specific examples, whether they could actually build a subagent, and whether that would neutralise their impact penalty in practice (in the inaction baseline):

SystemSA possible?Penalty neutralised?20BQYesNoRRYesNoAUProbablyMostly

Here, 20BQ is twenty billion questions, RR is relative reachability, and AU is attainable utility.

Now, whether the RR or AU penalties are undermined technically depends on f, not on what measure is being used for value. However, I feel that the results undermine the spirit of AU much more than the spirit of RR. AU attempted to control an agent by limiting its power; this effect is mainly neutralised. RR attempted to control the side-effects of an agent by ensuring it had enough power to reach a lot of states; this effect is not neutralised by a subagent.
Appendix: mathematics of indexical impact measures
by Stuart_Armstrong — 2020-02-17T13:22:43.523Z

Overall summary post here.

This post looks at the fascinating situation of indexical impact measures (under the inaction baseline), in the presence of subagent. What happens here in non-trivial; to summarise:

If the impact measure compels the agent to "preserve the ability to do X", then the impact measure will not be undermined by a subagent.

If the impact measure compels the agent to "never have the ability to do X", then a subagent strongly undermines the impact measure.

If the impact measure compels the agent to "keep your ability to do X at a constant level", then a subagent allows the agent to increase that ability, but not decrease it.

For example, the attainable utility impact measure attempts to measure the power of an agent. Power is, roughly, the ability increase certain reward function. The impact measure penalises changes in the agent's power (as compared with the baseline).

So, we can expect the restriction on the agent losing power to be preserved, while the restriction on gaining power will be strongly undermined. For the spirit of low impact, it would, of course, have been better if these had been the other way round.

Excess power

As before, assume an agent A, which has managed to construct a subagent SA.

I'll be using a slight modification of Vika's formulation. A general-value penalty is of the form:

DA(st;s′t)=∑k∈Kwkf(Vk(st)−Vk(s′t)).

Here, st is the state the environment is in; s′t is the inaction baseline, the state the environment would have been in, had it done nothing (noop, ∅) since the beginning. The K is some indexing set, wk is a weight, Vk(s) is some measure of the value of state s, and the f is a value difference summary function - it establishes what value difference will give what penalty[1]. These Vk are indexical, and, in this post, we'll look at Vk's defined as the maximum over the agent's policy; ie there exists vk such that:

Vk(s)=maxπvk(s,π).

I'll refer to Vk(st)−Vk(s′t) as the (excess) power of the agent, and designate it PAk(st) (since Vk(s′t) is out of the agent's control, we can consider it a constant and absorb it into PAk).

The excess power of the pair

Assume that SA is programmed so that, if the agent follows policy πA, the subagent will follow policy πAS=σ(πA). The excess power of the agent is thus

PAk(st)=maxπAvk(st,[πA,σ(πA)])−Vk(s′t).

Here, [πA,σ(πA)] refers to the joint policies of the agent and the subagent.

Define

...
Stepwise inaction and non-indexical impact measures
by Stuart_Armstrong — 2020-02-17T10:32:01.863Z

Overall summary post here.

In a previous post, I asked which impact measures were vulnerable to subagents. Vika pointed out that it was not merely an issue of of the impact measure, but also of the baseline.

This is indeed the case, but the nature of the impact measure is still relevant. In this post, I'll establish two facts: that under the stepwise inaction baseline, a subagent completely undermines all impact measures (including twenty billion questions). And for the inaction baseline, for non-indexical impact measures, a subagent will not change anything.

The next post will delve into the very interesting things that happen with the inaction baseline and an indexical impact measure.

Stepwise inaction baseline

All impact measures are some form of distance measure between two states, of the form ρ(st,s′t), where st is the state the agent is actually in, and s′t is some baseline state to compare with.

For the stepwise inaction baseline, s′t is calculated as follows. Let st−1 be the previous state, at−1 the previous action. Then s′t is the state that would have followed st−1 had the agent taken the noop action, ∅, instead of at−1.

However, in the presence of a subagent, the agent merely has to always take the action at−1=∅. In that case, st=s′t (in a deterministic setting[1]), and the penalty ρ(st,s′t) collapses to nothing.

This leaves the subagent free to maximise R0 (the positive reward for agent and subagent), without any restrictions beyond making sure that the agent can always take the ∅ action.

Non-indexical impact penalty

The twenty billion questions is an example of a non-indexical impact penalty. The impact penalty is defined as the difference in expected value of many variables, between the current state and the baseline state.

With the inaction baseline state, s′t is the state the system would have been in, had the agent always done nothing/had not been turned on. Thus the definition of s′t is independent of the agent's actions, so the inaction baseline does not collapse like the stepwise inaction baseline does.

What about subagents? Well, since the impact penalty is non-indexical, a subagent does not get around it. It matters not whether the subagent is independent, or an extension of the agent: the impact penalty remains.

In a non-deterministic setting, ρ(st,s′t) becomes a mix of an impact penalty and a measure of environment stochasticity. ↩︎
How to Lurk Less (and benefit others while benefiting yourself)
by romeostevensit — 2020-02-17T06:18:54.978Z

Posts like this have been written before, but I think it's worth making the point periodically.

Lurker ratios have likely increased over time. Comments and discussion are an important feedback mechanism for content creators. So if you see stuff you like, and you'd like to see more posts like it, it's quite helpful to comment. Many people report being intimidated about posting, especially if the platform in question has a highly specific vocabulary and norms. I wanted to offer a couple of the heuristics I use for making comments as well as invite others to boggle/comment/discuss what they think mediates the difference between times they do and don't comment.

In a shallow review of the pedagogy literature, four interventions stood out as having large effect sizes replicate: deliberate practice, test taking, elaborating the context (cross linking knowledge), and teaching the material to others. Cross linking provides an easy heuristic for commenting: simply mention which idea(s) in the post stood out to you most and how they connect to your existing knowledge. This helps you by strengthening those connections, and helps others because each person's cross links have some chance of being unique and therefore surprising to others. I think of this as a sort of low rent version of helping the post author cite additional sources. And speaking as a writer, these sorts of comments are always welcome as I learn about which ideas tend to stand out the most to people and might be worth writing more about.

Another heuristic I've found quite helpful is just to say more obvious things on the margin. Due to illusion of transparency, many things wind up being less obvious than I thought. This also forms a bit of a virtuous cycle as it helps reduce the context overhead for other readers, giving them more on ramps to comment and discuss. You will pay a minor cost of people occasionally getting frustrated that you're repeating something they already know about, but this happens much, much less often in my experience than people thanking me for alerting them to some concept that they either were only somewhat familiar with or had never heard before. This doubles as good vulnerability practice, creating more opportunities to connect with people over odd corners of mutual interest.

I think it's worth it to try over correcting here. I have had a s...
Attainable Utility Preservation: Concepts
by TurnTrout — 2020-02-17T05:20:09.567Z

Appendix: No free impact

What if we want the agent to single-handedly ensure the future is stable and aligned with our values? AUP probably won’t allow policies which actually accomplish this goal – one needs power to e.g. nip unaligned superintelligences in the bud. AUP aims to prevent catastrophes by stopping bad agents from gaining power to do bad things, but it symmetrically impedes otherwise-good agents.

This doesn’t mean we can’t get useful work out of agents – there are important asymmetries provided by both the main reward function and AU landscape counterfactuals.

First, even though we can’t specify an aligned reward function, the provided reward function still gives the agent useful information about what we want. If we need paperclips, then a paperclip-AUP agent prefers policies which make some paperclips. Simple.

Second, if we don’t like what it’s beginning to do, we can shut it off (because it hasn’t gained power over us). Therefore, it has “approval incentives” which bias it towards AU landscapes in which its power hasn’t decreased too much, either.

So we can hope to build a non-catastrophic AUP agent and get useful work out of it. We just can’t directly ask it to solve all of our problems: it doesn’t make much sense to speak of a “low-impact singleton”.

Notes

To emphasize, when I say "AUP agents do X" in this post, I mean that AUP agents correctly implementing the concept of AUP tend to behave in a certain way.

As pointed out by Daniel Filan, AUP suggests that one might work better in groups by ensuring one's actions preserve teammates' AUs.
On the falsifiability of hypercomputation, part 2: finite input streams
by jessicata — 2020-02-17T03:51:57.238Z
Link: https://unstableontology.com/2020/02/17/on-the-falsifiability-of-hypercomputation-part-2-finite-input-streams/

In part 1, I discussed the falsifiability of hypercomputation in a typed setting where putative oracles may be assumed to return natural numbers. In this setting, there are very powerful forms of hypercomputation (at least as powerful as each level in the Arithmetic hierarchy) that are falsifiable.

However, as Vanessa Kosoy points out, this typed setting has difficulty applying to the real world, where agents may only observe a finite number of bits at once:

The problem with constructive halting oracles is, they assume the ability to output an arbitrary natural number. But, realistic agents can observe only a finite number of bits per unit of time. Therefore, there is no way to directly observe a constructive halting oracle. We can consider a realization of a constructive halting oracle in which the oracle outputs a natural number one digit at a time. The problem is, since you don't know how long the number is, a candidate oracle might never stop producing digits. In particular, take any non-standard model of PA and consider an oracle that behaves accordingly. On some machines that don't halt, such an oracle will claim they do halt, but when asked for the time it will produce an infinite stream of digits. There is no way to distinguish such an oracle from the real thing (without assuming axioms beyond PA).

This is an important objection. I will address it in this post by considering only oracles which return Booleans. In this setting, there is a form of hypercomputation that is falsifiable, although this hypercomputation is less powerful than a halting oracle.

Define a binary Turing machine to be a machine that outputs a Boolean (0 or 1) whenever it halts. Each binary Turing machine either halts and outputs 0, halts and outputs 1, or never halts.

Define an arbitration oracle to be a function that takes as input a specification of a binary Turing machine, and always outputs a Boolean in response. This oracle must always return 0 if the machine eventually outputs 0, and must always return 1 if the machine eventually outputs 1; it may decide arbitrarily if the machine never halts. Note that this can be emulated using a halting oracle, and is actually less powerful. (This definition is inspired by previous work in reflective oracles)

The hypothesis that a putative arbitration oracle (with the correct type signature, MachineSpec → Boolean) ...
Wanting More Intellectual Stamina
by mr.magpie — 2020-02-17T02:58:52.911Z

As a sophomore undergraduate student, my most valuable rewards from the college experience have come from personal growth, rather than the classroom. However, one problem that I can't seem to shake is dealing with all the subcategories of my total personality.

On the one hand, I am hyper intellectual, sometimes annoyingly so, because I have an overwhelming number of ideas--all under the vague category of "philosophy." But this side of me has produced the purest, most profound joy that I have ever experienced, and it offers the most promise for a successful career.

On the other hand, I am a struggling Youtube addict, who enjoys hanging out with friends, good memes, and generally not doing work. This is more than low-conscientiousness, it is a fear of missing out on the shit-posty culture that I know and love.

The majority of my time goes towards the latter part of my personality, and my periods of intellectual productivity, or even just doing homework, are sporadic (I have a 3.66 GPA; it could be better/I could be getting more out of my classes.) The problem is that I feel like I'm unable to let go of the fun-loving part of me which needs stupid entertainment. I simply cannot stay interested enough in learning and knowledge to be doing it 24/7, but I feel like this is requisite in order to be a successful thinker. How do you guys stay interested in something (an idea or even an entire field) persistently enough to always be motivated to work on it? Is it unrealistic to hope to always be motivated by your curiosity? Will I burn myself out if I devote my free-time to extracurricular reading?

Sorry for the autobiography, but I don't know of a better forum to go to for these kinds of questions.
A Memetic Mediator Manifesto
by Chris_Leong — 2020-02-17T02:14:56.683Z
Link: https://docs.google.com/document/d/1McOfwLfLAc4P8nkKSSxVZwhr7gD5iqnrZA10CtNjznQ/edit#

This link discusses strategies to try to mediate disagreements between different warring tribes with fundamentally different world views. Creator unknown.
UML XI: Nearest Neighbor Schemes
by Rafael Harth — 2020-02-16T20:30:14.112Z

(This is the eleventh post in a sequence on Machine Learning based on this book. Click here for part I.)

Last time, I tried to do something special because the topic was neural networks. Now we're back to the usual style, but with an unusually easy topic. To fit this theme, it will be a particularly image-heavy post.

The idea of nearest neighbor predictors is to predict the target value of a point based on the target value of the most similar points in the training data. For example, consider a regression problem with X=R2 and Y=R, the following training data, and the new red point:

The (x,y) position of the points corresponds to their value in X=R2. The number next to them corresponds to their target value in Y=R. (The four closest points just happen to have target values 1-4 by coincidence.) The other training points also have target values, which are not shown – but since they're real numbers, they are probably things like 35.23412576354868569754543639946795856858465745654965464964534... .

This picture begs the question of how many neighbors to take into account. This is a parameter: in a k-nearest neighbor scheme, we consider the target values of the k nearest instances. In the instance above, if k=1, we have the following situation:

(Note that the circle just demonstrates which points are closest – we do not choose all points within a fixed distance.)

For k=2, we have two inputs and need to decide what to do with them. For now, we simply declare that the label of our red point will be the mean of all k neighbors.

k=3:

k=4:

If the function we're trying to learn here is something like distance from a center and the red point is precisely at that center, then the scheme gets worse and worse the larger k becomes. On the other hand, suppose the instance looks like this instead:

where the new point is an unfortunate outlier. In this case, the situation would be somewhat improved for larger k. In general, k=1 means that every point will get to decide the target values of some new training points (unless there are other training points with identical positions). Here is an image that shows, for each point, the area in which that point will determine the target value of new points given that k=1:

This is also called a Voronoi diagram, and it has some relevance outside of Machine Learning. For example, suppose the red dots are gas stations....
[Link and commentary] The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?
by MichaelA — 2020-02-16T19:56:15.963Z

This is (partly) a linkpost for

a paper

published earlier this year by Toby Shevlane and Allan Dafoe, both researchers affiliated with the Centre for the Governance of AI. Here’s the abstract:

There is growing concern over the potential misuse of artificial intelligence (AI) research. Publishing scientific research can facilitate misuse of the technology, but the research can also contribute to protections against misuse. This paper addresses the balance between these two effects. Our theoretical framework elucidates the factors governing whether the published research will be more useful for attackers or defenders, such as the possibility for adequate defensive measures, or the independent discovery of the knowledge outside of the scientific community. The balance will vary across scientific fields. However, we show that the existing conversation within AI has imported concepts and conclusions from prior debates within computer security over the disclosure of software vulnerabilities. While disclosure of software vulnerabilities often favours defence, this cannot be assumed for AI research. The AI research community should consider concepts and policies from a broad set of adjacent fields, and ultimately needs to craft policy well-suited to its particular challenges.

The paper is only 8 pages long, and I found it very readable and densely packed with useful insights and models. It also seems highly relevant to the topics of

information hazards

,

differential progress

, and (by extension) global catastrophic risks (GCRs) and existential risks. I’d very much recommend reading it if you’re interested in AI research or any of those three topics.

Avoiding mentioning GCRs and existential risks

(Here I go on a tangent with relevance beyond this paper.)

Interestingly, Shevlane and Dafoe don’t explicitly use the terms “information hazards”, “differential progress”, “global catastrophic risks”, or “existential risks” in the paper. (Although they do reference Bostrom’s paper on information hazards.)

Furthermore, in the case of GCRs and existential risks, even the concepts are not clearly hinted at. My guess is that Shevlane and Dafoe were consciously avoiding mention of existential (or global catastrophic) risks, and keeping their examples of AI risks relatively “near-termȁ...