trentmkelly/LessWrong-43k huggingface dataset:
The Results of My First LessWrong-inspired I Ching Divination
by DirectedEvolution — 2019-07-08T21:26:36.133Z

Part A - Exploratory Research

After reading "Steelmanning Divination," I was inspired to try it out. My question was "Should I begin using the I Ching for divination?" and I used IChingOnline to interpret my results.

I obtained hexagram 24, and I'm going to recapitulate my responses to the three interpretations on the site. At the end, I'll do a summary reflection. In "Part B," I heavily critique this whole experience. I think that is the most important part of this post.

1. Thunder regenerates deep within Earth's womb: Sage rulers recognized that the end of Earth's seasonal cycle was also the starting point of a new year and a time for dormancy. They closed the passes at the Solstice to enforce a rest from commerce and activity. The ruler himself did not travel.

You have passed this way before but you are not regressing. This is progress, for the cycle now repeats itself, and this time you are aware that it truly is a cycle. The return of old familiars is welcome. You can be as sure of this cycle as you are that seven days bring the start of a new week. Use this dormancy phase to plan which direction you will grow.

Response: The opening imagery does bring up a subtle emotional response. I make connections to my life: I am working and in school for significantly less hours this season. Several years ago, I was involved in a local spiritual community, which I broke away from entirely to set out on a new phase of life, one that I planned to be more rational, aligned with my values, and accomplished.

I was introduced to it in the first place by being unexpectedly asked to do a tarot reading for a woman I was on a first date with; she decided to invite me to a meeting of the spiritual community, and I stayed for several years. Hence, the second paragraph seems apropos, especially since I do not feel at all as though I'm "regressing," but rather that I'm using it in the context of new values, understandings, and purposes. It is in fact welcome. I am also in the midst of a career change involving a great deal of planning. This seems right on the money.

2. You are about to experience a rebirth -- about to be given another chance, a new lease on life. You have persevered, gone the distance through an entire cycle -- through the Spring of hope or new passion, through a Summer of growth and building, onl...
"Rationalizing" and "Sitting Bolt Upright in Alarm."
by Raemon — 2019-07-08T20:34:01.448Z

This is (sort of) a response to Blatant lies are the best kind!, although I'd been working on this prior to that post getting published. This post explores similar issues through my own frame, which seems at least somewhat different from Benquo's.

I've noticed a tendency for people to use the word "lie", when they want to communicate that a statement is deceptive or misleading, and that this is important.

And I think this is (often) technically wrong. I'm not sure everyone defines lie quite the same way, but in most cases where I hear it unqualified, I usually assume it means "to deliberately speak falsehood." Not all deceptive or misleading things are lies.

But it's perhaps a failure of the english language that there isn't a word for "rationalizing" or "motivated cognition" that is as rhetorically hefty.

If you say "Carl lied!", this is a big deal. People might get defensive (because they're friends with Carl), or they might get outraged (if they believe you and feel betrayed by Carl). Either way, something happens.

Whereas if Carl is making a motivated error, and you say "Carl is making a motivated error!", then people often shrug and go 'I dunno, people make motivated errors all the time?" And well, yeah. People do make motivated errors all the time. This is all doubly slippery if the other people are motivated in the same direction as Carl, which incentives them to not get too worked up about it.

But at least sometimes, the error is bad or important enough, or Carl has enough social influence, that it matters that he is making the error.

So it seems perhaps useful to have a word – a short, punchy word – that comes pre-cached with connotation like "Carl has a pattern of rationalizing about this topic, and that pattern is important, and the fact that this has continued awhile un-checked should be making you sit bolt upright in alarm and doing something different from whatever you are currently doing in relation to Carl."

Or, alternately: "It's not precisely a big deal that Carl in particular is doing this. Maybe everyone's doing this, and it'd be unfair to single Carl out. But, the fact that our social fabric is systematically causing people to distort their statements the way Carl is doing is real bad, and we should prioritize...
Some Comments on Stuart Armstrong's "Research Agenda v0.9"
by Charlie Steiner — 2019-07-08T19:03:37.038Z

Subject matter here.

I: Intro

I am extremely sympathetic to the program of AI safety by understanding value learning. Because of that sympathy, I have more thoughts than average prompted by Stuart Armstrong's post along those same lines.

Stuart's post mostly deals with "partial preferences," which are like simple statements of binary preference (A is better than B), but associated with a context - supposedly the "human's model" the human was using when they exhibited or stated that preference. Then the post says that you should sort these partial preferences according to meta-levels and aggregate them from the top down, updating your procedure after you finish each meta-level, eventually producing a utility function over world-histories.

Broadly, I'd say that my opinion is sort of like the bitter lesson. The bitter lesson in, say, image recognition, is that people wanted to do image recognition with a bunch of human-designed features and formal reasoning and human-understandable internal moving parts, and they tried that for a long time, and what worked was using way bigger models, way more computing power, much fewer human-understandable internal parts, and almost no human-designed features.

I like Stuart's outline more than most value learning proposals. But it still strikes me as primarily a list of human-designed features and human-understandable internal moving parts. We might be better off throwing away some of the details and abstracting in a way that allows for some of these problems to be solved by big models and computing power.

It's like the just-so story about ResNets, which is that they're a fix to humans thinking the insides of neural nets should look too much like human logic[^1]. I think speculating about the human-sized logical relationships between speculative parts inside the AI is easier but less useful than speculating about the algorithm that will connect your inputs to your outputs with a big model and lots of computing power, which may or may not have your logical steps as emergent features.

II: A long analogy about dams

If you want to design a dam, you don't draw the blueprint of the dam first and figure out what materials it should be made of later - first you learn a lot about hydrology and materials science so you know how steel and concrete and earth and water interact, then you...
[AN #59] How arguments for AI risk have changed over time
by Rohin Shah — 2019-07-08T17:20:01.998Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback;

Highlights

A shift in arguments for AI risk

(Tom Sittler): Early arguments for AI safety focus on existential risk cause by a failure of alignment combined with a sharp, discontinuous jump in AI capabilities. The discontinuity assumption is needed in order to argue for a treacherous turn, for example: without a discontinuity, we would presumably see less capable AI systems fail to hide their misaligned goals from us, or to attempt to deceive us without success. Similarly, in order for an AI system to obtain a decisive strategic advantage, it would need to be significantly more powerful than all the other AI systems already in existence, which requires some sort of discontinuity.

Now, there are several other arguments for AI risk, though none of them have been made in great detail and are spread out over a few blog posts. This post analyzes several of them and points out some open questions.

First, even without a discontinuity, a failure of alignment could lead to a bad future: since the AIs have more power and intelligence their values will determine what happens in the future, rather than ours. (Here it is the difference between AIs and humans that matters, whereas for a decisive strategic advantage it is the difference between the most intelligent agent and the next-most intelligent agents that matters.) See also

More realistic tales of doom

(

AN #50

) and

Three impacts of machine intelligence

. However, it isn't clear why we wouldn't be able to fix the misalignment at the early stages when the AI systems are not too powerful.

Even if we ignore alignment failures, there are other AI risk arguments. In particular, since AI will be a powerful technology, it could be used by malicious actors; it could help ensure robust totalitarian regimes; it could increase the likelihood of great-power war, and it could lead to stronger

competitive pressures that erode value

. With all of these arguments, it's not clear why they are specific to AI in particular, as opposed to any important technology, and the arguments for risk have not been sketched out in detail.

The post ends with an exhortation to AI safety researchers to clarify which sources of risk motivate...
NIST: draft plan for AI standards development
by ryan_b — 2019-07-08T14:13:09.314Z
Link: https://www.nist.gov/news-events/news/2019/07/nist-releases-draft-plan-federal-engagement-ai-standards-development

The National Institute of Standards and Technology, a part of the United States Department of Commerce, has released a draft plan for public comment. It was released July 02; comments are accepted until July 19; the final plan is due August 10. It is 36 pages long, including the boilerplate.

This is the early phase of the regulatory process. Whatever the plan winds up containing, it will form an important part of future decisions made by the US government.

I strongly believe it is a good idea to provide safety-related comments during this and all future public comment windows. It is not clear to me what the most influential kind of comment would be, nor is it clear to me what the best perspective on safety would be for a standards board. This causes me to expect that comments running the gamut on safety would be desirable. The default answer when a significant public concern is raised seems to be "we see this is important and will look into it going forward" which also seems (to me) to be the most-correct answer at this time.
Indifference: multiple changes, multiple agents
by Stuart_Armstrong — 2019-07-08T13:36:42.095Z

Formalism developed with Henrik Åslund.

The what and why of multiple indifference

There are certain agent designs where the agent can move smoothly from acting to optimising one utility/reward function, to optimising another. The agent doesn't object to the change, nor do they attempt to make it happen. It is indifferent to the change, because its reward is balanced to be precisely the same whether change happens or not. Originally, this was setup for a single change of utility/reward function at a single moment of time.

Here, we will present a formalism with multiple agents, all of whom are indifferent not only to changes in their own reward functions, but to changes in the reward functions of any of the other agents. We'll also generalise it to accommodate multiple changes of reward functions - maybe a different change at each timestep.

In order for these definitions to work, we will need to define some notions of counterfactuals, and some notions of optimality for several agents optimising their own separate reward functions.

Example: self-driving car races

It's clear why we would need to generalise to each timestep: it's important to be able to safely change an agent's goals more than once, as we are unlikely to get the goals perfectly right in only two tries. It's also important to generalise to multiple agents, as agents can be motivated to push humans to interrupt or not interrupt other agents. Indeed, this would be expected for a reward-maximising agent, and could lead to dangerous behaviour.

Consider the following example, adapted from this paper. Two self driving cars/agents, imaginatively called Car1 and Car2, are training for an important race later in the month.

The cars' reward functions are mainly competitive: the faster one wins.
Now, the important race will take place on a tropical race-track, but the pair are training on a cold one, where ice sometimes forms. If one of the car starts skidding, their controller takes over and remotely brakes that car, carefully, and applies indifference to that car's reward. They do this because they don't want the car to learn driving techniques that avoid ice: these techniques would be counter-productive on the "test distribution", ie the real race-track they will be competing on.

But, after a while, Car2 hits the cars hit on a sneaky tactic: forcing Car1 onto an ice patch. In that case, Car1 will get interrupt

...
Can I automatically cross-post to LW via RSS?
by  — 2019-07-08T05:04:55.829Z

I have a shortform blog elsewhere, and I'd like to cross-post the content of the posts to my personal blog here. I have it set that each new post updates an RSS feed.

On LW2.0, Is there an easy way to do cross-posting, or do I need to go mucking about in the API?
Is the sum individual informativeness of two independent variables no more than their joint informativeness? 
by Ronny Fernandez — 2019-07-08T02:51:28.221Z

Is it true that:

If I(X;Y) = 0 then I(S;X) + I(S;Y) <= I(S;X,Y)

Can you find a counterexample, or prove this and teach me your proof?

Someone showed me a simple analytic proof. I am still interested in seeing different ways people might prove this though.
Religion as Goodhart
by Shmi — 2019-07-08T00:38:36.852Z

This post is inspired by the SSC Book Review: The Secret of Our Success. To (double- or triple-)quote the relevant parts:

the best hunting strategy requires randomizing

and

If you’re generally predictable – and Scott Aaronson says you are – then outsourcing your decision to weird birds might be the best way to go.

This is a somewhat general pattern: to deal with smart adversaries effectively, randomization is, on average, the best strategy. Matching pennies and Rock/paper/scissors are some classic examples. Also, humans (and probably animals in general) are apparently poor randomizers. I assume because a built-in biological random number generator of adequate quality is expensive. I don't know if this is a good assumption. A quick online search does not refute or confirm this, and randomness is everywhere at the cellular level, but maybe not at the level of the single organisms.

Fortunately, true randomness is everywhere outside in the physical processes, and is cheap to access, and the augury and divination mentioned in the reviewed book are good examples of that. So, relying on external randomness might be an evolutionary adaptation that improves survival. (Is this so? Then we would see similar behavior in other animals' strategies for dealing with adversity, be it hunting, gathering or something else. Do we?)

Of course, the next step, consciously understanding what randomness is and why it is useful, is borderline impossible for any animal but a rather advanced human, and even then it's iffy: even statisticians are not immune to behavioral rituals. And given our awesome powers of rationalization, we justify the randomization rituals as predictive, not just useful.

And so we have a Goodhart-like situation, where a proxy becomes the goal, and is no longer a useful proxy:

The "true" goal is to improve the odds of finding or cultivating a resource necessary for survival.
The chosen proxy is to rely on an external randomizer.
The internal justification for it is that it has larger predictive powers then the person's logic or intuition.
Thus believing in an external power that is larger than oneself has evolutionary advantages and can stick, so the proxy becomes the goal, even though it gives little or no advantage in most situations that do not require randomization.
Ergo, "a God-shaped hole" in one's heart.

The above is larg...
First application round of the EAF Fund

by JesseClifton — 2019-07-08T00:20:56.565Z
Link: https://forum.effectivealtruism.org/posts/BmBpZjNJbZibLsYnu/first-application-round-of-the-eaf-fund-1

This is the first call for applications of the EAF Fund.

Logistical info

Application deadline: August 11, 2019
Funding available: $424,170
Application form: https://forms.gle/82Px7J5L7f6H3tkY8

Projects we’d like to fund

We’d like to fund projects aligned with the mission of the fund: improving the quality of the long-term future by supporting efforts to reduce risks of astronomical suffering (s-risks) from advanced artificial intelligence. You can learn more about what this means in our article on Cause prioritization for downside-focused value systems (section: “AI Alignment”).

When launching the fund, we listed specific areas we consider to be particularly important. You can also take a look at the two grants we made last year. Below we provide additional guidance on grants we’d be excited to make.

Grants for capacity-building (e.g., workshops, community infrastructure, scholarships, funding for self-study). We’re particularly interested in increasing capacity in terms of top talent and quality (e.g., nuance, depth) of output as opposed to the size of the community in general. Examples of grantees might be:

Somebody who is familiar with our research program to learn about existing AI alignment approaches or pursue a degree relevant to AI governance.
Somebody to organize an event or provide a resource that addresses a current need in the community of people focused on s-risks.

Grants that enable research in priority areas (e.g., teaching buy-outs, funding for compute, funding for conference attendance, funding for independent research). Below are examples of research topics we currently find particularly relevant.

Surrogate goals are a promising avenue for reducing downside risks from threats. Ideally, they would transform threat games to games involving threats against “surrogate goals” which would not be catastrophic if carried out. However, surrogate goals are not well-studied and a number of challenges need to be addressed. We’re expecting additional research to illuminate how and whether they are a feasible intervention for reducing s-risk.
Program games, and related ideas in machine learning such as learning with opponent-learning awareness, demonstrate how the ability of artificial agents to access one another’s source code (or policy parameters) can be used to achieve more cooperative outcomes. However, the technical study of program games and “open-source game

...
LW authors: How many clusters of norms do you (personally) want?
by Raemon — 2019-07-07T20:27:41.923Z

note: this post is about gathering data for "what might the archipelago model look like if implemented on LW", which is s separate question from "should the archipelago model be implemented on LW?"

Periodically, I've argued for applying Archipelago-style norms to LessWrong (i.e. give people the tools to establish different norms and culture, then let them experiment however they want, and let the best cultures attract participants)

One cost of this approach is that it's much harder to keep track of which space you're in, and what the rules are. This could be disorienting.

There are various ways to streamline that process. For example, we could (and I expect will, at least on desktop computers) make it so that as soon as you start typing a comment, the moderation norms for the relevant post appear next to your comment box, so it's much easier to see what norms you're expected to follow.

But if there were *hundreds* of different types of norms, this might still get a bit bewildering (in particular if each set of norms had a lot of nuance to it), and place too high a burden on commenters.

One thing I'm wondering is how many different normsets there actually are demand for, among authors. (I do think it's important to ground this out in "what authors want" rather than "what commenters want" since the authors are doing the bulk of the work, and the conversations won't happen at all without them, with the caveat that I think it's a fine outcome for an author to write a post with one set of comment-norms, and someone who prefers other norms to write up a post titled "Discussing Bob's Post X" that sets different norms).

Knobs that I could imagine an author wanting to turn include:

Do you want feedback on writing style?

(possibly with different answers for typos, high level structure, etc)

Is there a particular conversation you're trying to have? Is there a product you're trying to build? Or are your mostly just throwing your ideas out into the wild and see what happens?
Should commenters relate to your post as if it's in a playful/creative/exploratory stage, or more of a finished product that you want judged professionally (or somewhere else on a spectrum?
Do you prefer that you (and each commenter) be responsible for justifying their ideas, or do you prefer comm

...
How to make a giant whiteboard for $14 (plus nails)
by eukaryote — 2019-07-07T19:23:38.870Z
Link: https://eukaryotewritesblog.com/2019/07/07/a-giant-whiteboard-for-14-plus-nails/

All my friends love whiteboards, because they’re giant nerds. It’s only a good party when somebody starts trying to illustrating a point on the whiteboard. Also, to-do lists you can’t miss.

This isn’t my idea, but it’s good and I thought I’d share it. Home Depot and Lowes sell a material called “thrify white panelling” or “smooth white hardboard”. It’s cheap – mine was about $14 for a 4-foot-by-8-foot sheet. I asked the store staff to cut the panel in half so it would fit in my car, which they did for free.

It’s easily chipped, so watch out if you want it to look flawless.

Then I stuck it to the wall of my studio apartment. I used some kind of drywall anchor to screw it on, with washers, to get a good grip on the material. (I think using a drywall anchor was overkill, and can’t recommend them for renters until I see how cleanly they come out of the wall – so if you have a better low-damage attachment system, maybe try that first. I wasn’t able to get them to stick with Command velcro strips – the strips kept detaching from the panel material – but other people on the internet seem to have found success in this.)

It acts as a whiteboard as-is. Writing gets hard to erase if you leave it up there for a long time, but you can clean it with whiteboard-cleaning-solution, alcohol, or by coloring over the marks with another whiteboard marker and then erasing that. Internet people also report that you can buff the entire surface with turtle wax before hanging it, and this makes it more stain-resistant.

I didn’t do that, and it’s still pretty good, especially for $14.

Let me know if you try this!

(Possibly relevant: Andrew Critch's giant notepad for better thinking)