trentmkelly/LessWrong-43k huggingface dataset:
How has rationalism helped you?
by Sunny from QAD — 2019-08-24T01:31:06.616Z

This November, I will be participating in NaNoWriMo, an online event where participants have one month to write a 50,000-word manuscript for a novel. I'm fairly settled on the idea that I'm going to write about a person who is fairly smart, but who has no rationalist training, discovering rationalism and developing into a fully-fledged rationalist.

I'm looking for inspiration for what kind of problems they might learn to solve. How has rationalism helped you? There is no answer too big or too small. If rationalism helped you realize that you needed to divorce your spouse and change careers, that's a good answer; if rationalism changed the way you tie your shoelaces, I'm all ears. In particular, I'd like to hear:

The thing you used to do, which was lacking in some way.
The rationalist concept that challenged your habit.
What you do now.
Is LW making progress?
by zulupineapple — 2019-08-24T00:32:31.600Z

Some people have an intuition that with free exchange of ideas, the best ones will eventually come out on top. I'm less optimistic, so I ask if that's really happening.

The alternative would be to have the same people talking about the same problems without accumulating anything. People would still make updates, but some would update in opposite directions, leaving the total distribution of ideas in the community largely unchanged. There would be occasional great ideas, but they would soon get buried in the archives, without leaving much of an impact. I have some hope we're not living in that world and I want to see evidence to justify it.

One way to look at the question is to consider the sequences. They are 10 years old now. Are they in some way outdated? Are there some new and crucial ideas that they are missing?
Is there a simple parameter that controls human working memory capacity, which has been set tragically low?
by Liron — 2019-08-23T22:10:40.154Z

Here's the kind of thing I mean by "human working memory capacity" being "set tragically low": A 100-word sentence introducing a new concept is often annoyingly hard to wrap one's head around, compared to a longer and more "gentle" explanation.

The concept of working memory seems like a useful reduction of part of what makes intelligence work:

Most of us here think human intelligence can one day be increased a lot, because there doesn't seem to be a fundamental limit to intelligence located anywhere near human-level, but that's a non-constructive reductio type of argument.

On the other hand, it seems like there's an obvious constructive argument to why working memory should be able to be increased: Take whatever set of structures encode the current short-term brain's working memory contents, and analyze how big they are, like how many copies of some lower-level storage unit they have, and imagine architecting the brain to have more such copies of those systems encoding more memory contents.

Getting back to my question...

We all wish we were smarter, but we suspect that there are no easily-tweakable parameters that would yield a big increase in intelligence, or natural selection would have already tweaked them.

But doesn't it feel like working memory is an easily-tweakable parameter, and natural selection just didn't tweak it because it was good enough relative to the rest of the brain and the needs of the species, but it could actually be tweaked pretty easily in a couple thousand generations if there were the selection pressure to do it?

The reason it might make sense to say our small working memories are particularly tragic is if there is, as I suspect, a smaller counterfactual distance to them being larger and much more useful for modern life.
Optimization Provenance
by Adele Lopez — 2019-08-23T20:08:13.013Z

Transparency is vital for ML-type approaches to AI alignment, and is also an important part of agent foundations research. In this post, we lay out an agenda for formalizing transparency which we'll call the Optimization Provenance Agenda.

In particular, the goal is to create a notion of transparency strong enough that an attempted deception would be completely transparent. The basic idea is that at any point, not only should an agent's world model and subgoals be legible, but the entire provenance of all of the optimization processes which are part of the agent should be legible as well.

This agenda is a joint development between me and Evan Hubinger. Special thanks to Hjalmar Wijk and Evan Hubinger for their comments and feedback on this post!

Background notions

In order to discuss the notions here, it will be helpful to have working definitions of the key concepts involved.

Legibility

Intuitively, legibility means that a human should be able to look at something, and be able to understand it easily and correctly. If the thing in question is very large, we might at best only be able to have local legibility, where any given part below a certain size is legible, but the whole thing is not understandable by a human. In an amplification scenario, local legibility may be sufficient, with an amplified human being capable of understanding the global structure. For the purposes of this post, we'll consider legibility to also include these cases. One major issue with the concept of legibility is that it seems very difficult to create something legible through a legible process.

It seems plausible to me that existing ML techniques could be combined and extended to produce natural language descriptions of learned world models. However, this process itself would most likely be very illegible. Even in the case of human communication, it is possible for a human to produce a legible plan, but humans do not seem to be very capable of producing a legible explanation of the process which produced that plan. So it seems likely to me that we may have to decide some illegible process to trust in order to get off the ground with this approach. This could simply be trusting illegible human mental processes, or it could be something like trusting models produced in a mathematically simple way.

World model

A world model belongs to a decision making process, and is used by the process ...
Troll Bridge
by abramdemski — 2019-08-23T18:36:39.584Z

All of the results in this post, and most of the informal observations/interpretations, are due to Sam Eisenstat. I think the Troll Bridge story, as a way to make the decision problem understandable, is due to Tsvi; but I'm not sure.

Pure Logic Version

Troll Bridge is a decision problem which has been floating around for a while, but which has lacked a good introductory post. The original post gives the essential example, but it lacks the "troll bridge" story, which (1) makes it hard to understand, since it is just stated in mathematical abstraction, and (2) makes it difficult to find if you search for "troll bridge".

The basic idea is that you want to cross a bridge. However, there is a troll who will blow up the bridge with you on it, if (and only if) you cross it "for a dumb reason" — for example, due to unsound logic. You can get to where you want to go by a worse path (through the stream). This path is better than being blown up, though.

We apply a Löbian proof to show not only that you choose not to cross, but furthermore, that your counterfactual reasoning is confident that the bridge would have blown up if you had crossed. This is supposed to be a counterexample to various proposed notions of counterfactual, and for various proposed decision theories.

The pseudocode for the environment (more specifically, the utility gained from the environment) is as follows:

IE, if the agent crosses the bridge and is inconsistent, then U=-10. (□⊥ means "PA proves an inconsistency".) Otherwise, if the agent crosses the bridge, U=+10. If neither of these (IE, the agent does not cross the bridge), U=0.

The pseudocode for the agent could be as follows:

This is a little more complicated, but the idea is supposed to be that you search for every "action implies utility" pair, and take the action for which you can prove the highest utility (with some tie-breaking procedure). Importantly, this is the kind of proof-based decision theory which eliminates spurious counterfactuals in 5-and-10 type problems. It isn't that easy to trip up with Löbian proofs. (Historical/terminological note: This decision theory was initially called MUDT, and is still sometimes referred to in that way. However, I now often call it proof-based decision theory, because it isn't centrally a UDT. "Modal DT" (MDT) would be reasonable, but the modal operator involved is the "provability" operator, so "proof-b...
Understanding understanding
by mthq — 2019-08-23T18:10:36.796Z

How does 'understanding' a problem work, and when do we feel like we understood an explanation or proof? Having an opaque long formal proof often feels insufficient, similarly some arguments feel unsatisfying because they contain many subjectively arbitrary choices.
An explanation is a sequence of claims corroborating a statement. A reasoner understands an explanation (or rather has more understanding of an explanation) when each claim has low surprisal given its mental state after reading all of the previous statements, starting with the problem statement. The aggregate of the surprise over the entire explanation indicates how poorly understood it is. The measure of surprisal is essentially about the reconstructability of the explanation using something like Monte Carlo Tree Search over all possible explanations informed by the mental states of the reasoner.
The surprisal of claims is roughly how long it would take for the reasoner to come up with the claim given its current mental state. Since the space of such claims is exponential in the length of the claim the reasoner has to use some form of attention to guide its search. We can model this attention mechanism by an ontological graph. Such a graph encodes the collection of mental concepts and associative links between them. The mental state of the reasoner is an activation of an ensemble of concepts and the associative links make available other concepts, lowering their surprisal when invoked in the next step of the explanation.
When a step in an explanation is highly surprising some understanding is needed. The reasoner does this by modifying its ontology, creating new concepts or creating associations that make the step more obvious. I call such modifications insights, a good insight gives clarity and makes steps obvious and subjectively trivial.
To examine this situation consider the mutilated chessboard problem[1] and subsequent explanation:

Suppose a standard 8×8 chessboard has two diagonally opposite corners removed, leaving 62 squares. Is it possible to place 31 dominoes of size 2×1 so as to cover all of these squares?
The answer that it is impossible. Opposite corner squares always have the same color so that there are 30 and 32 white or black squares left. A domino placed on the chessboard will always cover one white square and one black square. So there will always be either two white or two black s

...
Actually updating
by SaraHax — 2019-08-23T17:46:01.924Z

Actually updating can be harder than it seems. Hearing the same advice from other people and only really understanding it the third time (though internally you felt like you really understood the first time) seems inefficient. Having to give yourself the same advice or have the same conversation with yourself over and over again also seems pretty inefficient. Recently, I’ve had significant progress with actually causing internal shifts, and the advice.. Well, you’ve probably heard it before. But hopefully, this time you’ll really get it.

Signs you might not be actually updating.

You do some focusing, and you discover a problem and you talk to a part of yourself and work through it. Then a week later, you find yourself having the same conversation with that part of you.
You change your mind about doing something but find that suspiciously your behaviour is not changing or doesn’t change for that long.
Someone says “X” and you go “oh yes, X”. Then, some time passes. Then someone else says “X” and you go “ohhh yes, X, now I get it”. Then, some time passes. Then someone else says “X” and you go “ohhhh yes of course, X, now I really get it”. Etc.
You feel like you believe X but yet keep finding yourself behaving as though you don’t believe X.
You feel like your rationalisations are running so deep that you can’t seem to catch them. You're doing things that seem to make sense but then something might happen that makes you feel like your behaviour was actually driven by a 'rationalisation'. To make this clearer, imagine something happens (event Y) that changes your opinion of X. Maybe you realise that the thing 'shouldn't' be much evidence at all. You decide it will only slightly affect your opinion. You may end up with a certain conclusion by looking at old evidence and this thing Y (that you have actively decided to only weight a small amount). Then, something may then happen that makes you really believe that thing Y is small evidence. Suddenly, you are looking at the same evidence and this thing Y (weighted the same amount as you tried to weight it before), but you have a different conclusion. It's like you are looking at the evidence through a different 'lens'. It seems like you are able to exploit uncertainty around evidence to get differing conclusions depending on this 'lens'. This whole thing might ma

...
When do utility functions constrain?
by Hoagy — 2019-08-23T17:19:06.414Z

The Problem

This post is an exploration of a very simple worry about the concept of utility maximisers - that they seem capable of explaining any exhibited behaviour. It is one that has, in different ways, has been brought up many times before. Rohin Shah, for example, complained that the behaviour of everything from robots to rocks can be described by utility functions. The conclusion seems to be that being an expected utility maximiser tells us nothing at all about the way a decision maker acts in the world - the utility function does not constrain. This clashes with arguments that suggest, for example, that a future humanity or AI would wish to self-modify its preferences to be representable as a utility function.

According to Wikipedia’s definition, a decision maker can be modelled as maximising expected utility if its preferences over mutually exclusive outcomes satisfy completeness, transitivity, continuity and independence. This is neat. However, it still leaves us with free choice in ordering over our set of mutually exclusive outcomes. The more outcomes we have, the less constrained we are by utility maximisation and when we look closely there are often a LOT of mutually exclusive outcomes, if we give ourselves truly free range over the variables in question.

In the context of thought around AI safety, introductions to these axioms have not denied this but added a gloss of something like

‘if you have a consistent direction in which you are trying to steer the future, you must be an expected utility maximizer’,

in this case from Benya's post in 2012. Here the work done by ‘consistent direction’ seems to imply that you have reduced the world to a sufficiently low dimensional space of outcomes. Though I agree with this sentiment, it seems fuzzy. I hope to add some clarity to this kind of qualification. To put it another way, I want to see what restrictions we need to add to regain something like the intuitive notion of the maximiser and see if it is still sufficiently general as to be worth applying to theoretical discussions of agents.

Pareto Frontier of Concepts

A quick aside: Why do we want to use the idea of a utility maximiser? The explanation that comes to mind is that it is felt to be a concept that lies on the Pareto frontier of a trade-off between the explanatory power of a concept and the lack of troubling assumptions. I doubt this framing is ne...
Parables of Constraint and Actualization 
by Spencer Wyman — 2019-08-23T16:56:16.247Z

I have an intuition that physics, logic, and computation are all closely connected, and that their bounded versions are also closely connected. I think this intuition started as wishful thinking. I didn't have time to study all three of my favorite subjects, so I hoped that studying one would eventually lead to all the others. The rest of this post aims to convince you (and me) that it wasn't just wishful thinking.

--------------------------------------------------------------------------------

Imagine you live in the world of mathematics. The things around you are groups and manifolds and numbers. You can't touch anything. You can touch symbols floating around the objects, and only by moving these symbols around can you interact with the objects. This is Logic. Eventually you notice that the symbols are made of the same stuff as everything else, and have their own symbols. By thinking hard and manipulating these symbol symbols very carefully, you find a way to get the symbols to manipulate themselves. This is a Computer and with it you interact with the world more easily. Logic actualizes computers.
Imagine, in your platonic world, you come across a library which claims to contain proofs of every true fact of arithmetic, and only true facts. You also discover that you can build a Computer out of not just logic, but also the numbers around you. Since it's made of numbers, the library should know about it. You teach the computer to ask the library for proofs that it will finish computing and halt, and for proofs that it will run forever. You also teach the computer to disobey the library. If the library tells the computer it will halt, it should run forever. If the library tells the computer it will run forever, it should halt. If the library gives you any proof, it will have lied. If it doesn't, then there are facts it does not know, so it lied at the beginning. The library fades away, it was an illusion all along. Computers restrain logic.
Imagine you grow tired of your platonic world and you want to build a universe with planets and people and everything else. What could you build it out of? The only thing at your disposal is math. You teach your computer what numbers are needed to represent each bit of stuff, and how every bit of stuff will change at every step, then turn it on. You watch as the platonic world of numbers transmutes into the world of physics...
Thoughts on Retrieving Knowledge from Neural Networks
by Jaime Ruiz — 2019-08-23T16:41:35.818Z

Disclaimer: this post is a collection of preliminary thoughts around a specific question for MSFP'19 blogpost writing day. I do not intend to present novel results or define a research agenda.

1. Motivation

The capital of Mozambique is Maputo. This is a fact about the world I learned in school many years ago. Some part of my brain must have changed in order to store this fact and allow me to remember it today. It may be possible, in principle, to recover this fact by simply looking at my brain.

In this post, I consider an analogous topic in the context of Artificial Intelligence: is it possible to retrieve the knowledge stored in a neural network? This line of enquiry could be a step towards more transparent AI.

2. Query-Based Network Analysis

Suppose we train a RL agent in the virtual sandbox environment Minetest. In this environment, agents can use tools called pickaxes to efficiently "mine" and collect objects made of stone. A pickaxe is an example of a tool. Tools have some properties that distinguish them from other items (e.g. tools cannot be "stacked" in the inventory).

We may try to retrieve the knowledge represented in the internal state of the agent by means of queries. For instance: at which point, during training, does the RL agent learn that a pickaxe is a tool?

2.1 Clarifying the Query

It seems plausible to me that this question could be asked "as it is" by an observer in order to gauge the agent's understanding of its environment. That is to say, at the moment of posing the question, it seems perfectly meaningful. However, further thought reveals some vagueness in it: what exactly is a pickaxe? What exactly is a tool? What does it really mean for a pickaxe to be a tool? It is funny how easily one can become confused with words like "really" and "exactly".

In this example, it seems plausible to interpret the question in reference to the source code of the Minetest engine. Let us assume, for the sake of the argument, that the source code uses an object-oriented approach, and contains a Tool class, and a Pickaxe class which is a subclass of the former. The fact "a pickaxe is a tool" can be recast as a claim that the Pickaxe class implements all relevant aspects of the Tool class.

When we ask similar queries for agents that learn in the real world (e.g. "when has a system learned the fact that the capital of Mozambique is Maputo?"), formalising the ...
Algorithmic Similarity
by LukasM — 2019-08-23T16:39:47.706Z

The Problem

If someone asked you and me to each write a python program to solves some novel problem then it wouldn't be surprising if our solutions looked quite different. If we then tried explaining to the other person the way our own program worked, one thing that might happen is that we would realize that in spite of surface differences in the way that we wrote it the solutions were actually quite similar. We might have called variables different names, done some steps in a different order and so on, but both programs worked for the same reasons.

Another thing that might happen is that even after really understanding how the other persons program worked the solutions still seemed different. Maybe each program would seem to rely on different mathematical facts. Maybe you would realize that the way I saw the problem was fundamentally different from the way you saw it, you'd never thought about it like that before, and so the approaches I considered were in a totally different regime from the ones you considered. Yours might run asymptotically faster than mine, in a way such that even if I tried to cut out redundancies and optimize my algorithm I could never make it run as fast as yours without fundamentally changing how it worked. Mine might generalize better in some direction, whereas yours might not generalize as well or maybe generalize in a different direction.

This is the kind of intuition that a formal theory of algorithmic similarity should seek to capture. Given two Turing machines that compute the same function it should be able to tell us to what degree they use the same or different algorithms. Exactly how course or fine-grained the classification should be is unclear, and maybe different versions could work with different levels of detail, but it should capture some of our intuitive notions of algorithmic similarity, e.g. that changing the name of a variable generally doesn't change the algorithm, whereas there are fundamental differences between the sieve of Erathostenes and the AKS primality test.

A different problem that might be the same problem or might be a different but related problem is the question of whether one program uses/contains another program.

A Variant

Consider a simple program that outputs 1 if x>y and else outputs 0. Call this program F1. And lets suppose that I know how to compute x-y when x>y or x=y. Then I might write ...