trentmkelly/LessWrong-43k huggingface dataset:
Probabilities, weights, sums: pretty much the same for reward functions
by Stuart_Armstrong — 2020-05-20T15:19:53.265Z

This post is a more minor post, that I'm putting up to reference in other posts.

Probabilities, weights, and expectations

You're an agent, with potential uncertainty over your reward function. You know you have to maximise

0.5R1+0.5R2

where R1 and R2 are reward functions. What do you do?

Well, how do we interpret the 0.5? Are they probabilities for which reward function is right? Or are they weights, telling you the relative importance of each one? Well, in fact:

If you won't be learning any more information to help you distinguish between reward functions, then weights and probabilities play the same role.

Thus, if you don't expect to learn any more reward function-relevant information, maximising reward given P(R1)=P(R2)=0.5 is the same as maximising the single reward function R3=0.5R1+0.5R2.

So, if we denote probabilities with in bold, maximising the following (given no reward-function learning) are all equivalent:

0.5R1+0.5R21(0.5R1+0.5R2)0.25R1+0.25R2+0.5(0.5R1+0.5R2)0.5(1.5R1−0.5R2)+0.5(1.5R2−0.5R1)

Now, given a probability distribution pR over reward functions, we can take its expectation E(pR). You can define this by talking about affine spaces and so on, but the simple version of it is: to take an expectation, rewrite every probability as a weight. So the result becomes:

If you won't be learning any more information to help you distinguish between reward functions, then distributions with same expectation are equivalent.

Expected evidence and unriggability

We've defined an unriggable learning process as one that respects conservation of expected evidence.

Now, conservation of expected evidence is about expectations. It basically says that, if π1 and π2 are two policies the agent could take, then for the probability distribution pR,

E(pR ∣π1)=E(pR ∣π2).

Suppose that pR is in fact riggable, and that we wanted to "correct" it to make it unriggable. Then we would want to add a correction term for any policy π. If we took π0 as a "default" policy, we could add a correction term to pR∣π:

(pR∣π)→(pR∣π)−E(pR∣π)+E(pR∣π0).

This would have the required unriggability properties. But how do you add to a probability distribution - and how do you subtract from it?

Bur recall that unriggability only cares about expectations, and expectations treat probabilities as weights. Adding weighted reward functions is perfectly fine. Generally there will be multiple ways o

...
Race for the Galaxy: 9 Choose 6
by jefftk — 2020-05-20T12:10:02.444Z

Spending a lot of time with

only

housemates, we've ended up playing a
lot of

Race for the
Galaxy. It's a card game that feels like a

German board game,
and it pulls the best part of solitaire games (minimal waiting for
others, with simultaneous play) into a game where your interaction
with other players is critically important.

The rules for
the game specify that everyone starts with one of a few
"start worlds", and then chooses four of six cards to be their initial
hand. Picking through the deck to find all the start worlds is
annoying, however, and you get bored of the small number of starts.
My friend group ended up with a different approach, where you each get
a hand of nine cards, choose six to keep, and don't have a start
world.

This is strategically much more interesting than the regular game.
You're looking at more cards, so you're more likely to be able to find
a strong fit, and more unusual fits become more likely. Figuring out
whether to keep a valuable card and build your strategy around it, or
discard it to pay for other things is even more critical.

We play that on your first turn you have to play Explore,
Develop, or Settle. In a two-player game this only
applies to one of your two choices. At times we've played without this
restriction, but playing Consume Trade or Produce first
turn, hoping to take advantage of someone else's Settle, makes
the game higher variance for minimal additional fun.

Here's an example initial hand, the most interesting of three random
starting hands I dealt:

I would probably start by discarding the three ones marked (below) "d"
to get my opening six, and then play Develop to put down
Contact Specialist for zero. If someone else played
Settle I would piggyback with Rebel Warrior Race, using
the two marked "m" as money. Alternatively, I could figure I had an
unusually coherent hand and that losing either Genes world
would be bad, and so I should start with Explore +1+1 and be
ready to piggyback on a Develop or Settle. I could also
see keeping Smuggling Lair instead of Research Labs and
going Contact Specialist to Smuggling Lair:

Over the last few
weeks I've been getting a much stronger handle on the game, and I feel
like I'm playing in a much more focused way than I have before. I
think a lot of it is playing so many games, but also having opponents
who've also gotten a lot...
[Meta] Three small suggestions for the LW-website
by B Jacobs — 2020-05-20T11:18:38.930Z

1. A way to 'unlist' posts

A while back I wrote a post about a new system of utilitarianism that in retrospect was kinda bad. Later I wrote a different post with a new system that expanded on the old idea and that was generally better in every single way. Now I had the dilemma of what to do with the old post. I want this site to exhibit quality posts which the old one wasn't, but I also want to document my thought process and have an archive of how ideas evolved. I compromised by deleting the old post but having a link to a pdf-file of the old idea in the new post.

A possible middle-ground would be a feature similar to youtube's 'unlisting'. The link to the post still works so it's all still documented and you can link to it if you want, but it doesn't directly show up in the search results. This will make your personal page (and LW in general) more 'cleaned up'.

2. Strikethrough and/or Retracting posts

You can retract comments with a simple click of the button but not posts, why? You can manually edited your post and use strikethrough to cross stuff out, but it is not readily available to you and you'll have to search the site to find out how to do that (it's two '~' before and two '~' after a word). Even then it is not easy to get rid of a strikethrough once you added it. None of these are impossible hurdles to jump over, but why should there be any hurdles at all?

3. A way to delete a tag

On one post I accidentally added a tag that was not really relevant to post. I wanted to delete the tag but found out it was impossible. Maybe a way to delete a tag that you yourself added (and no-one else agrees with) could prevent cluttered and/or wrong tags on posts?

Thoughts?

EDIT: I will add one microscopic nitpick to the list. A small spelling error:

1 comments
Source code size vs learned model size in ML and in humans?
by riceissa — 2020-05-20T08:47:14.563Z

To build intuition about content vs architecture in AI (which comes up a lot in discussions about AI takeoff that involve Robin Hanson), I've been wondering about content vs architecture size (where size is measured in number of bits).

Here's how I'm operationalizing content and architecture size for ML systems:

content size: The number of bits required to store the learned model of the ML system (e.g. all the floating point numbers in a neural network).

architecture size: The number of bits of source code. I'm not sure if it makes sense to include the source code of supporting software (e.g. standard machine learning libraries).

I tried looking at the AlphaGo paper to see if I could find this kind of information, but after trying for about 30 minutes was unable to find what I wanted. I can't tell if this is because I'm not acquainted enough with the ML field to locate this information or if that information just isn't in the paper.

Is this information easily available for various ML systems? What is the fastest way to gather this information?

I'm also wondering about this same content vs architecture size split in humans. For humans one way I'm thinking of it is as "amount of information encoded in inheritance mechanisms" vs "amount of information encoded in a typical adult human brain". I know that Eliezer Yudkowsky has cited 750 megabytes as the amount of information in the human DNA, and also emphasizes that most of this information is junk. This was in 2011 and I don't know if there's a new consensus or how to factor in epigenetic information. There is also content stored in genes, and I'm not sure how to separate out the content and architecture in genes.

I'm pretty uncertain about whether this is even a good way to think about this topic, so I would also appreciate any feedback on this question itself. For example, if this isn't an interesting question to ask, I would like to know why.
How to learn from a stronger rationalist in daily life?
by Thomas Kwa — 2020-05-20T04:55:51.794Z

As mentioned in my previous post, I currently estimate that I am about as strong in applied rationality as the average CS undergrad. But I know Mark Xu, who has gone to CFAR, and is certainly stronger than me, enough to occasionally find interventions that can fix my problems. I don't want my friendship to become a formal mentorship, and the gains from taking advice or copying someone else's interventions into one's own life run out quickly. There's even the risk that I stop thinking for myself when my best judgment is on average more wrong than base rates + someone else's opinion.

Personal details aside, are there exercises/drills in applied rationality where one participant is much stronger than the other, but doesn't have CFAR instructor-level skills? I'm especially interested in examples that work in situations similar to mine, both for selfish reasons and because I suspect this situation is fairly common. Even more valuable would be a general framework.
Signaling: Why People Have Conversations
by liam_hinzman — 2020-05-20T03:18:01.121Z
Link: https://liamhinzman.com/blog/signaling

How would you convince somebody that you’re an expert in quantum mechanics?

If you’re not an expert in this field, you shouldn’t be able to effectively apply any method you can imagine. If you could, then it wouldn’t be a useful signal.

If I told you about Schrodinger’s cat to you right now, you probably wouldn’t think I’m a quantum physicist. If this wasn’t the case anybody could Google and trick you into thinking they more than they do.

Signals are actions that demonstrate evolutionarily desirable traits or possessions, such as physical fitness, material wealth, or knowledge of quantum mechanics - which is seen as a signal of intelligence, a trait that is desirable to potential allies and mates.

For a signal to be seen as honest, it has to be costly and public.

Effective signals are easy to do if they are honest and reflect traits or material you possess, and costly if not. If I actually was an expert in quantum mechanics, then showing you research papers that I’ve written would be a good signal.

Signals have to be public to benefit the one signaling. Donating $10,000 to fight breast cancer won’t convince anybody of your empathetic and pro-social nature unless people know about it.

Signaling is a primary motive behind many of our behaviours such as:

Conversation: talking with others lets us signal our utility as a potential ally.
Charity: donating shows that you have an excess of wealth, and that you’re somebody who will help their allies.
Consumption: sharing that you went on an expensive vacation signals to others that you have an abundance of material wealth.

Conversations are Opportunities to Signal

(This view of signaling and conversation is mostly a summary of a chapter from The Elephant in the Brain by Hanson and Simler. For a more in-depth treatment of signaling and how it effects human behaviour I'd highly recommend reading it)

Without signaling, the usual explanation for why people have conversations is that it serves as a way to trade information. I tell you about the nearby berry bush, you tell me about the dangerous lion to the West. We cooperate through conversation, and we’re better off for it.

There are a few problems with this line of reasoning.

People like talking more than listening. If we participated in conversations primarily to gain information, then you’...
Announcing Progress Studies for Young Scholars, an online summer program in the history of technology
by jasoncrawford — 2020-05-20T00:52:53.706Z
Link: https://rootsofprogress.org/announcing-progress-studies-for-young-scholars

I’m thrilled to announce a new online learning program in progress studies for high school students: Progress Studies for Young Scholars.

Progress Studies for Young Scholars launches in June as a summer program, with daily online learning activities for 6 weeks. We’ll be covering the history of technology and invention: the challenges of life and work and how we solved them, leading to the amazing increase in living standards over the last few centuries. Topics will include the advances in materials; automation of manufacturing and agriculture; the progression of energy from steam to oil to electricity; how railroads, cars and airplanes shrank the world; the conquest of infectious disease through sanitation, vaccines, and antibiotics; and the rise of computers and the Internet. The course will also prompt students to consider the future of progress, and what part they want to play in it.

The program will be guided self-study, with daily reading, podcasts or video. Students can go through the material entirely on their own for free, or pay to join a study group with an instructor for daily discussion and Q&A. Pricing to be announced soon, but scholarships will be available!

In conjunction, we’re launching a speaker series of talks and interviews with experts in the history of progress, and those at the frontier pushing it forward. Speakers will include Tyler Cowen, Patrick Collison, Max Roser, Joel Mokyr, Deirdre McCloskey, Anton Howes, and many more.

This is a joint project between The Roots of Progress and Higher Ground Education, the largest operator of Montessori and Montessori-inspired schools in the US. I’ve known the leadership team at Higher Ground for many years and have deep respect for them—especially the way they treat learning as a process of self-creation on the part of the student.

Sign up to get announcements about the program, including the speaker series:

progressstudies.school

And please spread the word, especially to intellectually curious teenagers and their parents!
How to convince Y that X has committed a murder with >0.999999 probability? 
by Colin Tang — 2020-05-19T22:55:26.191Z

Suppose X has murdered someone with a knife, and is being tried in a courthouse. Two witnesses step forward and vividly describe the murder. The fingerprints on the knife match X's fingerprints. In fact, even X himself confesses to the crime. How likely is it that X is guilty?

It's easy to construct hypotheses in which X is innocent, but which still fit the evidence. E.g. X has an enemy, Z, who bribes the two witness to give false testimony. Z commits the murder, then plants X's fingerprints on the knife (handwave; assume Z is the type of person who will research and discover methods of transplanting fingerprints). X confesses to the murder which he did not commit because of the plea deal.

Is there any way to prove to Y (a single human) that X has committed the murder, with probability > 0.999999? (Even if Y witnesses the murder, there's a >0.000001 chance that Y was hallucinating, or that the supposed victim is actually an animatronic, etc.)
Haskenthetical
by philh — 2020-05-19T22:00:02.014Z

This is a toy language I've been designing, or at least implementing, for about a year.

It's another take on "Haskell with a Lisp syntax". I'm aware of prior art: Hackett, Axel and Liskell. I haven't looked closely at any of them, because doing that seemed like it might make me less likely to keep working on Haskenthetical.

I call it "toy" in the sense of… well, right now it's a toy like an RC plane is a toy. But my vague goal is to make it into a toy like the Wright flyer would have been a toy if it had been built in 2003. I'd like to get it, say, 80% of the way to being a "real" language. I have no intention or expectation of taking it the other 800% of the way. I have no intention or expectation of taking on responsibility-according-to-me here.

(And honestly, even the first 80% is super ambitious. I don't expect to get that far, it would just be nice to. If I never touch the project again after this, I won't consider my time wasted.)

If you're curious, the source code is available here¶1. If you have stack, stack build --fast should suffice to build and then stack exec -- haskenthe -h to run the executable.

So far I've implemented basic Hindley-Milner type inference, the ability to define new types, and pattern matching. The only built-in types are ->, Float (which is a Haskell Double under the hood), and String (a Haskell Text). Builtin functions are +, - and * (all of type -> Float (-> Float Float) and err! (of type -> String $a). I don't yet have division because I haven't decided how I want to handle division by 0. I don't expect I'll come up with anything particularly exciting there, but also I haven't felt the need for division yet.

(Actually, the types Either and , are also builtin, along with functions to work with them: constructors Left, Right and ,, and destructors either, car and cdr. But that's just because I added them before I added custom types, and I haven't bothered to remove them yet.)

I have a long list of things I'd like to include in future. Probably the ones that interest me most right now are macros2, extensible records, and compilation. I don't know how macros and compilation are supposed to fit together, but I have some vague ideas in my head. And clearly it's been done in the past, so I assume I can find out how.

Other things include IO, comments, imports, exhaustiveness checking and FFI. Maybe typeclasses, but I'm curious w...
A Problem With Patternism
by B Jacobs — 2020-05-19T20:16:54.835Z

Patternism is the belief that the thing that makes 'you' 'you' can be described as a simple pattern. When there is a second being with the same mental pattern as you, that being is also you. This comes up mostly in the debate around mind-upoading, where if you make a digital copy of yourself, that copy will be just as much you as the current you is.

Question: How do we quantify this pattern?

And I don't mean how do we encode epigenetic information or a human mind into a string of binary numbers (even though that is a whole issue in and of itself), I will assume in this post that's all perfectly doable. How will we quantify when something is no longer part of the same general pattern?

I mean when we have a string of binary numbers with a couple of ones switched to zeros we can just tally up the errors and determine how much percent is different from the copy. But what if there is stuff missing or added? What if there is a big block of code that is duplicated in the copy? Should we count that as a one error or dozens? What if there are big chunks of codes that appear in both copies but in different places? Or in a different order? Or interspersed into the rest of the code?

But even in the first example are problems, not all ones and zeros are created equal. We care a lot about wether certain features of our being are switches 'on' or 'off' but not so much for others. Do we have to compare someone's personal desires? How do we quantify that? Or should we quantify what most people would deem an important change? Why? How? I fear that there is no real way to do this objectively and since there will always be small mutations/errors in copying you can never know which of the other "you's" is the most like you. Which I think is a pretty heavy blow for patternism.

EDIT: Apparently that last sentence caused some confusion so let me clarify. I'm not saying it's a blow for the truthfulness of the theory, since that is just a matter of definition (and I'm not interested in disputing definitions). I'm saying it's a blow for the usefulness of the theory since it doesn't help us generate new insights by making new and accurate predictions.
Bag-Drying Clips
by jefftk — 2020-05-19T14:00:02.481Z

With the amount of

buying
from the commercial supply chain that we've been doing, I've been
needing to divide up a lot of large containers. We typically use
ziplock bags, and they're sturdy enough that they could be washed and
reused. We mostly haven't been reusing them, though, because while I
don't mind washing them they're a pain to dry. They're too flimsy to
stay open by themselves, and if they get bumped shut they tend to stay
shut.

I decided to attach some clothespins to the underside of the shelf
above our sink to hang drying bags from. It was pretty quick. First
I separated some clothespins, and drilled a small hole in the center
of the half that still had the metal spring attached:

Then I screwed through it into the shelf above, making sure to use a
screw short enough that it wouldn't poke through:

Once it was snug, I slid the other half of the clothespin back on:

Here are some bags drying:

Comment via: facebook
Learning and manipulating learning
by Stuart_Armstrong — 2020-05-19T13:02:41.838Z

1 Introduction

This post will introduce our new paper "Pitfalls of Learning a Reward Function Online", now online for IJCAI2020.

It shows some of the difficulties with things we might think of as "preference learning processes", and the useful conditions we could impose to get around these.

The tl;dr summary is:

Things that seem like preference learning processes - including "have a prior, update it based on evidence" - have problems that allow the AI to manipulate the process.

Some things that don't seem like learning processes at all, actually are.

Part of the problem is that learning preferences is not well-grounded - we have to specify a learning process that allows the AI to connect facts about the world with facts about preferences.

There are many ways of specifying these, and most have problems.

Forget about "capturing the correct variable in the outside world"; it's tricky to design a learning process that "captures ANY variables in the outside world".

Thus we'll start by abstractly defining what a "preference learning process" is a very general way, rather than worrying about what we're learning: "how to learn" precedes "what to learn".

Then we'll add two useful conditions for such processes: unriggability, which implies the process respects conservation of expected evidence, and uninfluenceability, which implies the process derives from learning background variables in the environment.

We've shown that the syntactic/algebraic condition of unriggability is (almost) equivalent to the semantic condition of uninfluenceability.

Finally, we've shown that if the learning process is neither unriggable nor uninfluenceable, then the AI can manipulate the learning process, and there are situations where the AI's optimal policy is to sacrifice, with certainty, reward for every possible reward function.

1.1 Blast from the past: misleadingly named tokens

Good Old-Fashioned AI (sometimes called symbolic AI) did not work out. To define something, it wasn't enough to just name a token, and then set it up in relation to a few other named tokens, according to our own intuition about how these tokens related.

Saying "happiness is a state of mind", or "light is a wave", isn't nearly enough to define "happiness", "state of mind", "light", or "wave".

Similarly, designating something as "learning", and giving it some properties that we'd expect learning to have, isn't enou...