trentmkelly/LessWrong-43k huggingface dataset:
So you want to be a wizard
by  — 2019-02-15T15:43:48.274Z
Link: https://jvns.ca/blog/so-you-want-to-be-a-wizard/?utm_source=hackernewsletter&utm_medium=email&utm_term=watching

This is a link post to a recent talk by Julia Evans, popular programming blogger, called "So you want to be a wizard". I'm linking it on LW because I think it's one of the better takes I've seen on how to "apply rationality" to programming. A few interesting examples of things she discusses that I think will appeal to the LW audience:

How she built a game to get better at Fermi estimate of how long different computer operations should take.

How believing that she can achieve a gears-level understanding of things she doesn't yet know much about makes her a better programmer and debugger.

How she uses "premortem"s to predict what will go wrong with projects.

While I suspect none of the above will be new to much of the LW audience, I think she does a good job of connecting each strategy she discusses to real-world examples that help you understand what it would actually look like to apply them.
Cooperation is for Winners
by Jacob Falkovich — 2019-02-15T14:58:08.949Z

Cross-posted from Putanumonit.

My last post was about learning to compete well from sports. But why compete at all? After all, I’m the one who wrote that winning is for losers, and we should avoid getting sucked into zero-sum contests.

Competing well doesn’t necessarily mean having to compete more, for one thing. Instead, it does allow you to choose where to compete. More importantly, sports-like competitions create prestige hierarchies (as opposed to dominance hierarchies). Climbing those is not intended to make you a fearsome boss but a valuable ally, one that others want to cooperate with. Getting to cooperation often requires winning competitions.

Wei Dai commented on the previous post:

More seriously, these days I think of competition as more of a problem than a solution. Some of the most important x-risks (e.g., advanced AI) are x-risks mainly because of competitive dynamics. If people weren’t competing for the prestige/power/money of being first to create AGI or to make advances in AI in general, we’d be able to solve AI safety problems at leisure.

How does one get to solve AI safety problems at their leisure, in a cooperative environment? Obviously, by working at the Machine Intelligence Research Institute.

But to work at MIRI one must first get a job at MIRI, and since MIRI has few spots and many applicants this means outcompeting other candidates for the job. MIRI itself is funded by donations, so it has to outcompete other non-profits for grants.

Hopefully, both competitions are conducted in a sportsmanlike way by demonstrating research achievements instead of by sabotaging competitors. But it’s still a competition – applying for jobs or grans is stressful and entails a high risk of failure, often due to circumstances outside one’s control.

Organizations have to manage very carefully the process of directing the winners of a competitive selection process to an ultimately cooperative endeavor. Here’s a wonderful post looking at this issue in two almost identical situations: chicken coops and academia.

So in the late 70s and early 80s there were active breeding programs to produce hens who laid more eggs. Take the highest egg layers in each generation and let them be the only ones that breed to produce the next generation. […]

Only one problem. High productivity egg-laying is associated with aggression – indeed the highest egg layers are basically the ones th

...
Quantifying anthropic effects on the Fermi paradox
by Lukas Finnveden — 2019-02-15T10:51:04.298Z

Crossposted to the EA forum.

Summary:

I apply the self-indication assumption (a theory of anthropics) and some non-causal decision theories to the question of how common spacefaring civilisations are in the Universe. These theories push strongly towards civilisations being very common, and combining them with the observation that we haven’t seen any extraterrestrial life yields a quite specific estimate of how common such civilisations are. If you accept the self-indication assumption, you should be almost certain that we’ll encounter other civilisations if we leave the galaxy. In this case, 95 % of the reachable universe will already be colonised when Earth-originating intelligence arrives, in expectation. Of the remaining 5 %, around 70 % would eventually be reached by other civilisations, while 30 % would have remained empty in our absence. Even if you don’t accept the self-indication assumption, most non-causal decision theories have the same practical implications. If you believe that other civilisations colonising the Universe is positive, this provides some reason to prefer interventions that increase the quality of the future over reducing non-AI extinction risk; if you think that other civilisations colonising the universe is negative, the opposite is true.

Introduction

There are billions of stars in the Milky Way, and billions of trillions of stars in the observable universe. The Fermi observation is the surprising observation that not a single one of them shows any signs of life, named after the Fermi paradox. There are several possible explanations for the Fermi observation: perhaps life is very unlikely to arise from any particular planet, perhaps life has just recently begun emerging in the Universe, or perhaps there is some reason that life never leaves the solar system in which it emerges. Without any further information, it may seem difficult to figure out which one it is: but the field of anthropics has a lot to say about this kind of situation.

Anthropics is the study of what we should do and what we should believe as a consequence of observing that we exist. For example, we can observe that life and civilisation appeared on Earth. Should we interpret this as strong evidence that life appears frequently? After all, life is far more likely to arise on this planet if life appears frequently than if it doesn’t; thus, life arising on this planet i... ,
How does OpenAI's language model affect our AI timeline estimates?
by jimrandomh — 2019-02-15T03:11:51.779Z

OpenAI recently announced progress in NLP, using a large transformer-based language model to tackle a variety of tasks and breaking performance records in many of them. It also generates synthetic short stories, which are surprisingly good.

How surprising are these results, given past models of how difficult language learning was and how far AI had progressed? Should we be significantly updating our estimates of AI timelines?
Who owns OpenAI's new language model?
by ioannes — 2019-02-14T17:51:26.367Z

OpenAI published about their very general language-writing & comprehension model (a) today.

Interestingly, they're not releasing the model itself:

Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights.

Does OpenAI hold a copyright for this model (and its other intellectual property)?

If the US government decided they wanted to requisition this model, would an OpenAI copyright defend against that? Would OpenAI have any real recourse against a government requisition?
The Prediction Pyramid: Why Fundamental Work is Needed for Prediction Work
by ozziegooen — 2019-02-14T16:21:13.564Z

Epistemic state: I feel like this post makes fairly intuitive claims, but I have uncertainty on many of the specifics.

In data science, it is a common mistake for organizations to focus on specific exciting parts like machine learning and data visualizations, while overlooking the infrastructural concerns required to allow for such things. There have been several attempts at making pyramids to showcase the necessary data science dependencies in order to make the most accessible parts realizable.

Similar could be said for predictions. Predictions require foundational work in order to be possible and effective. We can use the prediction pyramid below to show this dependency.

Evaluations

People beginning a prediction practice quickly run into the challenge of having well-specified questions. It's not enough to ask who will win a sports game, one needs to clarify how every exceptional situation is to be handled.[1]

Question specification is a big part of Metaculus. Often questions carry significant discussion even after a question is posed in order to discuss possible edge cases.

In addition to question specification, evaluations can be costly to perform. Even in simple cases it still requires manual work. In more complex cases evaluations could take a very long time. GiveWell does charity evaluations, and these can be extensive. This document discusses some other kinds of evaluations, in the "Possible Uses" section.

Ontologies

Say one is trying to determine which diseases will be important to worry about in 2025. One would first need a taxonomy of diseases that will not change until after 2025. If they were to somehow use a poor or an unusual taxonomy, resulting information wouldn't be useful to others.

In the case of diseases, decades of research years have been carried out in order to establish pragmatic and popular taxonomies. In other domains, new ontologies would need to be developed. Note that we consider ontologies to be a superset of taxonomies.

Another example: the usefulness of careers. 80,000 Hours is an expert here. They have a system which splits career paths into several distinct domains, and rates each one using six distinct attributes. They then do evaluations for each combination.

If it were assured they would continue to do so in the future, it would be relatively straightforward to forecast their future evaluations. If one wanted to do similar pred

...
Short story: An AGI's Repugnant Physics Experiment
by ozziegooen — 2019-02-14T14:46:30.651Z

After lots of dedicated work and a few very close calls, a somewhat aligned AGI is created with a complex utility function based on utilitarian-like values. It immediately amplifies its intelligence and ponders what its first steps should be.

Upon some thought, it notices a few strange edge cases in physics. The universe seems fairly limited, unless…

The chances of expanding far past the known universe seem slim, but there could be a way with enough thought and experimentation. The chances are small, but the payoff could be enormous.

The AGI proceeds to spend all of the available resources in the universe to improve its model of fundamental physics.
New York Restaurants I Love: Breakfast
by Zvi — 2019-02-14T13:10:01.072Z

Previously (and more generally useful): Restaurant Guide 1: Restaurants should not look like (most) restaurants, Restaurant Guide 2: Pizza

Also: Minimize Use of Standard Internet Food Delivery

I don’t want to lose sight of the bigger issues pointed at last time. But for the moment, we have pressing matters to attend to that have been put off far too long.

In the last two months, I have lost two of my favorite three places to eat. BLT Prime, my happy place, closed its doors first. Then, a day after I lost my best Indian restaurant Old Monk, and wrote about the need to support local places so they stay in business, I find out my favorite breakfast source, Patisserie Florentine, closed its doors.

Twice in a row, I left my house to eat delicious food, only to find the destination permanently closed. Quite the kick in the nuts.

Thus, I feel the need to write this. This is a tabulation of all the New York City restaurants that would make me actively upset to find out they were closed.

That doesn’t mean these are the best places in the city. Some of them are among the best. Some are simply the best that I’ve found, near where my apartment is, for a particular purpose I appreciate.

I hope that by telling the world about these places, I will encourage others to discover them, enjoy delicious food and help keep them in business. I also hope others will share the places they love, wherever you may live, and that others will alert me to places I have missed. New York is full of wonderful things, and I’m always finding new ones.

Let’s start with breakfast. Hopefully posting this will create the momentum to finish the job.

I’ve got five places I’d especially hate to see go.

1. Le Maisson de Macaron

My favorite breakfast these days is croissants. Great croissants are amazing explosions of flavor, perfect to start the day. Their lightness allows them to provide the experience I want without too many calories, which is vital for me – my metabolism severely restricts how much I can eat. I just lost my best source within quick walking distance, but there are still great options.

My favorite place in the world to eat breakfast is Le Maisson de Macaron, at 23rd Street and 6th Avenue.

This is a wonderful place to sit down and relax, if one has the time. If not, one can still take away the best croissants, chocolate croissants and almond croiss...
Alignment Newsletter #45
by Rohin Shah — 2019-02-14T02:10:01.155Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

Highlights

Learning Preferences by Looking at the World (Rohin Shah and Dmitrii Krasheninnikov): The key idea with this project that I worked on is that the state of the world is already optimized for our preferences, and so simply by looking at the world we can infer these preferences. Consider the case where there is a vase standing upright on the table. This is an unstable equilibrium -- it's very easy to knock over the vase so it is lying sideways, or is completely broken. The fact that this hasn't happened yet suggests that we care about vases being upright and intact; otherwise at some point we probably would have let it fall.

Since we have optimized the world for our preferences, the natural approach is to model this process, and then invert it to get the preferences. You could imagine that we could consider all possible reward functions, and put probability mass on them in proportion to how likely they make the current world state if a human optimized them. Basically, we are simulating the past in order to figure out what must have happened and why. With the vase example, we would notice that in any reward function where humans wanted to break vases, or were indifferent to broken vases, we would expect the current state to contain broken vases. Since we don't observe that, it must be the case that we care about keeping vases intact.

Our algorithm, Reward Learning by Simulating the Past (RLSP), takes this intuition and applies it in the framework of Maximum Causal Entropy IRL (AN #12), where you assume that the human was acting over T timesteps to produce the state that you observe. We then show a few gridworld environments in which applying RLSP can fix a misspecified reward function.

Rohin's opinion: In addition to this blog post and the paper, I also wrote a post on the Alignment Forum expressing opinions about the work. There are too many disparate opinions to put in here, so I'd recommend reading the post itself. I guess one thing I'll mention is that to infer preferences with a single state, you definitely need a good dynamics model, and a good set of features. While this may seem difficult to get, it's worth noting that dynamics are empirical facts ab...
Three Kinds of Research Documents: Exploration, Explanation, Academic
by ozziegooen — 2019-02-13T21:25:51.393Z

Aug 2020 Edit: Changed "Clarification" to "Exploration", thanks to a comment by Richard_Ngo

Epistemic Status: Low. This was a quick idea, but the grouping honesty doesn't work as well as I'd like. I still think it could be useful to some people though. Ideas appreciated.

Recently I have started writing more and have been trying to be more intentional with what I accomplish. Different documents have different purposes and it seemed useful to help clarify this. Here is a list of three specific different types I think are relevant on LessWrong and similar.

Exploration

I see exploration posts as generally the first instance of information being written down. Here it is important to get the essential ideas out there and to create consensus around terminology among the most interested readers. In some cases, the only interested reader may be the author, who would use the post just to help cement their ideas for themselves.

Exploration posts may not be immediately useful and require later posts or context for them to make sense. This is typically fine. There's often not a rush for them to be understood. In many cases, there is a lot of possible information to write down, so the first step is to ensure it's out there, even if it's slow, hard to read, or doesn't much make sense until later.

I think of many of Paul Christiano's posts as exploration posts. They're very numerous and novel, but quite confusing to many readers (at least, to myself and several people I've talked to). Sometimes the terminology changes from one post to the next. I used to see this is somewhat of a weakness, but now it comes across to me as a pragmatic option. If he were to have tried to make all of this readable to the average LessWrong reader, there's likely no way he could have written a portion as much.

One important point here is that if something is a exploration post, then the main relevant feedback is on the core content, not the presentation. Giving feedback on the readability can still be useful, but it should be understood and expected that this isn't the main goal.

Explanation

Explanation posts seek to explain content to people. The focus here is on accessibility. Often the main ideas are already documented somewhere, but the author thinks that they could do a better job explaining them to their intended audience.

I would categorize some of the recent posts on Embedded Agency as bei...
Humans interpreting humans
by Stuart_Armstrong — 2019-02-13T19:03:52.067Z

In a previous post, I showed how, given certain normative assumptions, one could distinguish agents H for whom anchoring was a bias, from those H′ for which it was a preference.

But agent H′ looks clearly ridiculous - how could anchoring be a bias, it makes no sense. And I agree with that assessment! H′'s preferences make no sense - if we think of it as a human.

Humans model each other in very similar ways

This is another way in which I think we can extract human preferences: using the fact that human models of each other, and self-models, are all incredibly similar. Consider the following astounding statements:

If somebody turns red, shouts at you, then punches you in the face, they are probably angry at you.

If somebody is drunk, they are less rational at implementing long-term plans.

If somebody close to you tells you an intimate secret, then they probably trust you.

Most people will agree with all those statements, to a large extent - including the "somebody" being talked about. But what is going on here? Have I not shown that you can't deduce preferences or rationality from behaviour? It's not like we've put the "somebody" in an FMRI scan to construct their internal model, so how do we know?

The thing is, that natural selection is lazy, and a) different humans use the same type of cognitive machinery to assess each other, and b) individual humans tend to use their own self-assessment machinery to assess other humans. Consequently, there tends to be large agreement between our own internal self-assessment models, our models of other people, other people's models of other people, and other people's self-assessment models of themselves:

This agreement is not perfect, by any means - I've mentioned that it varies from culture to culture, individual to individual, and even within the same individual. But even so, we can add the normative assumption:

β: If H is a human and G another human, then G's models of H's preferences and rationality are informative of H's preferences and rationality.

That explains why I said that H was a human, while H′ was not: my model of what a human would prefer in those circumstances was correct for H but not for H′.

Implicit models

Note that this modelling is often carried out implicitly, through selecting the scenarios, and tweaking the formal model, so as to make the agent being assessed more human-like. With many variables to

...
Anchoring vs Taste: a model
by Stuart_Armstrong — 2019-02-13T19:03:08.851Z

Here I'll develop my observation that anchoring bias is formally similar to taste based preferences, and develop some more formalism for learning the values/preferences/reward functions of a human.

Anchoring or taste

An agent H (think of them as a simplified human) confronts one of two scenarios:

In scenario I, the agent sees a movie scene where someone wonders how much to pay for a bar of chocolate, spins a wheel, and gets either £0.01 or £100. Then H is asked how much they would spend for the same bar of chocolate.

In scenario II, the agent sees a movie scene in which someone eats a bar of chocolate, which reveals that the bar has nuts, or doesn't. Then H is asked how much they would spend for the same bar of chocolate.

In both cases, H will spend £1 for the bar (£0.01/no nuts) or £3 (£100/nuts).

We want to say that scenario I is due to anchoring bias, while scenario II is due to taste differences. Can we?

Looking into the agent

We can't directly say anything about H just by their actions, of course - even with simplicity priors. But we can make some assumptions if we look inside their algorithm, and see how they model the situation.

Assume that H's internal structure consists of two pieces: a modeller M and an assessor A. Any input i is streamed to both M and A. Then M can interrogate A by sending an internal variable v, receives another variable in return, and then outputs o.

In pictures, this looks like this, where each variable has been indexed by the timestep at which it is transmitted:

Here the input i1 decomposes in m (the movie) and q (the question). Assume that these variables are sufficiently well grounded that when I describe them ("the modeller", "the movie", "the key variables", and so on), these descriptions mean what they seem to.

So the modeller M will construct a list of all the key variables, and pass these on to the assessor A to get an idea of the price. The price will return in v3, and then M will simply output that value as o4.

A human-like agent

First we'll design H to look human-like. In scenario I the modeller M will pass v2=q to the assessor A - only the question q= "how much is a bar of chocolate worth?" will be passed on (in a real world scenario, more details about what kind of chocolate it is would be included, but let's ignore those details here). The answer v3 will be £1 or £3, as indicated above, dependent on m (which is a

...
Individual profit-sharing?
by ioannes — 2019-02-13T17:58:41.388Z

Here's a sketch of an idea:

Design an open-source legal agreement that two people sign.
The contract states that each person agrees to give the other 1% of their annual earnings, each year for X years. (Ideally X = several decades; both duration & percentage could be customized)
Contract is legally binding; each year both parties pay out to each other.
Not exclusive: a person could be in multiple contracts simultaneously (e.g. 5 contracts with 5 friends, sharing in total 5% of their annual earnings).

Two motivations for signing a contract like this:

Diversify one's career & earnings risk by "investing" in admired peers.
Deepen one's relationship with the other signee (signing isn't a thing to be taken lightly); signing signals intimacy & desire to build a longterm relationship with the other person.

Of course there are lots of ways something like this could go awry.

Has anyone heard of people doing something like this?

What are existing mechanisms that do something like this? Examples I've encountered already include marriage (50% profit-sharing indefinitely, at least in the US) and Kibbutzim (100% profit-sharing during one's tour of duty).