trentmkelly/LessWrong-43k huggingface dataset:
Alignment Newsletter #46
by Rohin Shah — 2019-02-22T00:10:04.376Z

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

Highlights

Better Language Models and Their Implications (Alec Radford, Jeffrey Wu, Dario Amodei, Ilya Sutskever et al): OpenAI has trained a scaled up GPT model using unsupervised learning (specifically, predicting the next word given a very large context) on a very large dataset with presumably very large compute. The resulting language model can produce impressive language samples (with some cherry-picking) that to my eye are particularly good at handling long-range dependencies, which makes sense since it is based on the Transformer (see Transformer-XL entry in AN #44). It sets new state of the art performance on 7 out of 8 language modeling tasks, including difficult datasets such as LAMBADA, without using the training data for those tasks. It can also be used for more structured tasks by providing a particular context -- for example, to summarize a document, you can provide the document followed by "TL;DR:" in order to induce GPT-2 to "predict" a summary. (They use a different prediction algorithm in order to improve summarization results, but I suspect even with regular prediction you'd get something in the right ballpark.) On these more structured tasks, it doesn't get anywhere near the state of the art set by specialized systems -- but again, this is without any finetuning for the specific task that we are testing.

The paper argues that in order to get generally capable AI systems, we will need to train them on many different tasks, as in meta-learning. However, we might expect that we need hundreds of thousands of tasks in order to learn something general, just as we need hundreds of thousands of examples in order to develop good classifiers. Prediction of the next word in natural language is particularly good for this, because in order to predict well across a huge variety of text, you need to become good at many different tasks such as question answering, summarization, and even translation. The biggest challenge is in creating a dataset that has sufficient diversity -- they do this by scraping all outbound links from Reddit with at least 3 karma.

Unusually for research, but in accordance with its charter (AN #2), OpenAI has decided not to release the model...
How could "Kickstarter for Inadequate Equilibria" be used for evil or turn out to be net-negative?
by Raemon — 2019-02-21T21:36:07.707Z

Following up to "If a "Kickstarter for Inadequate Equlibria" was built, do you have a concrete inadequate equilibrium to fix?"

I think a kickstarter for coordinated action would be net positive, but it's the sort of general purpose powerful tool that might turn out bad in ways I can't easily predict. It might give too much power to mobs of people who don't know what they're doing, or have weird/bad goals.

How bad might it be if misused? What equilibrias might be we end up in in the world where everyone freely has access to such a tool?
If a "Kickstarter for Inadequate Equlibria" was built, do you have a concrete inadequate equilibrium to fix?
by Raemon — 2019-02-21T21:32:56.366Z

Yoav Ravid asks: "Is there an assurance-contract website in work?"

i.e. a site where, if there's a locally bad equilibrium that would be better if everyone changed strategies at once, but which requires a critical mass of people in order to be worthwhile, you can all say "I'll put the effort if other people put in the effort", and then if X people agree, you all go into work the next day and demand a policy change, or a go to a political rally, or change a social norm, or whatever.

Some attempts have been made at such a system. It's not that technically hard to build. But I think it'd need a couple major "flagship" Coordinated Actions in order to rally people's attention and turn it into a more frequently used tool.

So, if a good website existed to coordinate action, do you have a well operationalized action you'd want to coordinate? ("Everyone leaves Facebook at once" doesn't work IMO, because it doesn't say where people are moving to, or otherwise replacing FB's tools with.

"Everyone on one platform switches to another platform" seems viable.

"Everyone at my office signs a letter demanding change for a particular policy" seems viable (although in cases like this, where you maybe don't want your boss to know you're planning a revolution, and I'm not sure how to best achieve common knowledge without risk)

(For further reading, see "The Costly Coordination Mechanism of Common Knowledge" and "Inadequate Equilibria")
Life, not a game
by ArthurLidia — 2019-02-21T19:10:56.128Z

Life is a rigged game. Quit the red bloody market, seclude from all the randomness of it and disappear completely. But that's a premise behind “do not compare yourself to others instead compare to the person you where yesterday”. Claiming that one secludes himself from the need to see yourself as lower than a person or superior to one. In order words compete with yourself , or past self. But is it so, since competition in healthy ways where you agree to play. But we don't agree to be alive in the first place. Nobody asked to participate in the game of life. Wittgenstein the family, his father wanted to turn his child into capitalist agent and make profit for him. They all 3 killed themselves like a chain reaction.

Beyond that are people who play life like a game.I hear it all over: Of course I will play capitalism and bureaucratic way of hosting institution as a game. But at life itself you draw the line. See Weber on institutions and it's way on personal freedom.

And only a foolish would rely on habit alone. If habits aren't you then what counts as you, let's say all your notebooks and journal where burnt to the ground. But you still remember right? Its episodic memory that is the last saviour, but isn't that efficient. Then you are also your data (that many say they have privacy policies but sure they steal your data). So does that mean you must protect yourself . all your digital profile and identity? Taking a hard drive and owning a physical extension of you because believing in the cloud has a slight chance of being destroyed. Google activity and how it tracks you by the timestamp. It is amazing to have this data and being able to switch between ideas and where you read them.

Maps of meaning referring to the wikipedia page for the meaning of life. It's quite depressing to reach the end looking for meaning and glossing down the footnotes and googling the authors of the books. From the philosophies of Kant (you come as you like but you pay as you go) and utilitarianism, to Jainism and other Eastern philosophies.

Mircea Eliade is an example of a man who experienced both worlds, and as a result he wrote a theory of religion and of the myths. He wrote Maitreyi (a love story novel recently read and his other novel/journal) and experience the karma, devas (gods of Hindu) and their religion. But he got compared to Buddha. It's like believing you are Jesus and the ...
Ideas for Next Generation Prediction Technologies
by ozziegooen — 2019-02-21T11:38:57.798Z

2021 Note: This was written and posted in December, 2016. The date it shows on LessWrong is 2019; I believe this refers to a time the post was (very minorly) updated, as part of moving it to the Prediction-Driven Collaborative Reasoning Systems sequence.

Prediction markets are powerful, but also still quite niche. I believe that part of this lack of popularity could be solved with significantly better tools. During my work with Guesstimate I’ve thought a lot about this issue and have some ideas for what I would like to see in future attempts at prediction technologies.

1. Machine learning for forecast aggregation

In financial prediction markets, the aggregation method is the market price. In non-market prediction systems, simple algorithms are often used. For instance, in the Good Judgement Project, the consensus trends displays “the median of the most recent 40% of the current forecasts from each forecaster.”[1] Non-financial prediction aggregation is a pretty contested field with several proposed methods.[2][3][4]

I haven’t heard much about machine learning used for forecast aggregation. It would seem to me like many, many factors could be useful in aggregating forecasts. For instance, some elements of one’s social media profile may be indicative of their forecasting ability. Perhaps information about the educational differences between multiple individuals could provide insight on how correlated their knowledge is.

Perhaps aggregation methods, especially with training data, could partially detect and offset predictable human biases. If it is well known that people making estimates of project timelines are overconfident, then this could be taken into account. For instance, someone enters in “I think I will finish this project in 8 weeks”, and the system can infer something like, “Well, given the reference class I have of similar people making similar calls, I’d expect it to take 12.”

A strong machine learning system would of course require a lot of sample data, but small strides may be possible with even limited data. I imagine that if data is needed, lots of people on platforms like Mechanical Turk could be sampled.

2. Prediction interval input

The prediction tools I am familiar with focus on estimating the probabilities of binary events. This can be extremely limiting. For instance, instead of allowing users to estimate what Trump’s favorable rating would be...
What's your favorite LessWrong post?
by pepe_prime — 2019-02-21T10:39:53.919Z

This was last asked in 2014 on the annual survey. Since we have strong upvotes, you may vote for 2 posts. Each answer should link to a single post, and I will delete repeats. Please use a strong and weak upvote each for at most one answer (and posting an answer is a vote).

Basically any LessWrong URL is ok - links, events, questions, and archive links to pages only on LessWrong 1.0 are fine. I only expect regular posts and links to be chosen though.

Feel free to comment on answers if you're dying to explain your reasoning, or comment on this post if you're dying to mention your third favorite and lower posts.
Thoughts on Human Models
by Ramana Kumar — 2019-02-21T09:10:43.943Z

Human values and preferences are hard to specify, especially in complex domains. Accordingly, much AGI safety research has focused on approaches to AGI design that refer to human values and preferences indirectly, by learning a model that is grounded in expressions of human values (via stated preferences, observed behaviour, approval, etc.) and/or real-world processes that generate expressions of those values. There are additionally approaches aimed at modelling or imitating other aspects of human cognition or behaviour without an explicit aim of capturing human preferences (but usually in service of ultimately satisfying them). Let us refer to all these models as human models.

In this post, we discuss several reasons to be cautious about AGI designs that use human models. We suggest that the AGI safety research community put more effort into developing approaches that work well in the absence of human models, alongside the approaches that rely on human models. This would be a significant addition to the current safety research landscape, especially if we focus on working out and trying concrete approaches as opposed to developing theory. We also acknowledge various reasons why avoiding human models seems difficult.

Problems with Human Models

To be clear about human models, we draw a rough distinction between our actual preferences (which may not be fully accessible to us) and procedures for evaluating our preferences. The first thing, actual preferences, is what humans actually want upon reflection. Satisfying our actual preferences is a win. The second thing, procedures for evaluating preferences, refers to various proxies for our actual preferences such as our approval, or what looks good to us (with necessarily limited information or time for thinking). Human models are in the second category; consider, as an example, a highly accurate ML model of human yes/no approval on the set of descriptions of outcomes. Our first concern, described below, is about overfitting to human approval and thereby breaking its connection to our actual preferences. (This is a case of Goodhart’s law.)

Less Independent Audits

Imagine we have built an AGI system and we want to use it to design the mass transit system for a new city. The safety problems associated with such a project are well recognised; suppose we are not completely sure we have solved them, but are confident enough ...
Two Small Experiments on GPT-2
by jimrandomh — 2019-02-21T02:59:16.199Z

I did two small experiments on the GPT-2 small model. First experiment: can GPT-2-small answer sentiment analysis questions? (It can't.) Second experiment: When GPT-2 writes continuations of Howl, is it picking up the "Moloch in X!" template from its priming, or from a copy of Howl in its original training set? (It's from the training set.)

Sentiment analysis experiment:

I downloaded the MPQA Subjectivity Lexicon, which is a dictionary in which words are marked as positive or negative. For example hopelessness=>negative, humour=>positive, grace=>positive, corruption=>negative. I primed GPT-2 with a list of 20 questions like "Is a <noun> good? Yes. Is a <noun> good? No." followed by an unanswered question of the same form, and had it continue for one more word. In its priming, half the answers were yes and the other half were no. It answered "No" 37/40 times, and neither its answers nor its yes answers were better than chance.

Howl experiment:

When given some lines from Ginsberg's Howl as priming, it writes a good continuation (similar to the one Chelsea Voss and Qiaochu Yuan got from it). In particular, it uses the "Moloch in X!" template repeatedly.

If I take its continuation of Howl and feed it back in as a prompt, I get more Howl (Moloch in X!). If I take Howl and replace "Moloch" with "Lomoch", I get more Howl. But if I take its continuation of Howl from the first step and replace Moloch with Lomoch *there*, I get unrelated text which does not use the "Moloch in X!" template.

So, it isn't inferring the template from its priming; rather, it learned the template from its training set (which probably included Howl), and it produces Howl-like text iff it's given a cue strong enough to remind it of the source.
Predictive Reasoning Systems
by ozziegooen — 2019-02-20T19:44:45.778Z

Meta: This is meant to be a succinct and high-level overview. I expect to go more into detail on parts of the system in future posts. This work builds on this previous post.

Predictions can be a powerful tool but are typically difficult to use in isolation. Being sure to predict the right things is hard. Organizing information from predictions is hard. Coming up with decision options to be predicted is hard. If we seek to create a crowdsourced system of predictions to give us useful information, it would be really useful if we could figure out solutions to these other problems as well.

In the future, I expect that much of the most important prediction work will exist within larger ecosystems of collective reasoning. We can call such systems "Predictive Reasoning Systems."

To give a motivating example, there's been some speculation on Futarchy, a concept of government that involves using prediction markets to determine which government policies would be optimal. Attempts to do this with existing prediction systems could pose substantial challenges. Which specific prediction questions should be asked? How should resources be allocated among questions? The space of all possible policies is gigantic, so how will this best be ideated about and then narrowed down? Predictive Reasoning Systems outline a high-level construct that may be more capable of handling such high-level challenges.

Prediction Systems vs. Predictive Reasoning Systems

We can call a system used exclusively to make predictions a Prediction System.

One interacts with a Prediction System by posing and answering prediction questions. Points or subsidization can be assigned by users to prioritize work on what seems like the most useful questions.

In contrast, one would interact with a Predictive Reasoning System by giving it higher level goals. For example, "Optimally spend 1000 human hours to come up with whatever information would most help our business." The system would translate these goals into lots of lower level work, including the use of forecasting.

If a group of people poses multiple questions on a Prediction System, with one overarching goal in mind, one can refer to this [combination of Prediction System + question askers] as a simple Predictive Reasoning System. One can even say that wherever a Prediction System exists, a Predictive Reasoning System of some sort exists around it. That said, j...
When does introspection avoid the pitfalls of rumination?
by rk — 2019-02-20T14:14:46.798Z

A theme I see being explored in the LW-sphere is the importance of connecting to system 1 information in various ways.

There's lots of stuff about meditation, Focusing, connecting to sub-agents, the question "What would happen if [...] you introspected instead?".

On the other hand, it appears to be received wisdom that a collection of behaviours called rumination are maladaptive. But "focused attention on the symptoms of one's distress, and on its possible causes and consequences, as opposed to its solutions" doesn't sound a million miles from some of the techniques in the paragraph above.

What gives? Are some of the techniques bad? Is rumination OK? Are they importantly different, and if so, how?
What i learned giving a lecture on NVC
by Yoav Ravid — 2019-02-20T09:08:58.441Z

Recently, I gave a lecture on Nonviolent Communication (NVC) at a democratic school in Israel, the attendees (or at least the active ones), were mostly 11-12 year old girls, and a teacher.

The first thing i found was a potentially good lecture structure (which I'll talk about a little), and the second was about miscommunication.

I had an hour and a half, so i experimented with an unusual nvc-teaching structure (at least to me). I've had past experience teaching nvc, and two key principles i learned are (1) Without some sort of practice, the lecture isn't worth much. and (2) that practice isn't that easy; imaginary cases usually don't work, and getting people to be willing to open and be vulnerable is hard. up to now i tried different methods each time.

in this lecture, since it was only a hour-and-half, i decided to just give them a taste. and thought maybe the best way to do that is to start from the easiest and most fun, to the harder and more sensitive.

so the first thing i got them to try was actually to express appreciation (which is talked about last in the book), using the nvc model - observation (The thing you did I'm referring to), emotion (how i felt following it), and need (what need was fulfilled). It worked wonderfully better than other times when i tried teaching the 4 steps in other ways.

after each one of them picked another person and expressed their appreciation toward them, i explained about miscommunication and the techniques NVC has to handle it - paraphrasing, and asking for a "reflection" from someone on what they heard you say. i asked half of the pairs to initiate paraphrasing, and the other to ask for reflection. they then protested that they don't see how it's useful, so i professed "yeah, it's true, it better suites situations of negative messages, but i want you to practice it even though you're probably right that you totally understood".

so they played along, and i worried that it'll actually feel like a waste of time. but by the first pair to try, this is (in short) what happened:

Girl A "Can you please say what you heard me say?"

Girl B "you said you like that when you share things with me, i give my opinion"

Me, to Girl A "was this what you meant?"

Girl A "not exactly"

Me, quite excited it worked "Great! so now yo

...
Pavlov Generalizes
by abramdemski — 2019-02-20T09:03:11.437Z

Sarah Constantine recently wrote about the Pavlov strategy for iterated prisoner's dilemma. She points out that it deserves to be better-known, given that it has similar virtues to Tit-for-Tat (TfT). I strongly agree.

Much of my thinking about puzzles in decision theory such as cooperation/coordination/bargaining and logical causality (through the lense of logical time) has been shaped by an assumption that TfF type behavior is "correct" in some sense, and needs to be appropriately generalized. These intuitions have been formed without an awareness of Pavlov, and so, are worth reconsidering.
It turns out that Pavlov-type behavior is much easier to generalize that TfT. Pavlov-esque strategies only require seeing when things are going well or poorly for you. TfT-esque strategies require an agent to understand what cooperation vs defection means in a particular game. This requires the agent to understand the wider rules of the game (not just the payoff on a given round), locate the other agents involved (distinguishing them from the environment), infer the values of the other agents, define a set of common cooperative norms which can be coordinated on, and figure out whether other agents have followed them! Several of these elements are both difficult in practice and difficult to define in principle.

In this post, I would like to explain how to generalize Pavlov to a wide set of games (achieving some surprisingly strong, but not actually very good, coordination properties), and also convey some intuitions about Pavlov-style vs TfT-style coordination.

Generalizing Pavlov

The paper Achieving Pareto Optimality Through Distributed Learning by Marden, Young, and Pao describes a strategy which allows players in an iterated game to eventually converge to a Pareto-optimal action profile with high probability. In fact, the strategy described converges to action profiles which maximize the sum of all the utilities! That should sound too good to be true. Some big caveats are coming.

The strategy is, roughly, as follows. Agents have two different states: content and discontent. When content, the agent copies its own action from the previous round, whatever it was. When discontent, the agent plays randomly.

An agent flips between content and discontent based on how well things are going for it. There is a parameter, experimentation rate (e.r.), which determines how ...
Is there an assurance-contract website in work?
by Yoav Ravid — 2019-02-20T06:14:00.633Z

I bet I'm not the only one who, after finishing inadequate equilibria, thought in excitement "Ok, so where's the KickStarter for better Nash equlibria?".

I didn't find any existing site that does something like it, but i wonder if maybe someone in this community is working on it.

If you know anything about it, I'd appreciate a hint ;)