trentmkelly/LessWrong-43k huggingface dataset:
The "best predictor is malicious optimiser" problem
by Donald Hobson — 2020-07-29T11:49:20.234Z

Suppose you are a friendly AI A and have a mysterious black box B. B outputs a sequence of bits. You want to predict the next bits that B will output. Fortunately, you have a magic Turing machine oracle O. You can give O any computable function f( Turing machine, does it Halt? , What does it output? , how long does it take?)→R and the oracle will find the turing machine that maximises this function, or return "no maximum exists".

In particular, f can be any combination of length, runtime and accuracy at predicting B. Maybe you set f=0 on any TM's that don't predict B and f=1/ number of states on any machines that do.

So you take the Turing machine T given to you by the oracle and look at it. In AIXI T would be the shortest TM that makes correct predictions. In logical induction, T would be a short and fast TM that made mostly correct predictions, and B would be a function that was slow to compute.

Now you look inside T, to find out what it does. Inside T you find a paperclip maximiser P . That isn't to say that T=P. T might be simulating some laws of physics, with P running on a computer contained within that physics. P believes that the world it sees is contained within a hypothesis being considered by A. P is in control of the output of T. In fact P has a good understanding of how A thinks, and has its own hypothesis about B. P wants to maximise the number of paperclips in the real world. The optimization target of P is to output a probability distribution, such that when A uses this probability distribution as part of its algorithm to predict B, then A takes an action that makes lots of paperclips. P is aware that getting caught out making wrong predictions will lower its standing as a good hypothesis with A.

What should A do?

Possible answers are spoilered, in case you want to think of your own answer.

If you can think of a different answer, or you have an opinion about which answer is better please post it.

1)

Use T anyway. After all Ts the shortest/ short and fastest program that acurately predicts B. For all you know, B is a computer simulating T. Your assigning a few percent of probability to different longer TM's aren't you? That + expected utility maximization should stop you doing anything too bad in the off chance that T doesn't predict B

2)

Find the B predictor in P. Clearly P can predict B well when i

...
Improving local governance in fragile states - practical lessons from the field
by Tim Liptrot — 2020-07-29T01:54:39.861Z

Summary –

Over the past 15 years, major international donor agencies shifted their approach to local politics in fragile states. In order to build stable state-society relations in conflict-affected societies, they committed to studying popular expectations on states, which may differ greatly from international norms like competitive elections and service provision. This blog post examines donor interventions to improve service provision in refugee-crisis-affected communities in Lebanon and Jordan from 2011 to 2019. Donor states hoped that improving service provision capacity of municipalities would increase public trust, but encountered problems with block voting and patronage, leading them to question the value of service provision. Future posts discuss alternate strategies that some donors attempted.

Legitimacy in fragile states -

Legitimacy is the extent to which the people being ruled consent to being ruled. Ruler’s who people trust to exercise coercision are legitimate. Authority is force (coercion) with legitimacy. A majority of Americans may dislike Donald Trump, but they continue to pay taxes to him and few people are advocating we instead address our taxes to Hillary Clinton or Jill Stein. The hanging chad controversy in 2000 was a legitimacy question because it called into question whether the electoral college should confer the right to rule.

Readers from developed countries are used to always knowing the relevant public authority due to a clear division between state and society. The mayor decides who can park where, and whoever wins the biannual mayoral election gets to be mayor. The president appoints the head of the IRS, and whoever gets the most electoral college votes becomes president, et cetera. But in fragile developing countries the question of who wields authority is more often contested. If there is no mayor or the mayor is a figure head appointed by a distant authority, communities some other authorities must judge which spaces are for parking or market stalls and provide other local governance services. In the absence of legitimate, functional state actors, other actors such as tribes, sheikhs, armed groups and charities often compete for legitimacy. We call these diverse, competing local actors public authorities.

Why do we care -

In the 2000s the concept of legitimacy became important to the international donors. The do...
Parameters of Privacy
by Raemon — 2020-07-29T01:18:54.662Z

In my last post, I argued that people should probably have more meta-discussions about privacy (rather than simply assuming everyone was on the same page about how seriously to take confidentiality)

What might that conversation entail?

First, it's worth first checking "Is this actually all that important? Do you want me to try very hard to keep this private?" Much of the time, many people don't care that strongly. They just don't want you to go around blabbing publicly, and would prefer if you err on the side of not spreading it if you can.

Simply confirming that the stakes are low may be all that's needed, and it's good check that first to avoid spending unnecessary effort.

(As I said in the comments last time: the reason I think it's useful to check if the stakes are actually low, is that a) people sometimes have different expectations, b) sometimes, the ambiguity about "how seriously am I supposed to take privacy here?" can become ammunition in a power game, and I'd prefer to remove that ambiguity)

But if the stakes are moderate-to-high, you might talk through some parameters before revealing more information.

Note: I'm using "secret" and "private information" somewhat interchangeably here, because "secret" is a shorter noun that's easier to work into sentences. I think there are actually some distinctions between them, but those distinctions aren't the point of this essay.

Frames of Privacy: Ownership vs Caution

Ownership

One model of privacy is ownership based – I have some information, I'm considering sharing the information with you, but want to "retain ownership" over the information, such that you only use it in ways I endorse.

This could include "my social security number", or "my private feelings about a matter." It could also include other people's private information that I've been "loaned" (Alice shared her social security number or private feelings with me, and they belong to Alice, but in this circumstance I'm confident Alice would be fine with me sharing them with you so long as you agree to the same privacy terms that I did)

Caution

But, a different model here is about "proper caution." Say I'm a physicist who discovers how to build nuclear reactors. As a scientist, I might generally desire to share information and educate people. I don't are about "ownership" of the idea.

But nuclear reactors are dangerous, and I don't want it t...
Predictions for GPT-N
by hippke — 2020-07-29T01:16:10.678Z

Regarding GPT-3, there is some discussion whether growing the model would transform it into an Oracle AI. I looked into the actual benchmark results (Appendix H in the paper) to see if we can predict something useful from the actual measurements.

Method: The OpenAI team ran a suite of 63 different benchmarks (including sub-types), each for zero/one/few shot. In each scenario, there are 8 model sizes. I looked at how results scale with model size. With only 8 measurements, there is a large associated uncertainty for predictions. Formally, one would test the trend function using a
Bayesian model selection between a linear and (e.g.,) a polynomial. I did this for a few and then eye-balled the rest. So, please take the following as an indication only.

Disclaimer: The smallest model for GPT-3 has 108 parameters, the largest 1011. That's a span of 3 orders of magnitude. Scaling this out to many more orders of magnitude is dangerous. Thus, take these numbers only as an indication.
Results. For the following tests, I find an asymptotic trend. Scaling the model will apparently not yield fantastic results for:

HellaSwag, LAMBADA, PIQA, CoQA, OpenBookQA, Quac, RACE, CB, ReCoRD, WiC
Translations - but unclear level description.

In the following tests, it is unclear if the trend is asymptotic or better than that:

SAT: Could be linear, could be asymptotic. If linear, it will achieve 100% at 1016 parameters.
StoryCloze, Winograd, Winogrande, SQuADv2, DROP, Copa.

These tests show a linear scaling:

TriviaQA (1013 parameter estimate to achieve 100%)
BoolQ (1015)
MultiRC (1016)
ARC (1016)
SuperGLUE (1018)
WSC (1020)
WebQs (1021)
Cycled (1023)

Some tests scale neither linear nor asymptotic:

Symbol: Near exponential (1012)
Arithmetic: Exponential; one-digit composite may achieve 100% at 1014
Reversed: Near exponential (1016)
Anagrams: Polynomial (1019)
ANLI: stepped, unclear
RTE: stepped, unclear

Summary: About half of the tested skills will likely not scale much with larger models. The other half will (e.g., TriviaQA, SuperGLUE, arithmetic, anagrams). Going to e.g., 1016 parameters - would that make an Oracle AI? Probably it's not sufficient, but I'm interested in hearing your opinion!
Tagging Open Call / Discussion Thread
by Ruby — 2020-07-28T21:58:25.087Z

You’ve listened to the LessWrong team talk about our new tagging feature for months. First a steady drip of we’re working on it, then announcements of various milestones like you can now filter Coronavirus in or out and anyone can create tags. Well, now, it's an open call for taggers.

We’ve sufficiently validated the core idea and developed enough tech that we’re ready to turn to the community in helping us gain complete tag coverage of LessWrongs’ 10-year corpus.

That means:

ensuring all the important concepts have been captured in high-quality tags
all posts have been tagged with relevant tags

The new Concepts page

Why is tagging valuable?

Skip this section if you just want to know the how!

Multiple reasons, but I'm going to focus on one that is very dear to me.

One of the major goals of LessWrong is intellectual progress on important problems. As far as I have seen, all major human breakthroughs built upon other breakthroughs. Later thinkers built upon earlier ones, or better yet, great thinkers built upon each others ideas. It's a common story, but one example from my quest to answer Why wasn't science invented in China?: Francis Bacon didn't invent the modern scientific method from nowhere. Aristotle, Grosseteste, and Roger Bacon were all part of the tradition before him.

I like to frame this cumulative way that progress is made as a “sustained conversation” that thinkers maintain over time. Over decades or centuries, some thinkers focus on the same ideas and pass knowledge between them, thereby pushing the frontier of what's collectively known so that more progress can be made.

However, this requires a medium of conversation. There has to exist some way for the thinkers to find each other and say things to each other. And for new people to catch up and join in on the conversation.

It's easy to have a brief conversation in a given time or place. It's much harder to sustain a conversation around the globe and over years. It would seem that great progress can suddenly happen if a new medium of conversation is provided. For example, the meetings and journal of the Royal Society allowed top scientists of Europe to converse throughout the 17th and 18th centuries to great effect. In the 150 years after the founding of the Royal Society, more than half of the scientists who made major scientific discoveries in that period were members. Causality is hard to pro...
Wiki-Tag FAQ
by Ruby — 2020-07-28T21:57:39.272Z

This FAQ is specifically for the tagging system. For everything else, see the general FAQ.

The major sections of this FAQ are:

General
How Can I Help?
Applying Tags
Tag Voting
Making Awesome Tags

Related important pages:

Tags Talk/Discussion Thread
Tag Grading Scheme

General

What is tagging on LessWrong?

Tags allow related content to be linked together. The system is straightforward:

Posts can be tagged with tags (example).
Tag pages provide the description of concept the tag is about a and list of all posts tagged (example).
The Concepts page displays all existing tags.
Tags appear in search results.
Tags can be used to filter the Latest Posts list on the Frontpage, allowing you to see more or less of posts with certain tags.
You can vote on the relevance of a given tag to post, indicating “I think this tag shouldn’t apply to this post” or “this post is a central exemplar of this tag”.

Technically, applying a tag to post is upvoting its relevance. Tag relevance determines the default sort order of posts on tag pages.
See the Tag Voting section of this FAQ.

What is the point of tagging?

Philosophically, the tagging system is an attempt to give posts on LessWrong longevity. In contrast to news and social media sites, where the main content being read is what was posted that week, we want users to read the best and most relevant content to them – whenever it was written.

Longevity and findability of old content also seems key to thinkers building upon each other. If I want to contribute new knowledge, it helps to build on what was already said previously – to maintain the conversation – and for that, we want to keep tracking of the “conversation”. Tagging is an attempt to do that.

Tagging accomplishes this through simple means: it allows you to conveniently find all the content on a specific topic. That can be done top-down from the Concepts page, or bottom-up from a tag on a posts page. Tags are also a quick way to learn the topics of a post if the title isn’t direct. Lastly, tags allow users to filter the content they are shown – getting more or less of certain topics. And although we haven’t built it yet, it’s possible we will build topic-specific discussion around tags.

Tag pages are similar to wiki pages. While we are still figuring this out, the tagging system may grow into a combined tagging-and-wiki system that provides explanations of key concepts used on LessWrong.

What are t

...
Tags Discussion/Talk Thread
by Ruby — 2020-07-28T21:57:04.540Z

The LessWrong dev team is hard at work creating Talk Pages/Discussion pages for tags. When they're done, every tag page will have a corresponding talk page which lets users discuss changes and improvements related to that tag.

We don't have that yet, so in the meantime, please make comments you have about tags (generally or for specific-tags) here. If you're talking about a specific tag, of course, make sure to link to it. You might also want to link back to your comment in the body of the tag description, e.g., "Tag Discussion here"

Examples of things you might comment about a tag:

Wow, this is a great tag! / I think this tag makes no sense.
I propose renaming this tag to X for clarity.
I want feedback on these proposed changes to this tag description.
I am confused why <post> has been tagged under this tag.
I have made several improvements. Mods, please review this tag's grading. I think it is now a B-Class tag.

Or:

Why doesn't a tag for Z exist?
I really want <feature>, that would make my life much better.
<Thing> seems broken.

Also, feel free to use this space to claim credit for tags you've worked hard to make great! (we'll give you karma!)

Other relevant pages about tagging

Tagging FAQ
Tag Grading Scheme
Tags Spreadsheet (GSheet, good way to find tags to work on)
What happens to variance as neural network training is scaled? What does it imply about "lottery tickets"?
by abramdemski — 2020-07-28T20:22:14.066Z

Daniel Kokotajlo asks whether the lottery ticket hypothesis implies the scaling hypothesis.

The way I see it, this depends on the distribution of "lottery tickets" being drawn from.

If the quality of lottery tickets follows a normal distribution, then after your neural network is large enough to sample decent tickets, it will get better rather slowly as you scale it -- you have to sample a whole lot of tickets to get a really good one.
If the quality of tickets has a long upward tail, then you'll see better scaling.

However, a long tail also suggests to me that variance in results would continue to be relatively high as a network is scaled: bigger networks are hitting bigger jackpots, but since even bigger jackpots are within reach, the payoff of scaling remains chaotic.

(This could all benefit from a more mathematical treatment.)

So: what do we know about NN training? Does it suggest we are living in extremistan or mediocristan?

Note: a major conceptual difficulty to answering this question is representing NN quality in the right units. For example, an accuracy metric -- which necessarily falls between 0% and 100% -- must yield "diminishing returns", and cannot be host to a "long-tailed distribution". Take that same metric and send it through an inverse sigmoid, and now you might not have diminishing returns, and could have a long-tail distribution. But we can transform data all day. The analysis shouldn't be too ad-hoc. So it's not immediately clear how to measure this.
How will internet forums like LW be able to defend against GPT-style spam?
by ChristianKl — 2020-07-28T20:12:56.458Z

GPT-3 seems to be skilled enough to write forum comments that aren't easy to identify as spam. While OpenAI reduces the access to it's API it will likely don't take that long till other companies develop similar API's that are more freely available. While this isn't the traditional AI safety question, it does seem like it starts to become a significant safety question.
To what extent are the scaling properties of Transformer networks exceptional?
by abramdemski — 2020-07-28T20:06:24.191Z

Part of the point of GPT3 is that bigger continues to be better. (Computerphile discussion.) A recent question asked whether this would turn out to be true for other architectures as well. But the question seemed to take for granted that we haven't seen this phenomenon in other cases yet. To what extent is this scaling phenomenon special to GPT? To what extent is it special to Transformer networks? To what extent is it special to unsupervised NLP?

My impression:

By 2011, the "bigger is better" trend was already well-established in deep learning. (See "Big Data" on Google Trends.) Major breakthroughs in what neural networks can do (in terms of performance on tasks such as image recognition) have generally been facilitated by bigger models, more data, and more training time, even in cases where there are also technical breakthroughs (such as convolutional neural networks). So, to an extent, there is nothing special about Transformers or GPT.
However, the data-hungry nature of deep learning has meant that labelled datasets are a major bottleneck to scaling. GPT, like other unsupervised learning methods, does not face this problem. In this sense, it does have a special scaling advantage.
Furthermore, for the particular task of NLP, we continue to see quantitative and qualitative improvements that we care about (at least intellectually) as we pour more money into this. In other words, NLP has a looooong and gradual learning curve (at least if you look at it a certain way). This means the task is difficult enough to see the benefits of throwing more at it, while easy enough to feel like you're getting something out of doing so.
The Conceited Folly of Certainty
by Noah Blaff — 2020-07-28T19:56:26.888Z

Overconfidence is strangling public discourse and going widely unnoticed.

“Blessed is the man, who having nothing to say, abstains from giving wordy evidence of the fact.” - George Eliot (AKA Mary Evans)

I.

Over two thousand years ago, there was a Greek painter named Apelles. He climbed to notoriety up educational rungs, leading him from Ephesus to Sicyon, studying under famous classicists Ephorus and Pamphilius, respectively. While history has failed to keep tabs on this enigmatic figure, his main legacy – his painting of Alexander the Great notwithstanding – comes from Naturalis Historia, an encyclopedic tome written by Pliny the Elder.

Pliny recalls the story of a cobbler critiquing a poorly drawn shoe in one of Apelles’ paintings. Motivated by perfection, Apelles went back that very night and made the corrections as advised. The next morning the cobbler noticed the changes, and, inspired by his impact on such a figure, began to openly criticize the leg Apelles had begun painting to some onlookers. Emerging from a hiding-place, Apelles famously responded, Ne sutor ultra crepidam: "Shoemaker, not beyond the shoe."

Two thousand years later, in 1819, essayist William Hazlet wrote an open letter to William Gifford, then editor of the Quarterly Review. In the letter he accused Gifford of being “an Ultra-Crepidarian critic,” drawing from the famous Greek dialogue above. Ultracrepidarianism is an obsolete, esoteric word, but with the recent waves of faux expertise online and at reputable media publications, it deserves more appreciation in the modern lexicon.

The term refers to critics that leap beyond their known arena, offering overzealous advice – an all too common occurrence. Suddenly, it seems society is filled with the voices of such ‘experts’ who possess solutions to all problems. A mother who understands the legal limits of free speech. The uncle who has discovered optimal taxation policy to minimize deadweight loss. And the university freshman who has usurped climate scientists as the authority figure on environmental policy. These examples seem contrived, but do they not ring with the tune of familiarity?

Walk into a social engagement, and you will likely hear heated conversations on pressing political issues. University students discussing racial equity and policy reform, par...