trentmkelly/LessWrong-43k huggingface dataset:
Aggregating forecasts
by Jsevillamol — 2020-07-23T18:04:37.477Z

[EDIT: I have written an update of this post here]

Epistemic status: confused. I also seem to have spent some time thinking about it and still went with my gut without having a great reason for it.

When you ask several people or build several models to forecast the probability of an event you are left with the question of how to aggregate them to harness their collective wisdom.

Arguably, the best possible thing would be to examine the divergence in opinions and try to update towards a shared view. But this is not always possible - sharing models is hard and time consuming. How can we then aggregate forecasts naively?

I have been interested in this problem for a while, specially in the context of running forecasting workshops where my colleagues produce different estimations but we dont have enough time to discuss the differences.

To approach this problem I first lay my intuition, then skim two papers approaching this question and finally outline my best guess from what I learned.

I have a weak intution that taking the geometric mean is the best simple way to aggregate probabilities. When I try to inquire why do I believe that, my explanation is something like:

1) we want the aggregation procedure to be be simple (to not overfit), widely used (which is evidence of usefulness) and have a good theoretical justification

2) arithmetic means are simple, widely used and are the MLE estimator for the expected value of a normal distribution, which we expect to be common-ish based on the limit theorem

3) but summing probabilities is ontologically wrong / feels weird. p_1 + p_2 is not a probability.

4) a common approach in this kind of situation is to take the logarithm of your quantities of interest

5) the MLE of the expected value of a log normal is the geometric mean of the measures

So in practice I mostly use geometric means to aggregate forecasts and don't feel to bad about it.

There is still a part of me that feels like the explanation above is too handwavy, so I have been reading a bit into the literature of aggregating forecasts to try to undertstand better the topic.

Satopaa et al write about this issue, by deriving an estimator of the best possible estimator based on some statistical assumptions and then testing it on synthetic and real data.

The estimator has the form

Where $p_i$ are the individual forecasts, $N$ is the number of forecasts and $a$ is a parameter indica...
Can you get AGI from a Transformer?
by Steven Byrnes — 2020-07-23T15:27:51.712Z

UPDATE IN 2023: I wrote this a long time ago and you should NOT assume that I still agree with all or even most of what I wrote here. I’m keeping it posted as-is for historical interest.

Introduction

I want to share my thoughts about the calculations that Transformers (such as GPT-3) do, and the calculations that I think are required for general intelligence, and how well they line up, and what I think GPT-3 is doing under the hood, and why I think an arbitrary transformer-based GPT-N might be incapable of doing certain tasks are seemingly essential for a system to qualify as an AGI.

Epistemic status: Very low confidence, to the point that I almost decided to delete this without posting it. I think some of my opinions here are very unpopular, and I would love any feedback or discussion.

Before we get into it, I want to make a couple background claims. The point here is basically to argue that the question “Can you get general intelligence by sufficiently scaling up a Transformer?” is worth asking, and does not have an answer of “Obviously yes, duh!!!” You can skip this part if you already agree with me on that.

Background Claim 1: There are types of information processing that cannot be cast in the form of Deep Neural Net (DNN)-type calculations (= matrix multiplications, ReLUs, etc.), except with an exorbitant performance penalty.

(Update: After discussion in the comments section here, I should have framed this section differently: I should have said "For any given specific learning algorithm / architecture framework (e.g. "GPT-like transformer architectures"), there are types of information processing that cannot be performed within that specific framework, except with an exorbitant performance penalty". I should never have mentioned "DNN-type calculations (=matrix multiplications, ReLUs, etc.)", because that's awfully vague (what exactly does the "etc." mean?), and anyway it's irrelevant to this post. Thanks gwern.)

By “information processing” I mean anything from sorting algorithms to data compression, random access memories, hash tables, whatever.

Let’s take Monte Carlo Tree Search (MCTS) as an example. AlphaZero does MCTS because DeepMind engineers explicitly programmed it to do MCTS—not because a generic RNN or other deep learning system spontaneously discovered, during gradient descent, that MCTS is a good idea.

Now, in principle, DNNs are universal func...
Covid 7/23: The Second Summit
by Zvi — 2020-07-23T14:40:00.980Z

Last week: Covid 7/16: Becoming the Mask

The news continues to be cautiously good.

Positive test rates are likely peaking. Deaths are increasing slower than one would have expected, and could well peak a few weeks from now without having risen much further.

This, in context, is what success looks like.

As Florida and Arizona go now, so goes the rest of the second wave later. If the hardest hit places are turning things around without much underlying regulatory change, and with continued huge testing delays, then that tells us that some combination of herd immunity and voluntary behavioral changes is getting us there already.

It was a quiet week by 2020 standards.

There was plenty of news, most importantly the whole thing where the Federal government graduating from banditry and piracy to kidnapping people off the streets in unmarked vans, which is outrageous and a dire threat to our freedom. That does not seem to be getting the proper level of alarm or attention.

There was also the explorations of GPT-3, which also does not seem to be getting the proper level of alarm or attention. People are literally talking about AI taking jobs away from programmers without stopping to think that maybe that might be burying the lead a little. We like to say There is No Fire Alarm for Artificial General Intelligence but let’s raise the possibility that the alarm is there and it’s working fine and it’s us that are the problem.

If you want the most fun improv experience I’ve ever seen that’s also kind of a harbinger of the end of the world, try the dragon model of AI dungeon here. It’s not as good as the pure prompt version, but it’s available to the public.

Also, Slate Star Codex is back! And there was much rejoicing. Woo-hoo.

In other great news, parents in San Francisco have realized that they can get into groups and hire teachers, and provide better educations in healthier and safer environments while spending less money than schools do, and can do so while hiring more teachers, giving them safer, more interesting and more rewarding jobs, and paying them more. It’s almost like we should have been doing this all along, and might not want to go back after the epidemic ends. I rat on San Francisco a lot, so credit where credit is due.

But none of that is relevant to Covid-19. That’s not what we’re here to do right now.

Instead, let’s run the numbers.

Positive Test Counts

Date

...
Are parasitic worms an effective weapon against MS?
by Demeter — 2020-07-23T13:59:48.327Z

I'm in the process of acquiring hookworms to trial within my own body. While I'm doing that, I'm researching the effectiveness of using immune modulating parasites to treat rheumatoid arthritis and multiple sclerosis, and I'd love a little help. My reasons for doing all this are given below. But first, some background on what the hell I'm talking about.

Some research groups are running clinical trials where helminths (intestinal worms) are administered to patients with autoimmune disease and other inflammatory conditions (University of Wisconsin School of Medicine and Public Health, University of Nottingham, Australian Institute of Tropical Health & Medicine). As a result of increasing activity within this field over the last ten years, a small online community has formed around helminthic therapy, with the aim of making parasitic worms widely accessible to people with inflammatory conditions ranging from severe allergies to MS.

Proponents of worm therapy have a long list of references not only demonstrating that helminths counter the inflammatory response, but also theorizing that the elevated frequency of inflammatory disease in higher-latitude and more developed nations is due in large part to the absence of immune mediating parasites that co-evolved with humanity near the equator (this concept is related to the hygiene hypothesis-- see this PNAS article for an update regarding they hygiene hypothesis, where helminths are specifically discussed).

However, not everyone is on board with the parasites-for-all initiative touted on pro-worm websites. Even when research teams have observed a positive effect, they have been reluctant to recommend live parasites as a long term treatment option, preferring to study the anti inflammatory secretions of the parasites for drug development. This strategy avoids the risks of parasite infection gone wrong, but proponents of helminth therapy are doubtful that a pharmaceutical approach will capture the full effect of live parasites. They are also concerned that the drugs may not arrive for a long time, whereas patients with progressive illnesses have time-sensitive needs.

Where I'm coming from and what I'm after:

I have casually followed the occasional news regarding helminth clinical trials for years, having a mother who lives with the chronic pain of rheumatoid arthritis. I'm ge...
Restocking
by jefftk — 2020-07-23T13:40:05.833Z

The pandemic has illustrated the resiliency benefits of keeping extras
on hand. During the grocery panic, when everyone was trying to stock
up on weeks' worth of groceries at the same time, people who had
extra set aside didn't need to go out to a potentially dangerous
environment. Similarly, by avoiding shopping at this time when there
was sudden massive demand, these people were able to help blunt the
shock.

It's far from over, but at least around here the supply chain seems to
mostly have recovered. You still can't get N95 masks, but there's no
trouble getting flour, rice, beans, toilet paper, etc. If you've let
your supply run down over the last few months, or didn't have extra
set aside to begin with, now would be a good time to think about
fixing that. Not only could the pandemic get worse in a way that
starts to threaten the supply chain again, but many more conventional
disasters, like earthquakes or hurricanes, could be much worse with
the pandemic as a background.

As I wrote just before the pandemic, it's worth putting
time into thinking about potential disasters and getting at least
somewhat prepared for them. Lots of links in that post if you're
looking for advice on how to approach this.

Comment via: facebook
GPT-3 Gems
by TurnTrout — 2020-07-23T00:46:36.815Z

GPT-3 is captivating, and not just because of its potential reasoning abilities. This post will be a living collection of my favorite experiences with the network.

Bold text is my input; square brackets contain my commentary. As of 8/19/20, AIDungeon's Custom mode's first completion is sampled from GPT-2. Therefore, initial replies are not from GPT-3, except in the following subsections (in which I bypassed the mechanism):

Title sensationalization

As of 4/21/21, I have API access; model completions come from the davinci model - not AIDungeon:

Another (outer) alignment failure story

Long-form Writing

Beisutsukai Class Project: Solve Intent Alignment

"I've been thinking about this for far longer than you have. Building an AI is like building a person; it's not just a matter of throwing some parts together and suddenly you have a sentient being that does exactly what you want. AI design is a delicate art that requires hundreds of precise calibrations to get right."

Styrlyn slumped back in his chair, a gloomy look on his face. "And... we just don't have time to do that." He stared at the table.

~ GPT-3

I prompted GPT-3 with modified versions of Eliezer's Beisutsukai (ベイズ使い; Japanese: "Bayes-user") stories (prompt available here), where the class project now involves solving intent alignment instead of quantum gravity. In my opinion, some of its proposed alignment solutions are higher-quality than most solutions proposed by newbie LessWrongers back in the day.

I did a few rerolls and trimming of intermediate outputs; assume an average of 3 per completion.

... Taji looked over his sheets. "Okay, I think we've got to assume that every avenue that Eld science was trying is a blind alley, or they would have found it. And if this is possible to do in one month, the answer must be, in some sense, elegant. So no human mistake models. If we start doing anything that looks like we should call it 'utility function patching', we'd better stop."

He turned to Brennan. "Do you have any ideas?"

"I don't think so," she said. She glanced at her paper, frowning. "There are two problems here."

One problem was simple enough: she didn't understand what it meant for an AI to "want" to help a human. This was hard because as far as she could tell, there hadn't been much research done on this question. It seemed to be something that only humans did, but there wasn't any evidence that this was tru

...
Weak HCH accesses EXP
by evhub — 2020-07-22T22:36:43.925Z

This post is a follow-up to my “Alignment proposals and complexity classes” post. Thanks to Sam Eisenstat for helping with part of the proof here.

Previously, I proved that imitative amplification with weak HCH, approval-based amplification, and recursive reward modeling access PSPACE while AI safety via market making accesses EXP. At the time, I wasn't sure whether my market making proof would generalize to the others, so I just published it with the PSPACE proofs instead. However, I have since become convinced that the proof does generalize—and that it generalizes for all of the proposals I mentioned—such that imitative amplification with weak HCH, approval-based amplification, and recursive reward modeling all actually access EXP. This post attempts to prove that.

Updated list of proposals by complexity class

P: Imitation learning (trivial)

PSPACE: AI safety via debate (proof)

EXP: AI safety via market making (proof), Imitative amplification with weak HCH (proof below), Approval-based amplification (proof below), Recursive reward modeling (proof below)

NEXP: Debate with cross-examination (proof)

R: Imitative amplification with strong HCH (proof), AI safety via market making with pointers (proof)

Proofs

Imitative amplification with weak HCH accesses EXP

The proof here is similar in structure to my previous proof that weak HCH accesses PSPACE, so I'll only explain where this proof differs from that one.

First, since l∈EXP, we know that for any x∈X, Tl(x) halts in O(2poly(n)) steps where n=|x|. Thus, we can construct a function fl(n)=c1+c2ec3nc4 such that for all x∈X, Tl(x) halts in less than or equal to fl(x) steps by picking c3, c4 large enough that they dominate all other terms in the polynomial for all n∈N. Note that fl is then computable in time polynomial in n.

Second, let H's new strategy be as follows:

Given p, let s, x=M(p:f(|x|)). Then, return accept/reject based on whether s is an accept or reject state (it will always be one or the other)

Given p:0, return s0, x0 where s0 is the starting state and x0 is the empty tape symbol.

Given p:i, let
s, x=M(p:i−1)xleft=M(p:i−1:−1)xright=M(p:i−1:1).
Then, return s′, x′ where s′ is the next state of Tl from state s over symbol x (where if we're already in an accept/reject state we just stay there) and x′ is the new tape symbol at the head (which can be determined given s, x, xlef

...
Replicating the replication crisis with GPT-3?
by skybrian — 2020-07-22T21:20:34.865Z

I am getting worried that people are having so much fun doing interesting stuff with GPT-3 and AI Dungeon that they're forgetting how easy it is to fool yourself. Maybe we should think about how many different cognitive biases are in play here? Here are some features that make it particularly easy during casual exploration.

First, it works much like autocomplete, which makes it the most natural thing in the world to "correct" the transcript to be more interesting. You can undo and retry, or trim off extra text if it generates more than you want.

Randomness is turned on by default, so if you try multiple times then you will get multiple replies and keep going until you get a good one. It would be better science but less fun to keep the entire distribution rather than stopping at a good one. Randomness also makes a lot of gamblers' fallacies more likely.

Suppose you don't do that. Then you have to decide whether to share the transcript. You will probably share the interesting transcripts and not the boring failures, resulting in a "file drawer" bias.

And even if you don't do that, "interesting" transcripts will be linked to and upvoted and reshared, for another kind of survivor bias.

What other biases do you think will be a problem?
A New Way to Visualize Biases
by dtm — 2020-07-22T17:57:48.245Z

I have started using a visual metaphor to diagram biases in my attempts to remove and mitigate them in myself. I have found this to be incredibly useful, particularly when dealing with multiple compounding biases.

I view an inference as an interaction between external inputs/premises and the resulting cognitions/conclusions. It can be read either as "if x then y," or "x therefore y." A basic inference looks like this:

A biased inference looks like this:

This is obviously a simplification of complex cognitive shit, but it's meant to be more of a functional interface than any kind of theory.

So to run through a few example biases, the fallacy of the undistributed middle:

The planning fallacy:

Planning fallacy corrected:

A little awkward, but it can capture basic failures in Bayesian reasoning as well:

Bayesian reasoning corrected:

And an example of compounding biases resulting in a distorted worldview:

I'm curious if anyone sees downsides to this framework, has other ideas to improve it, or thinks I'm hopelessly naive for even trying to capture human reasoning in a tidy diagram.
[AN #109]: Teaching neural nets to generalize the way humans would
by Rohin Shah — 2020-07-22T17:10:04.508Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

HIGHLIGHTS

Better priors as a safety problem and Learning the prior (Paul Christiano) (summarized by Rohin): Any machine learning algorithm (including neural nets) has some inductive bias, which can be thought of as its “prior” over what the data it will receive will look like. In the case of neural nets (and any other general ML algorithm to date), this prior is significantly worse than human priors, since it does not encode e.g. causal reasoning or logic. Even if we avoid priors that depended on us previously seeing data, we would still want to update on facts like “I think therefore I am”. With a better prior, our ML models would be able to learn more sample efficiently. While this is so far a capabilities problem, there are two main ways in which it affects alignment.

First, as argued in Inaccessible information (AN #104), the regular neural net prior will learn models which can predict accessible information. However, our goals depend on inaccessible information, and so we would have to do some “extra work” in order to extract the inaccessible information from the learned models in order to build agents that do what we want. This leads to a competitiveness hit, relative to agents whose goals depend only on accessible information, and so during training we might expect to consistently get agents whose goals depend on accessible information instead of the goals we actually want.

Second, since the regular neural net prior is so weak, there is an incentive to learn a better prior, and then have that better prior perform the task. This is effectively an incentive for the neural net to learn a mesa optimizer (AN #58), which need not be aligned with us, and so would generalize differently than we would, potentially catastrophically.

Let’s formalize this a bit more. We have some evidence about the world, given by a dataset D = {(x1, y1), (x2, y2), ...} (we assume that it’s a prediction task -- note that most self-supervised tasks can be written in this form). We will later need to make predictions on the dat...
Online Meetup: Balaji Srinivasan on "Nation of Emigrants"
by JoshuaFox — 2020-07-22T14:35:53.969Z

In our ongoing SlateStarCodex online series, Balaji Srinivasan will speak on "A Nation of Emigrants: How Technology Will Enable Collective Bargaining with Governments". See abstract below.

After the talk and Q&A, we will also allow plenty of time for online mingling and discussion.

Click here to register, up to an hour before the talk, and we'll send you an invitation to the Google Meet.

The talk is August 2 at at 10:30 PDT, 17:30 GMT, 20:30 IDT. This link shows your local time.

Abstract:

Technology is the driving force of history. It lies upstream of culture, and thus upstream of politics. And it determines which fringe ideas have become newly feasible, and what elements of official consensus have become suddenly obsolete.

Our thesis has five stages. First, we describe how recent technological developments have dramatically shifted the balance of power away from states and towards individuals. Next, we diagnose the problems with existing states - particularly related to immigration and regulation - that would drive individuals to seek alternatives. Third and fourth, we give a concrete set of recommendations enabled by modern technology that companies and individuals alike can act upon to gain financial independence.

Finally, we describe how the combination of technological driving forces, regulatory repulsion, and newly independent individuals leads to a novel phenomenon that we term crowdchoice: internet-coordinated mass migrations that enable collective bargaining with governments.