trentmkelly/LessWrong-43k huggingface dataset:
How superforecasting could be manipulated
by DirectedEvolution — 2020-04-17T06:47:51.289Z

Introduction

What if Philip Tetlock wanted to game his own superforecasting system? He runs his own consulting firm. The information he's selling is not available to the public. How would he do it in a way that isn't violating any of the rules he's set up for forecasting?

I'm not saying he is, not at all. I respect Tetlock greatly, admire the work he's done, and seek to learn from it. In fact, it's in the spirit he brings to his enterprise that I'm questioning it. After all, his work is based on calling out the wiggle room "expert" commentators give themselves, and trying to get rid of it. I'm examining whether there might yet be a little bit of wiggle room left in the system.

How superforecasting could be manipulated

My understanding is that Tetlock designates as superforecasters the top 2% of participants in his tournaments. He evaluates their accuracy to see whether it stays steady or regresses to the mean, and finds that superforecasters really are better than the average person at predicting complex world events.

In theory, the way the system could be manipulated is in the selection of questions.

Superforecasters are eventually set apart from regular forecasters. For example, the questions answered at Good Judgment, Tetlock's professional consulting firm, are not visible to the public. I have to assume they're not (all) mirrored at Good Judgment Open, his open-to-all platform for forecasting. Hence, no way for even a client to compare how superforecasters do on these questions compared to the average Joanne.

How could their accuracy be inflated, even while answering unambiguous questions and being evaluated in a purely algorithmic fashion?

By choosing primarily questions that are in fact easy to answer for a professional, yet appear difficult to the typical client.

Here are two versions of the same question:

"Will Donald Trump and Kim Jong Un meet on or before Dec. 31, 2020?"

"Will Donald Trump and Kim Jong Un meet in Seoul on or before Dec. 31, 2020?"

Most readers here are savvy enough to realize that the odds of these world leaders meeting in Seoul are lower than them meeting at all, and familiar enough with the conjunction fallacy to know that "in Seoul" may actually make the second version seem more plausible to some readers. A professional superforecaster (or question wri...
Two-player Catan
by jefftk — 2020-04-17T00:50:03.239Z

Here's a way to play Settlers of Catan with two players that

David and I worked out this
weekend. We've only played one game with it so far, and didn't think
very hard about tradeoffs, so there are probably better ways to do it.

Set up the board as normal.

Each player controls two colors. We did white/blue and
red/orange.

Placement is A B B' A' A' B' B A

Turns aren't color specific, except that you can only build for
one color on a turn.

If you own a port with either color you can fully use it on
your turn.

Development cards are color specific, which really only matters
for the 1 VP cards

Each player has one hand, and the robber limit is 14 instead of
7. Possibly this should be more like ten or even seven.

The game still ends when one color gets to 10 VP, but the goal
is to not come in last. Whichever player has the color with the
fewest VP at the end of the game loses. The player who got to 10 VP
first wins ties.

Trading is allowed, I guess, though since the game is zero-sum
you wouldn't actually trade with each other.

Hands are open, not to
facilitate trading but to make the strategy more interesting by
letting you better model your opponent.

We played two games, but stopped the first one very early on. We
agreed that one of us had an 80% chance of winning and so counted it
as 80% of a win. Like a doubling
cube this helps concentrate your time onto interesting games.

I enjoyed playing, though I don't know how many more games of it I'd
like to play. In generally I enjoy the dynamic of games with more
people a lot more, though under the current circumstances that's not
always practical. It has less waiting than regular Catan, which is a
plus.

I think the 36-card variant ("gamblers fallacy dice") would help here
a lot more than in regular Catan, since otherwise the game can get
unbalanced quickly due to quirks of the die. In general I like the
way the card variant adds strategy, especially if it's easy for the
players to tell what cards are yet to come.

Because there's no trading, ports are a lot more important than in a
regular game. David picked a port-heavy strategy while I went for
variety, and (n=1) ports works better.

After playing I looked around for other sets of two-player rules and
there are a
lot of
variants out
there! I haven't looked through them in detail to figure
out whether I like this one best.

...
Integrating Hidden Variables Improves Approximation
by johnswentworth — 2020-04-16T21:43:04.639Z

Fun fact: the KL divergence of distribution P[X] from distribution Q[X] is convex in the pair P,Q. Writing it out: DKL(λP1[X]+(1−λ)P2[X]||λQ1[X]+(1−λ)Q2[X])≤λDKL(P1[X]||Q1[X])+(1−λ)DKL(P2[X]||Q2[X]) with 0≤λ≤1.

This is particularly interesting if we take P and Q to be two different models, and take the indices 1, 2 to be different values of another random variable Y with distribution P[Y] given by (λ,1−λ). In that case, the above inequality becomes:

DKL(P[X]||Q[X])≤EY[DKL(P[X|Y]||Q[X|Y])]

In English: the divergence between our models of the X-distribution ignoring Y is at least as small as the average divergence between our models of the X-distribution given Y. This is true regardless of what the two models are - any approximation of the observable distribution improves (or gets no worse) when we integrate out a hidden variable, compared to fixing the value of the hidden variable.

Of course, this doesn't say anything about how much the approximation improves. Presumably for bad approximations, the divergence will not converge to anywhere near zero as we integrate more and more hidden variables. And if the hidden variable doesn't actually interact with the observables significantly, then presumably the divergence decrease will be near-zero.

So when would we expect this to matter?

I'd expect it to matter mainly when the observable X consists of multiple variables which are "far apart" in a large model - i.e. there are many hidden variables mediating the interactions between observables. In other words, I'd expect this phenomenon to mainly be relevant to information at a distance. It's a hint that information at a distance, in complex systems, converges to some sort of universal behavior/properties, which is simpler in some sense than the full fine-grained model.
Treatments correlated with harm
by Connor_Flexman — 2020-04-16T21:02:58.126Z

There's a billion different ways studies can go wrong through failure of statistics to correctly capture reality.
One very specific one is: your successful treatments will often look like they're making things worse.
This is merely a special case of confounding, but it comes up often enough I want to highlight it. And it seems specifically important for the discussion of masks and respirators.
For example (not a doctor): every new leukemia drug has a survival rate of like 30%. Base rate survival is like 60%. Does every new leukemia drug work worse than normal? Obviously not, it's that the patients trying the new drug are the ones for whom everything else has failed, so they've been signed up for a clinical trial. These patients have very resistant cancers. And we can verify this, because when the drug does get adopted then survival rates go up.
Doctors have mostly figured this out with this subset of leukemia drugs. They do adjustments for how resistant the crazy-resistant cancers in the trial are, the adjustments aren't good enough and show the drug has no effect or is bad, they throw the adjustments out and do something else (like compare the results of this new drug to some other new drug that's almost like a control group), and they eventually seem to come to decent conclusions about whether to use the new drug in a setting that's more conducive to sane trial results.
However, in a lot of cases, they don't seem to figure this out, and it results in truly terrible treatment policies. The clearest example I've seen is in a few localized forms of cancer (like various sarcomas), where amputation shows vastly worse survival than local resection. As crazy as this seemed, it was minutely possible that side effects of amputation were in fact so bad that this was true, but after looking into it extensively I am very sure that in most extremity amputations it is not true. Doctors who see more aggressive tumors advocate amputation, these more aggressive tumors have worse overall survival regardless of how you treat, and amputation winds up looking statistically worse than local resection. And now the researchers who looked at this are advocating for fewer amputations, which will cost lives.
I've seen this in other cases too. The most recent: in patients with COPD, taking inhaled corticosteroids (ICS) is associated with h...
Evaluating Predictions in Hindsight
by Zvi — 2020-04-16T17:20:01.054Z

Epistemic Status: Confident I have useful things to say, but apologies for the long post because I don’t think it’s worth my time to make it shorter. Better to get thoughts down for those who want to read them.

Scott Alexander’s latest post points to the question of how best to evaluate predictions. The way he characterized leading predictions on Trump and Brexit, that ‘prediction is hard,’ instinctively bothered me. Characterizing the coronavirus situation the same way bothered me even more.

(Before anyone asks, I was ahead of the public but definitely dropped the ball on the early coronavirus prediction front, in the sense that I failed to make a lot of money and failed to warn the public, and waited far too long to move out of New York City. A large part of that fault was that I predicted things badly by my own standards. A large part was other failures. I take full responsibility. But that’s not what I want to talk about right now.)

How does one evaluate past predictions?

As someone who used to place wagers and/or price prediction markets for a living, and later traded with what I believe are some of the best traders on the planet, I’ve thought about this question a lot.

We can divide situation into easy mode, where there are a large number of independent predictions made and robust markets or probabilities against which one can evaluate those predictions, and hard mode, where this is not true, and you are often evaluating an individual prediction of a singular event.

Easy Mode

Most of my time was spent largely in the ‘easy mode.‘ Here, easy mode is when one is predicting lots of things for which there are established market prices, or at a minimum baseline fair values that one can be evaluated against. You have a lot of data points, and are comparing your predictions and decisions to a known baseline.

Easy mode makes it realistic to seek a metric that cannot be easily fooled, where you can use your results as evidence to prove what you are doing ‘works’ in some sense.

There is no one metric that is best even in easy mode. There are a few different ones that have merit. I’ll go through at least some of them.

Method One: Money Talks, Bull*** Walks

Did you make money?

If you did, congratulations. Good predicting.

If you didn’t, sorry. Bad predicting. If you didn’t bet, it doesn’t count.

This method has a lot to recommend it. It’s especially great over long periods o...
Deminatalist Total Utilitarianism
by Vanessa Kosoy — 2020-04-16T15:53:13.953Z

TLDR: I propose a system of population ethics that arguably solves all major paradoxes, and formalizes some intuitions that prior systems have not. AFAIK this is original but I'm not an expert so maybe it's not.

This idea was inspired by a discussion in the "EA Corner" discord server, and specifically by a different proposal by discord user LeoTal.

Foreward: The Role of Utilitarnism

I don't believe utilitarianism is literally the correct system of ethics (I do endorse consequentialism). Human values are complex and no simple mathematical formula can capture them. Moreover, ethics is subjective and might different between cultures and between individuals.

However, I think it is useful to find simple approximate models of ethics, for two reasons.

First, my perspective is that ethics is just another name for someone's preferences, or a certain subset of someone's preferences. The source of preferences is ultimately intuition. However, intuition only applies to the familiar. You know that you prefer strawberries to lemons, just because. This preference is obvious enough to require no analysis. But, when you encounter the unfamiliar, intuition can fail. Is it better to cure a village of malaria or build a new school where there is none? Is it better to save one human or 1000 dogs? Can a computer simulation be worthy of moral consideration? What if it's homomorphically encrypted? Who knows?

In order to extrapolate your intuition from the familiar to the unfamiliar, you need models. You need to find an explicit, verbal representation that matches your intuition in the familiar cases, and that can be unambiguously applied to the unfamiliar case. And here you're justified to apply some Occam razor, despite the complexity of values, as long as you don't shave away too much.

Second, in order to cooperate and coordinate effectively we need to make our preferences explicit to each other and find a common ground we can agree on. I can make private choices based on intuition alone, but if I want to convince someone or we want to decide together which charity to support, we need something that can be communicated, analyzed and debated.

This is why I think questions like population ethics are important: not as a quest to find the One True Formula of morality, but as a tool for decision making in situations that are unintuitive and/or require cooperation.

Motivation

The system I pr

...
Leaders support global truce to turn fire on coronavirus
by JohnCDraper — 2020-04-16T14:49:55.210Z

International leaders are throwing their weight behind a proposal for a global ceasefire as the number of coronavirus cases worldwide passes the 2 million mark.

President Macron of France said yesterday that President Trump, President Xi, and Boris Johnson had all confirmed to him that they backed the plea for a truce made by António Guterres, secretary-general of the UN.

Mr Macron said he was hopeful of securing the support of President Putin, leader of the fifth permanent member of the Security Council...

See https://www.thetimes.co.uk/article/leaders-support-gloal-truce-to-turn-fire-on-coronavirus-r00p5wlw2
Brief Response to Suspended Reason on Parallels Between Skyrms on Signaling and Yudkowsky on Language and Evidence
by Zack_M_Davis — 2020-04-16T03:44:06.940Z

(This is the continuation of a Twitter conversation, hosted as a Less Wrong post because Less Wrong lets me post more than 280 characters at a time.)

I'm having trouble finding a "juicy" passage to quote due to ... uh, genre differences? Skyrms isn't preaching about the True Nature of Language to a popular audience, but rather explaining game-theory models to an academic audience with occasional remarks about how these ideas apply to human language, animal behavior, cellular systems, &c. The nine-page paper "Evolution of Signalling Systems with Multiple Senders and Receivers" gives a much shorter taste of the technical content, but I don't anticipate you counting it as a precedent on its own, so I'll give my brief summary of some of what I got out of Signals: Evolution, Learning, and Information and how it parallels Yudkowsky's work on language and evidence.

(One could argue that the fact that I feel like I have to do my own summarizing rather than just quoting Skyrms is a hint that what I'm seeing, I got from Skyrms-reinterpreted-in-light-of-Yudkowsky, rather than it being easy to get from Skyrms himself. But I think the insight is mostly in the math, and the relative ease of juicy quoting is just a genre difference.)

Skyrms expounds on sender–receiver games as introduced by David Lewis in 1969. We have two abstract agents, a "sender" and a "receiver". The sender gets to observe some state of Nature and send a "signal" to the receiver, who will then take some action, and get a payoff depending on the action taken and the state of Nature. For simplicity, suppose there are two states (S1, S2), two possible signals (s1, s2), and two possible acts (A1, A2), and the receiver gets a better payoff when it takes action A1 (respectively A2) when reality is in S1 (respectively S2).

The thing is, the signals are completely opaque—the receiver can distinguish one from the other, but there's no reason for s1 to mean that the state of Nature is S1 rather than S2—the names are totally arbitrary. (I could have said "red" and "blue", or "♥" and "❄" rather than s1 and s2.)

And yet, if you run an evolutionary-game-theory simulation where behavior that gets payoffs is more likely to be copied, our agents learn to form one of the two consistent signaling systems where each of the signals corresponds to one of the states of Nature.

And that's where language comes from!! Starting w

...
AI Alignment Podcast: An Overview of Technical AI Alignment in 2018 and 2019 with Buck Shlegeris and Rohin Shah
by Palus Astra — 2020-04-16T00:50:37.578Z

Just a year ago we released a two part episode titled An Overview of Technical AI Alignment with Rohin Shah. That conversation provided details on the views of central AI alignment research organizations and many of the ongoing research efforts for designing safe and aligned systems. Much has happened in the past twelve months, so we've invited Rohin — along with fellow researcher Buck Shlegeris — back for a follow-up conversation. Today's episode focuses especially on the state of current research efforts for beneficial AI, as well as Buck's and Rohin's thoughts about the varying approaches and the difficulties we still face. This podcast thus serves as a non-exhaustive overview of how the field of AI alignment has updated and how thinking is progressing.

Topics discussed in this episode include:

Rohin's and Buck's optimisms and pessimism about different approaches to aligned AI
Traditional arguments for AI as an x-risk
Modeling agents as expected utility maximizers
Ambitious value learning and specification learning/narrow value learning
Agency and optimization
Robustness
Scaling to superhuman abilities
Universality
Impact regularization
Causal models, oracles, and decision theory
Discontinuous and continuous takeoff scenarios
Probability of AI-induced existential risk
Timelines for AGI
Information hazards

You can find the page for this podcast here: https://futureoflife.org/2020/04/15/an-overview-of-technical-ai-alignment-in-2018-and-2019-with-buck-shlegeris-and-rohin-shah/

Transcript

Note: The following transcript has been edited for style and clarity.

Lucas Perry: Welcome to the AI Alignment Podcast. I’m Lucas Perry. Today we have a special episode with Buck Shlegeris and Rohin Shah that serves as a review of progress in technical AI alignment over 2018 and 2019. This episode serves as an awesome birds eye view of the varying focus areas of technical AI alignment research and also helps to develop a sense of the field. I found this conversation to be super valuable for helping me to better understand the state and current trajectory of technical AI alignment research. This podcast covers traditional arguments for AI as an x-risk, what AI alignment is, the modeling of agents as expected utility maximizers, iterated distillation and amplification, AI safety via debate, agency and optimization, value learning, robustn...
Noise Simplifies
by johnswentworth — 2020-04-15T19:48:39.452Z

They say that the flap of a butterfly’s wings can change the course of a hurricane. Sadly, this does not give us an effective strategy for controlling hurricane trajectories - if a flap of my butterfly’s wings can change the hurricane’s course, then so can the flap of any other butterfly’s wings. Unless I know exactly how all of the world’s butterflies are flapping, I do not know how my own butterfly’s wings must flap to achieve the course I want.

On the other hand, even though I do not know how the vast majority of the world’s butterflies are flapping, it is far easier to estimate a distribution of possible hurricane-paths in my state of ignorance than it would be to compute the exact hurricane path in a state of full knowledge.

In general, we usually think of computing with uncertainty as harder than computing without it - Bayesian reasoning requires tracking all possible worlds, which is exponentially more difficult than just tracking one. Yet in practice, uncertainty often makes large systems simpler to reason about. Why? Because noise wipes out long-range correlations. Unless I know how the wings of all the world’s butterflies are flapping, the flapping of any particular butterfly gives me no information at all about the path of a hurricane. So in practice, weather forecasters do not need to worry about tracking butterflies.

Let’s look at a toy model, to build some intuition for this idea.

I have a long list of randomly-chosen numbers between 1 and 10, and I want to know whether their sum is even or odd. If I know all of the numbers, then I can perform a relatively complicated calculation: I can add them all up, divide by 2, and find the remainder. But if there is even just one number in the list whose value I do not know, then all the rest of the numbers tell me absolutely nothing about whether the sum is even or odd. Even just a little bit of noise wipes out all of the signal completely.

On the other hand, if I don’t know the values of one or more of the numbers, then my “model” for this system is much simpler: whether the sum is even or odd is independent of all the numbers I know. If I’m reasoning about the sum, I can just forget about all those numbers entirely. In this sense, noise makes the system much simpler to reason about; there’s no need for all that...
Probabilistic Blue-Eyed Islanders
by johnswentworth — 2020-04-15T17:55:00.429Z

A puzzle: what happens in the blue-eyed islanders puzzle if we allow a small probability p for each each person to make a mistake each night? Assume that the probability of a mistake is the same for each and is common knowledge, the islanders are all ideal Bayesian reasoners, and let's say that 80% is the confidence cutoff past which someone leaves.