trentmkelly/LessWrong-43k huggingface dataset:
Metaculus Introduces New 'Conditional Pair' Forecast Questions for Making Conditional Predictions
by ChristianWilliams — 2023-02-20T13:36:19.649Z
Link: https://www.metaculus.com/questions/15205/introducing-conditional-pairs/

Predict P(A|B) & P(A|B') With New Conditional Pairs

Events don't take place in isolation. Often we want to know the likelihood of an event occurring if another one does.

Metaculus has launched conditional pairs, a new kind of forecast question that enables forecasters to predict the probability of an event given the outcome of another, and provides forecast consumers with greater clarity on the relationships between events.

This post explains the motivation behind conditional pairs, how to interpret them, and how to start making conditional forecasts.

(Check out this video explainer for more on conditional pairs and how to forecast with them.)

How Do Conditional Pairs Work?

A conditional pair poses two conditional questions (or "conditionals"):

If Question B resolves Yes how will Question A resolve?
If Question B resolves No how will Question A resolve?

For example, a forecaster may want to predict on a question such as this:

Will Alphabet’s Market Capitalization Fall Below $1 Trillion by 2025?

The forecast depends—on many things. But consider one factor: Bing's share of the search engine market.

And so if one knew that Bing's search engine market would be least 5% in March of 2024, they could make a more informed forecast. They might assign greater likelihood to Alphabet's decline.

Our conditional pair is then:

If Bing's market share is more than 5% in March, 2024, will Alphabet's market capitalization fall below $1 trillion?
If Bing's market share is not more than 5% in March, 2024, will Alphabet's market capitalization fall below $1 trillion?

(Start forecasting on this conditional pair here.)

Two forecasters could have the same forecasts for Bing’s market share and Alphabet’s market cap while having very different mental models of their relationship. Conditional pairs help make these sometimes implicit differences explicit so they can be discussed and scored.

Start Forecasting on Conditional Pairs

Here are some newly created conditional pairs to start forecasting on:

If Human-Machine Intelligence Parity Is Reached by 2040, Will the US Impose Compute Capacity Restrictions Before 2050?

If Chinese GDP Overtakes US GDP by 2030, Will the US Go to War With China by 2035?

If There Is a Bilateral Ceasefire in Ukraine by 2024, Will There Be Large-Scale Conflict With Russia by 2030?

If There Is a US Debt Default by 2024, Will Democrats Win the 2024 US Presiden

...
On Investigating Conspiracy Theories
by Zvi — 2023-02-20T12:50:00.891Z

Scott Alexander wrote yet more words defending his decision to write two posts totaling 25,000 words about Ivermectin. Then he wrote a second post trying again.

More centrally, his first post, of which I very much approve, is defending the most important idea of all: Think for yourself, shmuck!

I want to make clear my violent agreement with all of the following.

Think for yourself, shmuck!

When it seems worthwhile, do your own research.

The ones telling you not to ‘do your own research’ are probably the baddies.

Also applies to the ones telling you to ‘trust us and Trust the Science™’ and calling you an idiot or racist or calling for you to be censored if you disagree.

Baddies or not, those people still are more likely to be more right about any given point than those saying they are wrong or lying to you, unless you have seen them lying or being wrong a lot about closely related things previously. And your own research will often not seem worthwhile if you consider opportunity costs.

When people draw wrong conclusions like Ivermectin being effective or that Atlantis used to exist or whatever, telling people that they are idiots or racists for drawing that conclusion is not going to be super effective.

Pointing out ‘the other side are conspiracy theorists’ or ‘the people who believe this also believe these other terrible things’ does not prove the other side is wrong, nor is it going to convince anyone on the other side that they are wrong.

If you instead explain and work through the evidence, there is a chance someone might be convinced, that is God’s work, you are providing a public service.

There are not ‘legitimate’ and ‘illegitimate’ places to Do Science. You can virtuously Do Science to It, for all values of It and of You.

No, we cannot assume that the medical establishment, or any other establishment, will always get such questions right. That is not how any of this works. Even the best possible version of the medical (or other) establishment will sometimes get it wrong, if no one points it out without being dismissed as a conspiracy theorist or racist then the establishment will keep getting it wrong and so will you, and criticism is the only known antidote to error in such situations.

I would also add, from Kavanagh’s response to Scott in a comment, my disagreement with this particular thing, regarding scuba divi...
The Estimation Game: a monthly Fermi estimation web app
by Sage Future — 2023-02-20T11:33:04.736Z

Announcing the first monthly

Estimation Game

:

Answer 10

Fermi estimation questions

, like “How many piano tuners are there in New York?”
Train your estimation skills and get more comfortable putting numbers on things
Team up with friends, or play solo
See how your scores compare on the

global leaderboard

The game is around 10-40 minutes, depending on how much you want to discuss and reflect on your estimates

You can

play The Estimation Game on Quantified Intuitions

, solo, or with friends. The February game is live for one week (until Sunday 26th).

We’ll release a new Estimation Game each month. Lots of people tell us they’d like to get more practice doing BOTECs and estimating, but they don’t get around to it. So we’ve designed The Estimation Game to give you the impetus to do a bit of estimation each month in a fun context.

You might use this as a sandbox to experiment with different methods of estimating. You could decompose the question into easier-to-estimate quantities - make estimates in your head, discuss with friends, use a bit of paper, or even build a scrappy

Guesstimate

or

Squiggle

model.

We’d appreciate your feedback in the comments, in our

Discord

, or at

adam@sage-future.org

. We’d love to have

suggestions for questions

for future rounds of The Estimation Game - this will help us keep the game varied and fun in future months!

Info for organisers

If you run a community group or meetup, we’ve designed the Estimation Game to be super easy to run as an off-the-shelf event. Check out our

info for organisers

page for resources and FAQs.

If you’re running a large-scale event and want to run a custom Estimation Game at it, let us know and we can help you set it up. We’re planning to pilot Estimation Games at EAGx Nordics (and maybe EAGx Cambridge).

About Quantified Intuitions

We built Quantified Intuitions as an epistemics training site. See our

previous post for more on our motivation

. Alongside the monthly

Estimation Game

, we’ve made two permanent tools:

Pastcasting

: Predict past events to rapidly practise forecasting

Calibration

: Answer EA-themed trivia questions to calibrate your uncertainty

Thanks to our test groups in London, to community builders who gave feedback, in particular Robert Harling, Adash Herrenschmidt-Moller, and Sam Robinson, and to Chana Messinger at CEA for the...
The idea that ChatGPT is simply “predicting” the next word is, at best, misleading
by Bill Benzon — 2023-02-20T11:32:06.635Z

Cross-posted from New Savanna.

But it may also be flat-out wrong. We’ll see when we get a better idea of how inference works in the underlying language model.

* * * * *

Yes, I know that ChatGPT is trained by having it predict the next word, and the next, and the next, for billions and billions of words. The result of all that training is that ChatGPT builds up a complex structure of weights on the 175 billion parameters of its model. It is that structure that emits word after word during inference. Training and inference are two different processes, but that point is not well-made in accounts written for the general public.

Let's get back to the main thread.

I maintain, for example, that when ChatGPT begins a story with the words “Once upon a time,” which it does fairly often, that it “knows” where it is going and that its choice of words is conditioned on that “knowledge” as well as upon the prior words in the stream. It has invoked a ‘story telling procedure’ and that procedure conditions its word choice. Just what that procedure is, and how it works, I don’t know, nor do I know how it is invoked. I do know, that it is not invoked by the phrase “once upon a time” since ChatGPT doesn’t always use that phrase when telling a story. Rather, that phrase is called up through the procedure.

Consider an analogy from jazz. When I set out to improvise a solo on, say, “A Night in Tunisia,” I don’t know what notes I’m going to play from moment to moment, much less do I know how I’m going to end, though I often know when I’m going to end. How do I know that? That’s fixed by the convention in place at the beginning of the tune; that convention says that how many choruses you’re going to play. So, I’ve started my solo. My note choices are, of course, conditioned by what I’ve already played. But they’re also conditioned by my knowledge of when the solo ends.

Something like that must be going on when ChatGPT tells a story. It’s not working against time in the way a musician is, but it does have a sense of what is required to end the story. And it knows what it must do, what kinds of events must take place, in order to get from the beginning to the end. In particular, I’ve been working with stories where the trajectories have five segments: Donné, Disturb, Plan, Execute, Celebrate. The whole trajectory is ‘in place’ when ChatGPT begins telling the sto...
Russell Conjugations list & voting thread
by Daniel Kokotajlo — 2023-02-20T06:39:44.021Z

Russell conjugations humorously illustrate how the same concept can be expressed with different phrasings that carry positive, neutral, or negative valence. Examples below.

I wish they were taught in schools. I think it's a high bang-for-buck rationality technique, to 'conjugate' and translate back and forth between the positive and negative valences. If you can do this effortlessly and automatically, you are more likely to do so in situations where you are in danger of making epistemic or moral errors if you don't.
Below in the comments I'm making a list of Russell conjugations I've heard and generated. Please add more of your own. Agreement-upvote them if you think they are a solid contribution to the list, agreement-downvote them if you think the overall list would be better off without them.
If we get enough, maybe it'll be a useful fine-tuning or few-shot prompt for LLMs. (The votes will be helpful for quality control.) Maybe. I can dream.
Emergent Deception and Emergent Optimization
by jsteinhardt — 2023-02-20T02:40:09.912Z

[Note: this post was drafted before Sydney (the Bing chatbot) was released, but Sydney demonstrates some particularly good examples of some of the issues I discuss below. I've therefore added a few Sydney-related notes in relevant places.]

I’ve previously argued that machine learning systems often exhibit emergent capabilities, and that these capabilities could lead to unintended negative consequences. But how can we reason concretely about these consequences? There’s two principles I find useful for reasoning about future emergent capabilities:

If a capability would help get lower training loss, it will likely emerge in the future, even if we don’t observe much of it now.

As ML models get larger and are trained on more and better data, simpler heuristics will tend to get replaced by more complex heuristics.

Using these principles, I’ll describe two specific emergent capabilities that I’m particularly worried about: deception (fooling human supervisors rather than doing the intended task), and optimization (choosing from a diverse space of actions based on their long-term consequences).

Deception is worrying for obvious reasons. Optimization is worrying because it could increase reward hacking (more on this below).

I’ll start with some general comments on how to reason about emergence, then talk about deception and optimization.

Predicting Emergent Capabilities

Recall that emergence is when qualitative changes arise from quantitative increases in scale. In Future ML Systems will be Qualitatively Different, I documented several instances of emergence in machine learning, such as the emergence of in-context learning in GPT-2 and GPT-3. Since then, even more examples have appeared, many of which are nicely summarized in Wei et al. (2022). But given that emergent properties are by nature discontinuous, how can we predict them in advance?

Principle 1: Lower Training Loss

One property we can make use of is scaling laws: as models become larger and are trained on more data, they predictably achieve lower loss on their training distribution. Consequently, if a capability would help a model achieve lower training loss but is not present in existing models, it’s a good candidate for future emergent behavior.[1]

This heuristic does a good job of retrodicting many past examples of emergence. In-context learning helps decrease the training loss, since knowing “what sort of...
AGI doesn't need understanding, intention, or consciousness in order to kill us, only intelligence
by James Blaha — 2023-02-20T00:55:34.329Z

Why the development of artificial general intelligence could be the most dangerous new arms race since nuclear weapons

The rise of transformer-based architectures, such as ChatGPT and Stable Diffusion, has brought us one step closer to the possibility of creating an Artificial General Intelligence (AGI) system — a technology that can perform any intellectual task that a human being can. While nearly all current AI systems are designed to narrowly perform specific tasks, AGI would be capable of adapting to new situations and learning from them, with flexibility and adaptability similar to that of humans.

The potential benefits of AGI are undeniable, promising a world in which we can automate drudgery, create wealth on an unprecedented scale, and solve some of the world’s most pressing problems. However, as we move closer to realizing the dream of AGI, it’s essential that we consider the risks that come with this technology.

These risks range from the potential for job displacement, bias, weaponization, misuse, and abuse to unintended side effects and the possibility of unaligned and uncontrolled goal optimization. The latter is of particular concern, as it poses an existential risk to humanity, with the potential for AGI systems to pursue goals with super-human efficiency that are not aligned with our values or interests.

Given the profound risks associated with AGI, it’s critical that we carefully consider the implications of this technology and take steps to mitigate these risks. In a world that’s already grappling with complex ethical questions around relatively simple technologies like social media, the development of AGI demands our utmost attention, careful consideration, and caution.

If this all sounds like science fiction, please bear with me. By the end of this article, I hope to convince you of three things:

AGI is possible to build
It is possible the first AGI will be built soon
AGI which is possible to build soon is inherently existentially dangerous

AGI is possible to build

As the world of technology and artificial intelligence continues to advance, researchers are finding more and more evidence that building a generally intelligent agent capable of surpassing human intelligence is within our reach. Despite some residual uncertainty, the possibility of developing an agent with capabilities similar to, or even exceeding that of the human bra...
Remote AI Alignment Overhang?
by tryactions — 2023-02-19T22:30:47.530Z

Background link: Do any AI alignment orgs hire remotely?

Disclaimer: The generator of me writing this post is my own desire for remote AI Alignment opportunities to exist. I have Alignment research in mind while writing this, but similar arguments might apply to other Alignment-related jobs?

This is a post about a capabilities overhang, but not in the usual sense.

The Problem (if there is one)

It seems to me that there's been substantial growth in ways to get involved with Alignment. However, there are very few advertised jobs in Alignment that do not involve moving to specific locations, mostly the Bay Area. I hypothesize that people who might otherwise be productive are dropping out of Alignment, or never engaging at all, because they're not willing to move.

Who are these talented people I'm hypothesizing, and why aren't they willing to move? They might:

Have weak or non-existent social support networks that make risk-taking riskier.
Have responsibilities they've taken on (spouse/kids/mortgage/elderly parents) that give them less freedom to move.
Have non-prestigious backgrounds, such that they might have trouble finding work if they find themselves out of a job.
Have little money, such that they have less runway to reverse course if a move doesn't pan out.
Have an established lifestyle that they don't want to change.

Overall, I could summarize this as having less spare resources or being more firmly rooted somewhere that's not the Bay Area. I'm pretty sure this is going to be strongly correlated with being born in a lower economic class -- you'll have less resources to get started in life, and by the time you do have resources (if ever), you're more likely to have rooted in a less expensive location outside of a major city. Being rooted outside a major city also makes it less likely you'll develop prestigious connections.

Although it might suck for those people, this lack of opportunity isn't necessarily an important problem to solve if your goal is improving Alignment. The rest of this article is me unpacking my thoughts about why I think it might be important.

Brainstorming Solutions

Two ideas for improving the situation would be:
1. Create alignment organizations in more cities, or
2. Create/convert alignment organizations to allow remote work.

Creating organizations in new cities has natural limits. In a world with 300 total A...
A Neural Network undergoing Gradient-based Training as a Complex System
by carboniferous_umbraculum  — 2023-02-19T22:08:56.243Z

In Thought Experiments Provide a Third Anchor, Jacob Steinhardt wrote about the relative merits of a few different reference classes when it comes to reasoning and making predictions about future machine learning systems. He refers to these reference classes as ‘anchors’ and writes:

There are many other anchors that could be helpful for predicting future ML systems... I am most excited about better understanding complex systems, which include biological systems, brains, organizations, economies, and ecosystems and thus subsume most of the reference classes discussed so far. It seems to me that complex systems have received little attention relative to their germaneness to ML.

I, too, recently became curious about this complex systems ‘anchor’ and this post is an attempt to get down some of my thoughts about it.

In starting to think about this, there are two questions that I would want to know the answer to:

Is there a well-defined 'field' of complex systems research that has its own techniques and conceptual frameworks, and such that the central objects of study - i.e. complex systems themselves - can be robustly characterized? And
Are deep neural networks complex systems according to that characterization?

Perhaps the best thing that we could hope for would be that the answer to both questions were 'yes', and in such a way that there existed techniques, methods, insights etc. that have been applied to other complex systems and that we might be able to translate into a form that were directly applicable to the study of neural networks. But at the very least, a positive answer to both questions would presumably suggest that broader insights and observations about how to think about complex systems may help us direct out inquiries about future ML systems, by helping to guide us towards good questions, appropriate research directions or better predictions.

The answer to the first question might be 'no'. i.e. It might be the case that informally, the label 'complex system' is applied to too many different kinds of system, each with their own idiosyncrasies, to the extent that a single unified notion is not possible, or would be trivial and useless. To further complicate matters, one might reasonably ask if - during a certain period of recent history in the higher education funding landscape - the phrase 'complex system' became associated with a ...
Another Way to Be Okay
by Gretta Duleba — 2023-02-19T20:49:31.895Z

Cross-posted on lesswrong.com and integrationbyparts.substack.com.

Compare and contrast with Duncan Sabien’s essay

A Way to Be Okay

, written in parallel and with collaboration.

See also:

Mental Health and the Alignment Problem

Photo by

Simon Berger

If you look around, there are plenty of reasons to be gravely worried. Existential risk from artificial intelligence, climate change, political upheaval, pandemics, and all kinds of systemic oppression – take your pick. Humans have been saying since the beginning of written history that the end was nigh, and maybe this time it’s true.

Today’s article is for people who are pretty freaked out. People who think something unprecedentedly awful might befall us in our natural lifetimes.

If you’re not one of those people, fantastic! Carry on, this one’s not for you. The point of this article is not to debate whether there’s doom around the corner, it’s about how to cope if you already believe in that doom. How are you supposed to be okay? How are you supposed to go on being a person, with that great dark doom cloud hanging over your head? Should you panic? Should you give up? What do you do?

I spend considerable time helping people grapple with that question. I can’t say I’ve completely solved it even for myself, much less for everyone else. I am in love with living and would prefer not to stop. I am a mother to three children, and when I look straight at my grief that their lives might be cut short, I can barely breathe for the sharp pain of it.

I have a few leads on how to be okay, despite it all. Here’s what I know.

Agency and Acceptance

If you’ve ever been to an AA meeting or seen one on TV, you’ve heard the

Serenity Prayer

. I’m not religious, myself, so I tend to omit the opening supplication to God, or just go with the Mother Goose version:

For every ailment under the sun

There is a remedy, or there is none;

If there be one, try to find it;

If there be none, never mind it.

Whichever flavor of doom is on your mind, it is unlikely that you personally are in a position to fix it. Even if you do happen to be a leading AI researcher or a vaccine-developing epidemiologist, you still won’t save everyone all by yourself, especially given that we’re up against multiple dooms at once. You’re going to have to figure out what you have control over and what you don’t. For the control bu...