trentmkelly/LessWrong-43k huggingface dataset:
Multiple stages of fallacy - justifications and non-justifications for the multiple stage fallacy
by AronT — 2023-06-13T17:37:20.762Z
Link: https://coordinationishard.substack.com/p/multiple-stages-of-fallacy

[Epistemic status: I am reasonably confident (75%) that these claims are largely correct.]

To estimate the probability of something happening, e.g. AI takeover, one approach is to break this down into stages and multiply the (conditional) probabilities together. I claim this approach yields systematically low answers. This is often referred to as the multiple stage fallacy - however - the term ‘fallacy’ in its label shouldn’t be viewed as an

indisputable verdict

.

Smart people disagree

with whether it truly is a fallacy, and it is the default method for

even very good forecasters

- so what’s going on? I haven’t been able to find a source that does this justice[1], and my impression is that consensus opinion on this overestimates the importance of disjunctive alternatives whilst underestimating the importance of correlated conditional probabilities. On a meta/ Bayesian level, I’ve seen it claimed several times that this is a fallacy because, empirically, it produces systematically low probabilities, but haven’t seen good evidence of this.

Disjunctive alternatives

What I see cited most often is that multiplying conditional probabilities fails to account for disjunctive routes. If x1,x2,x3 are events, and you are trying to estimate P(x3), then P(x3|x2)⋅P(x2|x1)⋅P(x1) doesn’t give the probability P(x3), it gives the probability P(x1∧x2∧x3).

If you want to compute the probability P(x3), you have to sum over all 4 possible combinations of x2 and x3 being true/ false. As the number of events grows, the number of combinations grows exponentially. In theory, this sounds like a problem, but in practice I believe the effect is smaller than claimed. In most situations where this fallacy appears to apply, I think it’s often reasonable to claim that the conclusion does actually imply the premises with reasonably high probability.

To take a concrete example, consider Carlsmith’s

report

on the risk of AI takeover. He lists 5 premises:

Timelines are short enough to build AGI by 2070.
There will be sufficient incentives to build AGI.
Alignment will be hard.
Systems will be deployed in ways that expose them to inputs that lead to them trying to seek power.
This will lead to AI takeover.

He computes the conditional probabilities of each premise, and then finally a conditional probability of an e...
TryContra Events
by jefftk — 2023-06-13T17:30:01.597Z

I've been maintaining a

spreadsheet
of contra dance events

for
a while. With the end of

the Dance Gypsy,
and

ContraDanceLinks
going stale, however, people asked if I could convert this into
something more usable. So! Here's

trycontra.com/events:

Initially it shows you all the events that it knows about. These are
dance weekends, camps, festivals, etc which have a bunch of contra
dancing. You can limit to just one state if you want:

If you click the "map" tab at the top you can see where these events
are:

Clicking on a marker on the map shows you more about it:

If your screen is wide enough it shows both views at the same time:

One thing that makes this tricky is that events are usually annual,
but for a while after an event it's not clear when the next one will
be or even if it will happen. I handle these by guessing that there
will be another one next year, same month, and showing an empty entry
with a question mark:

Note that this doesn't replace the main trycontra.com functionality of
listing local dance series. It also has a pretty different audience,
and so different design goals: the main site is aimed at people who
don't dance (yet!) and is very simple, while this is for people who
already contra dance and would like to go to more events.

Right now this only shows the data that I was already collecting in
the spreadsheet: name, date, url, location, contra bands, contra
callers. Several people have asked for additional data for filtering,
including:

Covid requirements (especially masking)

Event length (day, weekend, week)

Role terms

Family events

Which non-contra dances or performers are included

While I'd be happy to tweak the UI to add them, each bit of
information requires collection and maintenance. If someone else
wanted to volunteer to take on some amount of collection, however, I
could give them edit access to the sheet and they could go do that?

Let me know if you run into any bugs, inaccuracies, or other frustrations!
MetaAI: less is less for alignment.
by Cleo Nardo — 2023-06-13T14:08:45.209Z

Summary

In May 2023, MetaAI submitted a paper to arxiv called LIMA: Less Is More for Alignment. It's a pretty bad paper and (in my opinion) straightforwardly misleading. Let's get into it.

The Superficial Alignment Hypothesis

The authors present an interesting hypothesis about LLMs —

We define the Superficial Alignment Hypothesis: A model’s knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users.

If this hypothesis is correct, and alignment is largely about learning style, then a corollary of the Superficial Alignment Hypothesis is that one could sufficiently tune a pretrained language model with a rather small set of examples.

We hypothesize that alignment can be a simple process where the model learns the style or format for interacting with users, to expose the knowledge and capabilities that were already acquired during pretraining.

(1) This hypothesis would have profound implications for AI x-risk —

It suggests that we could build a safe competent oracle by pretraining an LLM on the entire internet corpus, and then finetuning the LLM on a curated dataset of safe competent responses.
It suggests that we could build an alignment researcher by pretraining an LLM on the entire internet corpus, and then finetuning the LLM on a curated dataset of alignment research.

(2) Moreover, as by Ulisse Mini writes in their review of the LIMA paper,

Along with TinyStories and QLoRA I'm becoming increasingly convinced that data quality is all you need, definitely seems to be the case for finetuning, and may be the case for base-model training as well. Better scaling laws through higher-quality corpus? Also for who haven't updated, it seems very likely that GPT-4 equivalents will be essentially free to self-host and tune within a year. Plan for this!

(3) Finally, the hypothesis would've supported many of the intuitions in the Simulators sequence by Janus, and I share these intuitions.

So I was pretty excited to read the paper! Unfortunately, the LIMA results were unimpressive upon inspection.

MetaAI's experiment

The authors finetune MetaAI's 65B parameter LLaMa language model on 1000 curated prompts and responses (mostly from StackExchange, wikiHow, and Reddit), and then compare it to five other LLMs (Alpaca 65B, DaVinci003, Bard, Claude, GPT4).

Method:

To compare LIMA

...
The Dial of Progress
by Zvi — 2023-06-13T13:40:06.354Z

“There is a single light of science. To brighten it anywhere is to brighten it everywhere.” – Isaac Asimov

You cannot stand what I’ve become
You much prefer the gentleman I was before
I was so easy to defeat, I was so easy to control
I didn’t even know there was a war

– Leonard Cohen, There is a War

“Pick a side, we’re at war.”

– Steven Colbert, The Colbert Report

Recently, both Tyler Cowen in response to the letter establishing consensus on the presence of AI extinction risk, and Marc Andreessen on the topic of the wide range of AI dangers and upsides, have come out with posts whose arguments seem bizarrely poor.

These are both excellent, highly intelligent thinkers. Both clearly want good things to happen to humanity and the world. I am confident they both mean well. And yet.

So what is happening?

A Theory

My theory is they and similar others believe discourse in 2023 cannot handle nuance.

Instead, effectively there is a single Dial of Destiny Progress, based on the extent our civilization places restrictions, requires permissions and places strangleholds on human activity, from AI to energy to housing and beyond.

If we turn this dial down, and slow or end such progress across the board, our civilization will perish. If you turn the dial up, we will prosper.

In this view, any talk of either extinction risks or other AI downsides is functionally an argument for turning the dial down, saying Boo Progress, when we instead desperately need it turn the dial up, and say Yay Progress.

It would, again in this view, be first best to say Yay Progress in most places, while making a careful narrow exception that lets us guard against extinction risks. Progress is no use if you are dead.

Alas, this is too nuanced, and thus impossible. Trying will not result in the narrow thing that would protect us. Instead, trying turns the dial down which does harm everywhere, and also does harm within AI because the new rules will favor insiders and target mundane utility without guarding against danger, and the harms you do elsewhere inhibit sane behavior.

Thus, the correct thing to do is shout Yay Progress from the rooftops, by whatever means are effective. One must think in terms of the effects of the rhetoric on the dial and the vibe, not on whether the individual points track underlying physical reality. Caring about individual points and how they relate to physical reality, in t...
TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI
by Andrew_Critch — 2023-06-13T05:04:46.756Z

Partly in response to calls for more detailed accounts of how AI could go wrong, e.g., from Ng and Bengio's recent exchange on Twitter, here's a new paper with Stuart Russell:

Discussion on Twitter... comments welcome!
https://twitter.com/AndrewCritchCA/status/1668476943208169473
arXiv draft:
"TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI"

Many of the ideas will not be new to LessWrong or the Alignment Forum, but holistically I hope the paper will make a good case to the world for using logically exhaustive arguments to identify risks (which, outside LessWrong, is often not assumed to be a valuable approach to thinking about risk).

I think the most important figure from the paper is this one:

... and, here are some highlights:

Self-fulfilling pessimism:
https://arxiv.org/pdf/2306.06924.pdf#page=4
Industries that could eventually get out of control in a closed loop:
https://arxiv.org/pdf/2306.06924.pdf#page=5
...as in this "production web" story:
https://arxiv.org/pdf/2306.06924.pdf#page=6
Two "bigger than expected" AI impact stories:
https://arxiv.org/pdf/2306.06924.pdf#page=8
Email helpers and corrupt mediators, which kinda go together:
https://arxiv.org/pdf/2306.06924.pdf#page=10
https://arxiv.org/pdf/2306.06924.pdf#page=11
Harmful A/B testing:
https://arxiv.org/pdf/2306.06924.pdf#page=12
Concerns about weaponization by criminals and states:
https://arxiv.org/pdf/2306.06924.pdf#page=13

Enjoy :)
<$750k grants for General Purpose AI Assurance/Safety Research
by Phosphorous — 2023-06-13T04:45:43.535Z
Link: https://cset.georgetown.edu/foundational-research-grants/

Georgetown University's Center for Security and Emerging Technology (CSET) is accepting applications for AI Safety / AI Assurance research grants.

They are offering up to $750k per project accepted, expended over 6-24 months.

1-2 page expression of interest due August 1.

Applicants should be based at an academic institution or nonprofit research organization.

More information here.

From CSET "We’re using 'assurance' here in a broad sense, meaning roughly 'the generation of evidence that an ML system is sufficiently safe for its intended use.'"
UFO Betting: Put Up or Shut Up
by RatsWrongAboutUAP — 2023-06-13T04:05:32.652Z

Are you confident in your current ontology? Are you convinced that ultimately all ufos are prosaic in nature?

If so, do you want some immediate free money?

I suspect that LW's are overconfident in their views on ufos/uap. As such, I'm willing to offer what I think many will find to be very appealing terms for a bet.

The Bet

Essentially, I wish to bet on the world and rationalists eventually experiencing significant ontological shock as it relates to the nature of some ufos/uap.

Offer me odds for a bet, and the maximum payout you are willing to commit to. I will pick 1+ from the pool and immediately pay out to you. In the event that I ultimately win the bet, then you will pay out back to me.

I'm looking to give out between $5k-10k, but depends on what kinds of offers I get, could be more or less.

The Terms

I Send you $X Immediately, You pay out Odds*X if I win

ie, You offer 200:1 odds with max payout $20,000 and I will send you $100 immediately.

5 year time horizon starting from the date we confirm our bet.
You offer the odds and maximum payout, I will pick from the available offers to maximize my expected returns, subject to my financial constraints.

Resolution Criteria

Two Worlds: All-ufos-are-ultimately-prosaic, and Not-all-ufos-are-ultimately-prosaic. I win the bet if we come to believe we likely live in the latter world. I win the bet if the ufo story ultimately gives us LW's a significant ontological shock. I win the bet if the ufo story ultimately causes the LW community to stop, melt, and catch fire. I've found it difficult to precisely nail down how to phrase this, so I hope its clear what kind of criteria I'm trying to get at.

Examples of things where if we come to believe at least one of them likely explain >0 ufo/uap cases, then I win the bet:

Aliens / Extraterrestrials

Biological
Machines (Von Neumann probes, for instance)

Actual magic/spiritual/paranormal/psychic phenomenon

This explicitly does NOT include merely advanced "mentalist" type things / show magic
ie, things like ESP, astral projection, demons, god(s), angels, ghosts, remote viewing, fairy's (actually anomalous, not just new kind of bird), etc.
Basically, the kinds of things that standard atheist materialists would reject as not being real.

Time travel

ie, future human activities (or otherwise)

Leftovers of an ancient civilization
Some other unknown non-human advanced civilization on earth
Matri

...
A bunch of videos in comments
by the gears to ascension — 2023-06-12T22:31:38.285Z

Hi. Welcome to my personal blog on lesswrong. I like youtube videos. I've posted lists of them before. Since then, more youtube videos have been made that I want to share, and tools for summarizing videos have come out! So I've wanted to share them here. To allow this to be evergreen where videos can be downvoted, I'll put the videos in comments with summaries. I'll use unvoting and downvoting my own comments to change the order, and others are free to downvote videos they don't think are particularly insightful as well.

If they really want to moderators are allowed to put this on the official frontpage tag but I think it'd be a little silly, as I'm not going to think too hard about exactly which videos to tell a particular story, and just share the ones I think would be cool to have shared. some of them might have bad epistemics, even!

video stereotype bullet lists ("summaries") by kagi unless otherwise noted

posting is not endorsement, just suggestion of possible value
[Linkpost] The neuroconnectionist research programme
by Bogdan Ionut Cirstea — 2023-06-12T21:58:57.722Z

This is a linkpost for https://www.nature.com/articles/s41583-023-00705-w (open access preprint: https://arxiv.org/abs/2209.03718)

Artificial neural networks (ANNs) inspired by biology are beginning to be widely used to model behavioural and neural data, an approach we call ‘neuroconnectionism’. ANNs have been not only lauded as the current best models of information processing in the brain but also criticized for failing to account for basic cognitive functions. In this Perspective article, we propose that arguing about the successes and failures of a restricted set of current ANNs is the wrong approach to assess the promise of neuroconnectionism for brain science. Instead, we take inspiration from the philosophy of science, and in particular from Lakatos, who showed that the core of a scientific research programme is often not directly falsifiable but should be assessed by its capacity to generate novel insights. Following this view, we present neuroconnectionism as a general research programme centred around ANNs as a computational language for expressing falsifiable theories about brain computation. We describe the core of the programme, the underlying computational framework and its tools for testing specific neuroscientific hypotheses and deriving novel understanding. Taking a longitudinal view, we review past and present neuroconnectionist projects and their responses to challenges and argue that the research programme is highly progressive, generating new and otherwise unreachable insights into the workings of the brain.

Personally, I'd be excited to see more people thinking about the intersection of neuroconnectionism and alignment (currently, this seems very neglected).

Some examples of potential areas of investigation (alignment-relevant human capacities) which also seem very neglected could include: instruction following, moral reasoning, moral emotions (e.g. compassion, empathy).
Contingency: A Conceptual Tool from Evolutionary Biology for Alignment
by clem_acs — 2023-06-12T20:54:04.315Z
Link: https://acsresearch.org/posts/contingency

-1. Motivation

As AI systems become increasingly powerful, the chance of them developing dangerous features becomes increasingly likely. A key concern of AI alignment researchers is that there are dangerous features, such as power-seeking or situational awareness, which are convergent.[1] This would mean that these capabilities would likely arise regardless of the particular trajectory AI development took. In a recent tweet, Eliezer Yudkowsky argues that this is the key question for AI alignment research:

> From my perspective, we are trying to evaluate a picture of convergent ruin rather than any pictures of contingent ruin. That is, I'm relatively less interested in contingent-doom scenarios like "What if superintelligence is easily alignable, but Greenpeace is first to obtain superintelligence and they align It entirely to protect Earth's environment and then the SI wipes out humanity to do that?" because this is not a case for almost-all paths leading to ruin. I worry that contingent scenarios like "Age of Ultron" can fascinate humans (particularly contingent scenarios where some Other Monkey Faction Gets AGI First oh noes), and derail all possibility of a discussion about whether, in fact, the superintelligence kills everyone regardless of who 'gets' It first.

> Similarly, if somebody is considering only the contingent-early-ruin scenario of "a prototype of a somewhat advanced AGI starts to consider how to wipe out humanity, and nobody spots any signs of that thought", that contingency can mask a convergent-ruin continuation of "even if they spot the early signs, everyone is still in an arms-race scenario, so they optimize away the visible signs of doom and keep going into ruin".

> So I think the key question is "Why would or wouldn't humanity be convergently doomed?" rather than "How might humanity be contingently doomed?" Convergent doom isn't the same as inevitable doom, but ought to be the sort of doom that defeats easy solution / historically characteristic levels of competence for Earth / the sort of efforts we've seen already.

> A doom that can only be averted by an effective multinational coalition shutting down all AI runs larger than GPT-4 for 6 years, to make time for a crash project on human intelligence augmentation that has to complete within 6 years, may not be an inevitable doom; but it was probably a

...
Book Review: Autoheterosexuality
by tailcalled — 2023-06-12T20:11:38.215Z

This review was originally written for the

Astral Codex Ten Book Review Contest

. Unfortunately it didn’t make it as one of the finalists. but since I made use of the LessWrong proofreading/feedback service, I am reposting it here. It can also be found on my gender blog.

If I ask ChatGPT to explain transgender people to me, then it often retreats into vague discussions of gender identity. It is very hard to get it to explain what these things mean, in terms of actual experiences people might have. And that might not be a coincidence - the concepts used to understand transness seem to be the result of a complicated political negotiation, at least as much as they are optimized to communicate people’s experiences.

Some people claim to do better, using elaborate words like autogynephilia and gynandromorphophilia and so on. They speak of typologies, science and history. Of developmental pathways and genetics. This ideology is sometimes called Blanchardianism, named after Ray Blanchard, a sex researcher who contributed many influential ideas within the sphere.

Phil Illy’s book,

Autoheterosexual: Attracted to Being the Other Sex

is the latest book pushing Blanchardianism. But is it any good?

Summary of the book on autogynephilia

Both

Phil

and I are probably most familiar with the concept of autogynephilia, so let’s start there. A substantial part of the book is about the feelings and experiences of autogynephiles.

Some individuals, born male, feel a deep happiness when imagining themselves as a woman and a deep sadness being a man, and this is often due to autogynephilia. Phil Illy explains that autogynephilia can be seen as an inverted form of sexual attraction to women, where the individual's attraction to women is applied to themselves. Autogynephilia is ultimately about sexuality, but Phil emphasizes that many of the feelings and experiences he describes are not driven by lust, but instead by strong emotional attachments. In some cases it can cause them to have medical treatment and make social changes to live as women.

Phil suggests that this accounts for most trans women in the US. Some people with strong autogynephilia might feel as if they have female body parts, like breasts or a vagina, even if they haven't undergone any medical procedures to obtain these features. The book also looks at how autogynephiles might act more like women in the way...