trentmkelly/LessWrong-43k huggingface dataset:
Panel discussion on AI consciousness with Rob Long and Jeff Sebo
by Aaron Bergman — 2023-09-09T03:38:17.662Z
Link: https://www.youtube.com/watch?v=-YC-ZhdcVck

Intro

Recent 80k guest and philosopher specializing in AI consciousness Rob Long (@rgb) recently participated in a panel discussion on his paper "Consciousness in Artificial Intelligence: Insights from the Science of Consciousness" (pdf) with co-authors Patrick Butlin, Yoshua Bengio, and Grace Lindsay and moderator Jeff Sebo (@jeffsebo).

You can watch it on Youtube (below), watch/listen as a podcast on Spotify [1], or read the transcript[2] below.

Paper abstract

Whether current or near-term AI systems could be conscious is a topic of scientific interest and increasing public concern. This report argues for, and exemplifies, a rigorous and empirically grounded approach to AI consciousness: assessing existing AI systems in detail, in light of our best-supported neuroscientific theories of consciousness. We survey several prominent scientific theories of consciousness, including recurrent processing theory, global workspace theory, higher-order theories, predictive processing, and attention schema theory. From these theories we derive "indicator properties" of consciousness, elucidated in computational terms that allow us to assess AI systems for these properties. We use these indicator properties to assess several recent AI systems, and we discuss how future systems might implement them. Our analysis suggests that no current AI systems are conscious, but also suggests that there are no obvious technical barriers to building AI systems which satisfy these indicators.

Youtube description

This event took place on Tuesday September 5, 2023 and was hosted by the NYU Mind, Ethics, and Policy Program.

About the event

This panel discussion featured four authors from the recently released and widely discussed AI consciousness report. This report argues for, and exemplifies, a rigorous and empirically grounded approach to AI consciousness: assessing existing AI systems in detail, in light of the best-supported neuroscientific theories of consciousness. The paper surveys several prominent scientific theories of consciousness, including recurrent processing theory, global workspace theory, higher-order theories, predictive processing, and attention schema theory. From these theories the authors derive "indicator properties" of consciousness, elucidated in computational terms that allow them to assess AI systems for these properties. They use these indicator ...
Possible Divergence in AGI Risk Tolerance between Selfish and Altruistic agents
by Brad West  — 2023-09-09T00:23:20.464Z

TLDR: There is a conflict of interest between the interests of the currently living and the entirety of future generations with regarding AGI because the prospective benefits AGI could provide in the upsides (such as extreme longevity and happiness) may be worth the risk of death at much higher likelihoods than would be the case if you were integrating the interests of all future generations (slight acceleration of AGI possible benefits almost never being worth risks of extinction).

It strikes me that there may be a conflict of interest between the interests of a living set of generations of humans and the entirety of future of humans regarding the cost-benefit analysis associated with ushering in AGI. For the purposes of this post, I will refer to the set of values reflecting the self-interest of an individual agent as “selfish” and the set of values reflecting the interests of all future generations in addition to the current as “altruistic.”

Let us consider a world in which the creation of AGI entails X-risks (risks of extinction) that decrease over time with development of alignment capabilities. In this world, in the event that an X-risk does not materialize, AGI would also be likely to able to cure aging and allow for lives of joy beyond imagination for the current generation and all future generations.

So, consider a situation in which one could activate AGI today with an X-risk of 50% or it would be released in 70 years with an X-risk of 1%. To an agent who is 55 years old, the coin-flip may make sense, from a selfish perspective: there is a 50% chance of living millennia in bliss and a 50% chance of dying vs. a very high chance of living 15-50 years longer. From a strictly selfish perspective, this gamble might make sense. Conversely, from an altruistic perspective, such a coin flip would be abhorrent: you are risking quadrillions upon quadrillions of blissful future lives for the benefit of yourself and the relatively small cohort who could benefit from the acceleration.

Of course, the addition of S-Risks (risks of creating worlds of immense suffering) could change the selfish equation as a selfish agent may view significant risks of subjecting him or herself to hell to be unacceptable. But agents may believe themselves capable of identifying an S-Risk emergence and capable of committing suicide (translating a personal S-R...
Capture the Flag Mechanistic Interpretability Challenges
by Alejandro Acelas — 2023-09-08T23:00:50.011Z

This project was developed as part of the Swiss Existential Risk Initiative Research Fellowship under the mentorship of Alexandre Variengien. The use of the first person throughout the document refers to Alejandro Acelas.

I designed three challenges meant as a training ground for improving your skills in analyzing transformer’s internals. Each challenge centers on a transformer model trained on an algorithmic task and, depending on the challenge, solutions are either a labeling function or a file with model weights. You can submit your solution to be automatically scored on how accurately it resembles a held-out test set at the corresponding CodaBench competition:

CodaBench Competition

Starting Kit Repository

In brief, the challenges are:

Repairing an Ablated Circuit:

Manually fill in the weights of an attention head to recover the performance of a model at its intended task

OOD Generalization:

Decipher the behavior of a model on 24-tokens-long prompts evaluating the model only on 9-tokens-long prompts

Discovering Hidden Capabilities (Backdoors):

Discover 5 different classes of inputs that cause a model to switch to a different prediction function.

These challenges are an experiment with a slightly different way of eliciting and training skills for mechanistic interpretability. Although there are several good alternatives for upskilling in the field, many of them lack a concretely defined objective that tells you whether you’ve succeeded at discovering something about a neural network’s internals.

Capture the Flag (CTF)

challenges are routinely used in the field of cybersecurity to introduce people to the field, provide opportunities for upskilling, and inform hiring decisions. I speculate that part of their success comes from two factors. First, although it’s hard to explicitly describe the skills needed for working in computer security, it can be easier to set challenges that require the use of such skills to be solved. Second, CTF challenges have a crisp objective (recovering an instantly recognizable string of text called ‘flag’), which allows both participants and external observers to more easily evaluate whether participants developed the necessary set of skills.

The following challenges attempt to serve a similar function for the field of mechanistic interpretability by providing a clear objective where understanding th...
What is to be done? (About the profit motive)
by Connor Barber — 2023-09-08T19:27:40.228Z

I've recently started reading posts and comments here on LessWrong and I've found it a great place to find accessible, productive, and often nuanced discussions of AI risks and their mitigation. One thing that's been on my mind is that seemingly everyone takes for granted that the world as it exists will eventually produce AI, particularly sooner than we have the necessary knowledge and tools to make sure it is friendly. Many seem to be convinced of the inevitability of this outcome, that we can do little to nothing to alter the course. Often referenced contributors to this likelihood are current incentive structures; profit, power, and the nature of current economic competition.

I'm therefore curious why I see so little discussion on the possibility of changing these current incentive structures. Mitigating the profit motive in favor of incentive structures more aligned with human well-being seems to me an obvious first step. In other words, to maximize the chance for aligned AI, we must first make an aligned society. Do people not discuss this idea here because it is viewed as impossible? Undesirable? Ineffective? I'd love to hear what you think.
What is the optimal frontier for due diligence?
by RobertM — 2023-09-08T18:20:03.300Z

The title isn't quite right. My current take is that:

There is a ~10k word post sharing some damaging information about someone else.
3-4k words are describing accusations that are disputed.

However, a few have already had substantial evidence provided against them.
The post author doesn't think the accuracy of those accusations is particularly cruxy for their major takeaways.

Those accusations generally sound pretty bad.
The post doesn't explain why the (disputed) accusations are salient, if they aren't cruxy for the post author's major takeaways.
The post author may not find those accusations cruxy, but I would be surprised if nobody else found them cruxy for deciding between "there's some bad stuff in here, but it seems possible that most of the harm was caused by some combination of large cultural differences and poor communication", and "nope nope nope".

This is to say, our actions have effects beyond the EA ecosystem. I think it's wrong to decide that, because other people may update incorrectly based on evidence you've provided, the harm that comes from those updates doesn't matter much.

The thing I'm noticing is some kind of missing mood. I think that there's been a failure to inhabit the least convenient possible world, and the general distribution over possible outcomes, and correspondingly attempt to move to the pareto-frontier of outcomes assuming that distribution. This comment in particular seems to be operating in the frame of "You weren't behaving the way I'd update positively on if the accusations were true". In the world where the disputed accusations are in fact false, for the specific reasons they provided, I think it would be pretty strange to take the suggested course of action.
Progress links digest, 2023-09-08: The Conservative Futurist, cargo airships, and more
by jasoncrawford — 2023-09-08T17:48:45.584Z
Link: https://rootsofprogress.org/links-digest-2023-09-08

Announcements

The Conservative Futurist: How to Create the Sci-Fi World We Were Promised is available for pre-order (via @JimPethokoukis)
Asimov Press is “an editorially-independent venture that will publish books and essays that make sense of biology, AI, and our collective future” (via @AsimovBio and @NikoMcCarty, who will be its first editor)
Statecraft, a new newsletter from the Institute for Progress, is an interview series about how policymakers actually get things done (via @rSanti97)
HumanProgress.org has been revamped, including a new interactive data page that lets you create your own visualizations (via @Marian_L_Tupy)
Maximum New York raised $17,500 of its $150,000 goal in the first week of its fundraiser. Founder Daniel Golliher is also starting chemotherapy for Hodgkin’s lymphoma—he says the prognosis is good, but send your best wishes (via @danielgolliher)
The US Megaprojects Database, contributions solicited (via @_brianpotter)

New issue of Works in Progress

Issue 12 features:

Why we didn’t get a malaria vaccine sooner
Making architecture easy. “Buildings should be designed to be agreeable—easy to like—not to be unpopular works of genius”
Growing the growth coalition. “If we want voters to encourage growth near them, we need to make it worth their while”
and more!

Articles

Anton Howes asks: does history have a replication crisis? (via @s8mb)

Video

Veritasium on cargo airships (via @elidourado, who else)

Queries

“I’m looking to talk (on the record) to expert on dealing with the result of car crashes—like an emergency medicine doctor, EMT, etc. Any suggestions?” (@binarybits)
“I want to go to a few talks at Berkeley or Stanford for fun—physics, biology, math + CS but also humanities. Any suggestions for ones w/ high quality content or attendees?” (@LauraDeming)

Quotes

“This then is our task, to gather the highest discoveries that have been made in the sciences, to render them clear and fascinating, and to offer them to childhood.” Montessori (via @mbateman)
“If the world farmer reaches the average yield of today’s U.S. corn grower during the next 70 years, 10 billion people eating as people now on average do will need only half of today’s cropland. The land spared exceeds Amazonia.” Jesse H. Ausubel (via @Marian_L_Tupy)

History

Life before modern communication technology: “Despite being born in the same year and only about 130 kms apart, Bach and Handel never met. In

...
Sum-threshold attacks
by TsviBT — 2023-09-08T17:13:37.044Z
Link: https://tsvibt.blogspot.com/2023/09/sum-threshold-attacks.html

How do you affect something far away, a lot, without anyone noticing?

(Note: you can safely skip sections. It is also safe to skip the essay entirely, or to read the whole thing backwards if you like.)

The frog's lawsuit

Attorney for the defendant: "So, Mr. Frog. You allege that my client caused you grievous bodily harm. How is it that you claim he harmed you?"

Frog: "Ribbit RIBbit ribbit."

Attorney: "Sir..."

Frog: "Just kidding. Well, I've been living in a pan for the past two years. When I started, I was the picture of health, and at first everything was fine. But over the course of the last six months, something changed. By last month, I was in the frog hospital with life-threatening third-degree burns."

Attorney: "And could you repeat what you told the jury about the role my client is alleged to have played in your emerging medical problems?"

Frog: "Like I said, I don't know exactly. But I know that when my owner wasn't away on business, every day he'd do something with the stove my pan was sitting on. And then my home would seem to be a bit hotter, always a bit hotter."

Attorney: "Your owner? You mean to say..."

Judge: "Let the record show that Mr. Frog is extending his tongue, indicating the defendant, Mr. Di'Alturner."

Attorney: "Let me ask you this, Mr. Frog. Is it right to say that my client——your owner——lives in an area with reasonably varied weather? It's not uncommon for the temperature to vary by ten degrees over the course of the day?"

Frog: "True."

Attorney: "And does my client leave windows open in his house?"

Frog: "He does."

Attorney: "So I wonder, how is it that you can tell that a slight raise in temperature that you experience——small, by your own admission——how can you be sure that it's due to my client operating his stove, and not due to normal fluctuations in the ambient air temperature?"

Frog: "I can tell because of the correlation. I tend to feel a slight warming after he's twiddled the dial."

Attorney: "Let me rephrase my question. Is there any single instance you can point to, where you can be sure——beyond a reasonable doubt——that the warming was due to my client's actions?"

Frog: "Ah, um, it's not that I'm sure that any one increase in temperature is because he turned the dial, but..."

Attorney: "Thank you. And would it be fair to say that you have no professional training in discerning temperature and changes thereof?"

Frog: "That would be accurat...
Debate series: should we push for a pause on the development of AI?
by Xodarap — 2023-09-08T16:29:51.367Z

In March of this year, 30,000 people, including leading AI figures like Yoshua Bengio and Stuart Russell, signed

a letter

calling on AI labs to pause the training of AI systems. While it seems unlikely that this letter will succeed in pausing the development of AI, it did draw substantial attention to slowing AI as a strategy for reducing existential risk.

While initial work has been done on this topic (

this sequence

links to some relevant work), many areas of uncertainty remain. I’ve asked a group of participants to discuss and debate various aspects of the value of advocating for a pause on the development of AI on the EA Forum, in a format loosely inspired by

Cato Unbound

.

On September 16, we will launch with three posts:

David Manheim will share a post giving an overview of what a pause would include, how a pause would work, and some possible concrete steps forward
Nora Belrose will post outlining some of the risks of a pause
Thomas Larsen will post a concrete policy proposal

After this, we will release one post per day, each from a different author
Many of the participants will also be commenting on each other’s work

Responses from Forum users are encouraged; you can share your own posts on this topic or comment on the posts from participants. You’ll be able to find the posts by looking at

this tag

(remember that you can subscribe to tags to be notified of new posts).

I think it is unlikely that this debate will result in a consensus agreement, but I hope that it will clarify the space of policy options, why those options may be beneficial or harmful, and what future work is needed.

People who have agreed to participate

These are in random order, and they’re participating as individuals, not representing any institution:

David Manheim

(ALTER)

Matthew Barnett

(Epoch AI)

Zach Stein-Perlman

(AI Impacts)

Holly Elmore

(AI pause advocate)

Buck Shlegeris

(Redwood Research)
Anonymous researcher (Major AI lab)
Anonymous professor (Anonymous University)

Rob Bensinger

(Machine Intelligence Research Institute)

Nora Belrose

(EleutherAI)

Thomas Larsen

(Center for AI Policy)

Quintin Pope

(Oregon State University)

Scott Alexander will be writing a summary/conclusion of the debate at the end.

Thanks to Lizka Vaintrob, JP Addison, and Jessica McCurdy for help organizing this, and Lizka (+ Midjourney) for the picture.
AI Probability Trees - Joe Carlsmith (2022)
by Nathan Young — 2023-09-08T15:40:24.892Z

I am reviewing the work AI experts on what they think will happen with AI. This is a summary of Joe Carlsmith’s thoughts from his paper. AI risk scares me but often I feel pretty disconnected from it. This has helped me think about it.

Here are Carlsmith’s thoughts in brief (he no longer fully endorses these):

Will it become possible to build Carlsmith’s version of AGI by 2070 (65%)

Carlsmith’s standard is outperforms the best humans at advanced tasks, can plan and has accurate models of the world

There will be incentives to build such systems (80%)
It will be much harder to build aligned systems than ones that just seem aligned (40%)
Some misaligned systems will cause $1T of damage (65%)
Such misaligned power seeking will permanently disempower humanity (40%)
This will constitute existential catastrophe (95%)

Here is an interactive version of his probability tree:

You can see all graphs I’ve done here:

https://estimaker.app/ai

.

You can watch a video he did here (unrelated to me, I haven't talked to joe about this):

https://www.youtube.com/watch?v=UbruBnv3pZU

Or 80k have done a similar write up here:

https://80000hours.org/problem-profiles/artificial-intelligence/#how-likely-is-an-AI-related-catastrophe

Longer explanations

What kind of AI is Carlsmith forecasting? What is APS?

Advanced capability: they outperform the best humans on some set of tasks which when performed at advanced levels grant significant power in today’s world (tasks like scientific research, business/military/political strategy, engineering, and persuasion/manipulation).

Agentic planning: they make and execute plans, in pursuit of objectives, on the basis of models of the world.

Strategic awareness: the models they use in making plans represent with reasonable accuracy the causal upshot of gaining and maintaining power over humans and the real-world environment.

Statement (Confidence)

Joe's commentary from his paper (abridged with ...)
My commentary

By 2070, it will become possible and financially feasible to build APS (Advanced, Planning, Strategically-aware) systems (65%)

Joe’s commentary: This comes centrally from my own subjective forecast of the trajectory of AI progress (not discussed in this report), which draws on various recent investigations at Open Philanthropy, along with expert (and personal) opinion.
My commentary: It’s

...
Invading Australia (Endless Formerlies Most Beautiful, or What I Learned On My Holiday)
by Oliver Sourbut — 2023-09-08T15:33:27.748Z
Link: https://www.oliversourbut.net/p/invading-australia

G'day! I recently got back from a really lovely extended break down under. There was a lot ('heaps') to see and do! - including visiting family and friends with my partner in crime, sampling some unspecified quantity of homebrewed rum, and taking afternoon tea with the locals (humans and wildlife).

A friendly local (crimson rosella) I met in the Bunya mountains

I learned a lot from hosts, guides, museums, and simply from keeping eyes and ears open while indulging my passion for wildlife and nature while walking in the bush. But the main topic I'll touch on here is the pervasive sense of barely-controlled or out-of-control ecosystem impacts, which is felt pretty viscerally there.

Oly, don't you mainly think and write about AI and computer science? This one is mostly (superficially) about nature, but don't worry, we'll get there.

In Un-unpluggability I actually had some Australian cases in mind when I wrote:

Replication and growth (with reinvestment) get special mentions as they naturally produce exponential expansion (until constraints are reached), which in practice often manifests as first imperceptible and then rapid escalation.

Replicating systems also give rise to a kind of robustness due both to redundancy and repair. They are notoriously difficult to shut down, which is why autonomous replication is rarely a deliberate part of human designs - though we see it employed under well-understood and controlled conditions in agriculture and some industry, maliciously in computer viruses and bioweapons, and sometimes accidentally in biosphere interventions. In fact, in biological, zoological, and related sciences, great care is usually taken to avoid inadvertently unleashing autonomously replicating systems, though this remains sometimes insufficient[1].

Examples:

pandemics

invasive species (e.g. plant weeds)

computer viruses

wildfire

rumours and ideologies?

But why is great care usually taken to avoid unleashing autonomously replicating systems? Well, it's been a hard-won lesson, known intimately within some disciplines and circles, while less perceptible or understood in others. And we're still making a lot of mistakes!

Australia, the big experiment

Australia, separated from the rest of the world's continents for tens of millions of years, was in many ways a natural experiment here: humans and our water-(and later air-)transport were one of the only ways in or...