trentmkelly/LessWrong-43k huggingface dataset:
How much to update on recent AI governance moves?
by habryka — 2023-11-16T23:46:01.601Z

Nate and I discuss the recent increase in public and political attention on AGI ruin. We try and figure out where, if anywhere, we feel extra hope as a result.

I think it's plausible that this is enough of a shake-up that the world starts relating to AGI in a pretty different way that might help a lot with AI risk going better. Nate is sceptical that a shake-up causes something that is this far from working to start working.

Relatedly, I also think that the extra time might be enough for a bunch of other game-board flipping plans (like cognitively enhancing humans or human uploads) to make significant progress. While Nate agrees that's relatively hopeful, he's pessimistic that humanity will take much advantage of longer timelines. For example, he's pessimistic that this will cause that many additional people to work on these things.

We eventually decided to keep parts of this conversation private. In one or two places that had cuts, the conversational flow is less smooth.

Oliver's & Nate's reactions to public attention on AGI

habryka

I guess my current feeling here is something like: "I do feel like the world is grappling with AGI a bunch more (though not necessarily better) than I was expecting". Like, I didn't expect AGI risk to be kind of a major topic in a presidential debate, and now think it's somewhat likely to happen.

I have really huge amounts of uncertainty on how likely it is for that attention to convert into good things.

I personally feel a pretty jaded or some kind of cynical feeling that's making me pay a lot of attention to "OK, but is anyone actually committing to anything real here?" and "is there a chance for this whole thing to reveal itself as smoke and mirrors?". Like, I remember a lot of excitement around the time of Elon Musk starting to pay a lot of attention to AGI Risk stuff, and I think that foray into things turned out to be a bad surprise instead of a good one. And I feel similar about the whole FTX situation.

When I look at the most recent government AI moves, I definitely notice similar hesitations. Like, a lot of measurement institutions are being set up, a lot of things are getting labeled with "AI Safety" or "AI Risk", but nothing actually concrete has happened. I don't expect the labs to scale slower or really change much about what they are doing in the next year. Though I do think there are ways for the things proposed to tra

...
New LessWrong feature: Dialogue Matching
by Bird Concept — 2023-11-16T21:27:16.763Z

The LessWrong team is shipping a new experimental feature today: dialogue matching!

I've been leading work on this (together with Ben Pace, kave, Ricki Heicklen, habryka and RobertM), so wanted to take some time to introduce what we built and share some thoughts on why I wanted to build it.

New feature! 🎉

There's now a dialogue matchmaking page at lesswrong.com/dialogueMatching

Here's how it works:

You can check a user you'd potentially be interested in having a dialogue with, if they were too
They can't see your checks unless you match

It also shows you some interesting data: your top upvoted users over the last 18 months, how much you agreed/disagreed with them, what topics they most frequently commented on, and what posts of theirs you most recently read.

Next, if you find a match, this happens:

You get a tiny form asking for topic ideas and format preferences, and then we create a dialogue that summarises your responses and suggests next steps based on them.

Currently, we're mostly sourcing auto-suggested topics from Ben's neat poll where people voted on interesting disagreement they'd want to see debated, and also stated their own views. I'm pretty excited to further explore this and other ways for auto-suggesting good topics. My hypothesis is that we're in a bit of a dialogue overhang: there are important conversations out there to be had, but that aren't happening. We just need to find them.

This feature is an experiment in making it easier to do many of the hard steps in having a dialogue: finding a partner, finding a topic, and coordinating on format.

To try the Dialogue Matching feature, feel free to head on over to lesswrong.com/dialogueMatching !

Me and the team are super keen to hear any and all feedback. Feel free to share in comments below or using the intercom button in the bottom right corner :)

Why build this?

A retreat organiser I worked with long ago told me: "the most valuable part of an event usually aren't the big talks, but the small group or 1-1 conversations you end up having in the hallways between talks."

I think this points at something important. When Lightcone runs events, we usually optimize the small group experience pretty hard. In fact, when building and renovating our campus Lighthaven, we designed it to have lots of little nooks and spaces in order to facilitate exactly this kind of interaction. ...
Towards Evaluating AI Systems for Moral Status Using Self-Reports
by Ethan Perez — 2023-11-16T20:18:51.730Z
Link: https://arxiv.org/abs/2311.08576

TLDR: In a new paper, we explore whether we could train future LLMs to accurately answer questions about themselves. If this works, LLM self-reports may help us test them for morally relevant states like consciousness.

We think it's possible to start preliminary experiments testing for moral status in language models now, so if you're interested in working with us, please reach out and/or apply to the Astra fellowship or SERI MATS (deadline November 17).

Tweet thread paper summary: Link

Abstract:

As AI systems become more advanced and widely deployed, there will likely be increasing debate over whether AI systems could have conscious experiences, desires, or other states of potential moral significance. It is important to inform these discussions with empirical evidence to the extent possible. We argue that under the right circumstances, self-reports, or an AI system’s statements about its own internal states, could provide an avenue for investigating whether AI systems have states of moral significance. Self-reports are the main way such states are assessed in humans ("Are you in pain?"), but self-reports from current systems like large language models are spurious for many reasons (e.g. often just reflecting what humans would say). To make self-reports more appropriate for this purpose, we propose to train models to answer many kinds of questions about themselves with known answers, while avoiding or limiting training incentives that bias self-reports. The hope of this approach is that models will develop introspection-like capabilities, and that these capabilities will generalize to questions about states of moral significance. We then propose methods for assessing the extent to which these techniques have succeeded: evaluating self-report consistency across contexts and between similar models, measuring the confidence and resilience of models’ self-reports, and using interpretability to corroborate self-reports. We also discuss challenges for our approach, from philosophical difficulties in interpreting self-reports to technical reasons why our proposal might fail. We hope our discussion inspires philosophers and AI researchers to criticize and improve our proposed methodology, as well as to run experiments to test whether self-reports can be made reliable enough to provide information about states of moral significance.

See also this earlier post for m...
Social Dark Matter
by Duncan Sabien (Inactive) — 2023-11-16T20:00:00.000Z

You know it must be out there, but you mostly never see it.

Author's Note 1: In something like 75% of possible futures, this will be the last essay that I publish on LessWrong. Future content will be available on my substack, where I'm hoping people will be willing to chip in a little commensurate with the value of the writing, and (after a delay) on my personal site (not yet live). I decided to post this final essay here rather than silently switching over because many LessWrong readers would otherwise never find out that they could still get new Duncanthoughts elsewhere.

Author’s Note 2: This essay is not intended to be revelatory. Instead, it’s attempting to get the consequences of a few very obvious things lodged into your brain, such that they actually occur to you from time to time as opposed to occurring to you approximately never.

Most people could tell you that 17 + 26 = 43 after a few seconds of thought or figuring, and it would be silly to write an essay about 17 + 26 equaling 43 and pretend that it was somehow groundbreaking or non-obvious.

But! If the point was to get you to see the relationship between 17, 26, and 43 very, very clearly, and to remember it sufficiently well that you would reflexively think “43” any time you saw 17 and 26 together in the wild, it might be worth taking the time to go slowly and say a bunch of obvious things over and over until it started to stick.

Thanks to Karim Alaa for the concept title. If you seek tl;dr, read the outline on the left and then skip to IX.

I. #MeToo

In September of 2017, if you had asked men in the United States “what percentage of the women that you personally know have experienced sexual assault?” most of them would have likely said a fairly low number.

In October of 2017, the hashtag #MeToo went viral.

In November of 2017, if you had asked men in the United States “what percentage of the women that you personally know have experienced sexual assault?” most of them would have given a much higher number than before.

(It’s difficult, for many people, to remember that they would have said a number that we now know to be outrageously low; by default most of us tend to project our present knowledge back onto our past selves. But the #MeToo movement was sufficiently recent, and the collective shock sufficiently well-documented, that we c...
AI #38: Let’s Make a Deal
by Zvi — 2023-11-16T19:50:05.442Z

Another busy week. GPT-5 starts, Biden and Xi meet and make somewhat of a deal, GPTs get explored, the EU AI Act on the verge of collapse by those trying to kill the part that might protect us, multiple very good podcasts. A highly interesting paper on potential deceptive alignment.

Despite things quieting down the last few days, it is still a lot. Hopefully things can remain quiet for a bit, perhaps I can even get in more work on that Jones Act post.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Structured prompts yay.

Language Models Don’t Offer Mundane Utility. Errors in error rates.

GPT-4 Real This Time. How do you protect against theft of your GPT?

Fun With Image Generation. Dalle-3 reluctant to let people have any fun.

Deepfaketown and Botpocalypse Soon. Terrorists ‘exploiting’ AI, they say.

A Bad Guy With an AI. Chemical weapon designs are not scary. What is?

They Took Our Jobs. Actors strike is over, economics of the panderverse.

Get Involved. Palisade, Nat Friedman, Topos Institute, Experience AI.

Introducing. DeepMind predicts weather, music tools. Gloves for voiced ASL.

In Other AI News. Work begins on GPT-5, Nvidia is defiant.

Quiet Speculations. How predictable are GPT-5’s abilities?

Anti Anti Trust. We are going to have a problem here.

The Quest for Sane Regulation. EU AI Act on verge of collapse or worse.

Bostrom Goes Unheard. Very good podcast, highly inaccurate representations. The link goes to a very long detailed analysis post, for those who care.

The Week in Audio. Barack Obama on Decoder.

Someone Picked up the Phone. Biden and Xi make a deal.

Mission Impossible. A solid very positive take.

Rhetorical Innovation. A new 2×2, ways to pick one’s poison.

Open Source AI is Unsafe and Nothing Can Fix This. Fun with CivitAi.

Aligning a Smarter Than Human Intelligence is Difficult. Interesting new papers.

People Are Worried About AI Killing Everyone. FTC Chair Lina Khan.

Other People Are Not As Worried. Timothy Lee, Baidu Reddy, Alex Kantowitz.

The Lighter Side. Insultingly short.

Language Models Offer Mundane Utility

GPTs mostly do not unlock strictly new capabilities, but they do let people store and share structured prompts.

Ethan Mollick: I think GPTs may have actually unlocked the potential of structured pr

...
Forecasting AI (Overview)
by jsteinhardt — 2023-11-16T19:00:04.218Z

This is a landing page for various posts I’ve written, and plan to write, about forecasting future developments in AI. I draw on the field of human judgmental forecasting, sometimes colloquially referred to as superforecasting. A hallmark of forecasting is that answers are probability distributions rather than single outcomes, so you should expect ranges rather than definitive answers (but ranges can still be informative!). If you are interested in learning more about this field, I teach a class on it with open-access notes, slides, and assignments.

For AI forecasting in particular, I first got into this area by forecasting progress on several benchmarks:

In Updates and Lessons from AI Forecasting, I describe a forecasting competition that I helped commission, which asked competitive forecasters to predict progress on four different benchmarks. This is a good place to start to understand what I mean by forecasting.

In AI Forecasting: One Year In, I look at the first year of results from the competition, and find that forecasters generally underpredicted progress in AI, especially on the MATH and MMLU benchmarks.

Motivated by this, in Forecasting ML Benchmarks in 2023 I provide my own forecasts of what state-of-the-art performance on MATH and MMLU will be in June 2023.

In AI Forecasting: Two Years In, I look at the second year of results from the competition. I found that the original forecasters continued to underpredict progress, but that a different platform (Metaculus) did better, and that my own forecasts were on par with Metaculus.

After these exercises in forecasting ML benchmarks, I turned to a more ambitious task: predicting the properties of AI models in 2030 across many different axes (capabilities, cost, speed, etc.). My overall predictions are given in What Will GPT-2030 Look Like?, which provides a concrete (but very uncertain) picture of what ML will look like at the end of this decade.

Finally, I am now turning to using forecasting to quantify and understand risks from AI:

In GPT-2030 and Catastrophic Drives: Four Vignettes, I use my GPT-2030 predictions as a starting point to understand the capabilities and corresponding risks of future ML models. I then speculate on four scenarios through which AI could lead to catastrophic outcomes.

In Base Rates for Catastrophe, I take a different approach, using data on historical catastrophes and extinc

...
We Should Talk About This More. Epistemic World Collapse as Imminent Safety Risk of Generative AI.
by Joerg Weiss — 2023-11-16T18:46:11.022Z

“So you’re telling me we can’t believe anything we’re shown anymore?” I’m asking. “That everything is altered? That everything’s a lie? That everyone will believe this?” “That’s a fact,” Palakon says. “So what’s true, then?” I cry out. “Nothing, Victor,” Palakon says. “There are different truths.”
Bret Easton Ellis – Glamorama, 1998

Photography has played a major role in allowing us to construct an understanding of the world whose general outlines are shared by billions of people. Functionally, photographic images extend our senses and tell us about the world outside of our immediate here and now. They are among the most important epistemological tools of humanity. Even today, after the transition to digital has made image manipulation easier and more widespread, they have not lost this role in our everyday lives. If we see a photograph or a video of something, we generally still believe in its factuality. At least in absence of reasons to doubt its veracity. With the advent of generative AI, this is about to change – sooner, more drastically and, given the abysmal state of our existing information landscape, with more dramatic consequences than we might intuit.

This, to me, seems to be an urgent AI safety concern that is simply not talked about enough. One reason is that other scenarios like paperclip optimizers, biases, mass unemployment and biohazards are dominating the safety discourse. We only have so much attention to give. Another reason is that photographic images are so deeply engrained in our everyday normal life. They are just part of our world and we take it for granted that we have the ability to see across oceans as well as through time. It is hard to imagine that very soon all of us, collectively, might have this ability severely diminished; or what the consequences of that might be.

I call this concern urgent, because we do not need to achieve AGI to get there. In fact, it seems likely no further large jumps in capabilities are needed at all. All that is missing seem to be smaller refinements of existing techniques. In some sense, the problems have already begun. I think of this as a safety issue with high p-doom and short timelines. Or in other words: It's likely to go bad, and soon.

Too ominous? Maybe I'm overdoing it? I'm going to try and convince you otherwise! Just before I start, one more thing: When I talk about photography or photographic ...
Intelligence in systems (human, AI) can be conceptualized as the resolution and throughput at which a system can process and affect Shannon information. 
by AiresJL — 2023-11-16T17:46:23.025Z

10+ years in Machine Learning Infrastructure Engineering - My perspective:

Intelligence in systems (human, AI) can be conceptualized as the resolution and throughput at which a system can process and affect Shannon information. This perspective emphasizes not just the quantity of information processed (throughput), but also the depth and detail with which it is handled (resolution), constrained to thermodynamic limitations.

Aside from improving the thermodynamics of the system here are the 7 levers we can use to improve its intelligence:

Physical Capacity: A system's intelligence increases as it expands its physical limits. This encompasses augmenting processing units (akin to neurons in humans or parameters in AI), improving thermal regulation, and maximizing energy throughput. Such enhancements enable a system to process information at a higher resolution and throughput.

Cooperation: When entities collaborate, the collective intelligence of the system increases. This is due to the improved resolution at which information can be processed and influenced, a principle manifest in ensemble methods in AI where multiple models aggregate their insights.

Conflict: The presence of conflict within or between systems can lead to an increase in intelligence. The necessity to adapt for survival and resolve conflicts escalates energy expenditure, which in turn refines the system's ability to process and affect information at a greater resolution and throughput.

Attention: Enhancing the range, depth, and sampling rate of information a system can process boosts its intelligence. This increase is achieved by allowing the system to operate with a wider context and more frequent assessments of information, thereby processing it at a higher resolution.

Actuation: Increasing the scope and precision of a system's actions directly impacts its intelligence. More diverse and precise actuation improves the system's capacity to affect information at a finer resolution.

Memory (Past): Building and utilizing shared physical memory elevates a system's intelligence by enabling it to process information over time at a higher resolution, fostering a more nuanced understanding of historical data.

Predictors (Future): The intelligence of a system can be significantly increased by evolving its core predictive model architecture. This includes developing new AI-designed architectures or employing tech...
Life on the Grid (Part 2)
by rogersbacon — 2023-11-16T17:22:08.296Z
Link: https://www.secretorum.life/p/life-on-the-grid-part-2

Previously:

Life on the Grid (Part 1)

Note: there is of course some continuation of themes from part 1, but this essay was written to stand on its own so don’t feel like you have to read the previous installment before digging into this one.

Confinement by

Sean Mundy

“We are all in the gutter, but some of us are looking at the stars.”
— Oscar Wilde

In part 1, I argued that our ability to navigate through both physical and mental landscapes (“fields of knowledge”) has degenerated, leaving us less willing to blaze trails or produce path-breaking innovations and generally lacking in agency and adventurousness. This degeneration of our navigational faculties has been caused by our reliance on automated wayfinding technologies and, more importantly, by an excessive “gridification” of the world, both materially (in our street networks and architecture) and socioeconomically (with our factory model schools and corporate ladders).

Here, we return to a theme that was only briefly touched on previously—the problem isn’t just that the world has become too grid-like, it is that, “nothing and nowhere escapes the techno-social net which we have cast over the planet. Uncharted territory has become a thing of the past.”

The seriousness of this problem cannot be overstated. Man feeds on terra incognita. The wildness of our imagination, the vitality of our spirit, the boldness of our dreams—these can only swell to their greatest extent when we feel as if there are hidden treasures or secrets waiting to be discovered.

For eons, our minds and cultures have evolved in delicate symbiosis with the Unknown, that place on the map labeled “Here Be Dragons”. Without this Unknown, that place where there may be cities of gold or fountains of youth, the heroes (but not just the heroes, all of us) have nowhere to journey and all of the things which can make us into heroes—bravery, fortitude, ingenuity, daring, and the like—begin to atrophy. Without this Unknown, we begin to feel confined, trapped, like a beautiful and dangerous animal in a small cage; we develop a claustrophobia; imagination and inspiration wither. For some reason, we aren’t as hopeful as we used to be, but we don’t know why. (part 1)

If the inaccessibility of physical terra incognita were the only problem, then that would be a good thing as the solution would be straightforward: colonize space, the so-called final frontier. Unfor...