trentmkelly/LessWrong-43k huggingface dataset:
Scenario planning for AI x-risk
by Corin Katzke — 2024-02-10T00:14:11.934Z
Link: https://forum.effectivealtruism.org/posts/tCq2fi6vhSsCDA5Js/scenario-planning-for-ai-x-risk

This post is part of a series by Convergence Analysis. In it, I’ll motivate and review some methods for applying scenario planning methods to AI x-risk strategy. Feedback and discussion are welcome.

Summary

AI is a particularly difficult domain in which to predict the future. Neither AI expertise nor forecasting methods yield reliable predictions. As a result, AI governance lacks the strategic clarity[1] necessary to evaluate and choose between different intermediate-term options.

To complement forecasting, I argue that AI governance researchers and strategists should explore scenario planning. This is a core feature of the AI Clarity program’s approach at Convergence Analysis. Scenario planning is a group of methods for evaluating strategies in domains defined by uncertainty. The common feature of these methods is that they evaluate strategies across several plausible futures, or “scenarios.”

One way scenario planning methods are differentiated is by how they use strategic parameters to generate scenarios. I use “strategic parameters” to mean features of the world that act as dimensions of uncertainty relevant to AI governance. With only a small number of scenarios, it’s possible to evaluate strategies deductively. Larger numbers of scenarios require more complicated methods.

Scenario planners for AI x-risk can evaluate strategies with respect to threat models and theories of victory. Threat models are descriptions of and proximal pathways to existential catastrophes. However, there are several challenges to specifying and using threat models. First, it isn’t clear which threat models are both plausible and describe truly existential outcomes. Second, less-than-existential threats might still indirectly affect x-risk. Third, expected AI x-risk mitigation is not a perfect proxy for expected value.

Theories of victory are descriptions of and proximal pathways to avoiding existential catastrophes. There is much less existing work on theories of victory than there is on threat models. Theories of victory should likely be developed to meet certain conditions, such as 1) plausibility 2) action-guidingness, and 3) sustainability.

I recommend that AI governance researchers and strategists use scenario planning methods both to evaluate strategies as well as identify the right questions to forecast.

The Problem

...
Close the Gates to an Inhuman Future: How and why we should choose to not develop superhuman general-purpose artificial intelligence
by aaguirre — 2024-02-09T20:25:11.634Z
Link: https://arxiv.org/abs/2311.09452

This is a revised version of a paper I put out late last year (including a new table about capabilities lacking in current AI and how we might get them.)

Interested in reactions and potential improvements, as I'm contemplating various possibilities for either expansion or distillation/repackaging of this basic picture, for a variety of audiences.
[Crosspost] Deep Dive: The Coming Technological Singularity - How to survive in a Post-human Era
by simulacra.exe — 2024-02-09T18:49:52.677Z

Crossposted from my Substack: Part One and Part Two

“The acceleration of technological progress has been the central feature of this century. I argue in this paper that we are on the edge of change comparable to the rise of human life on Earth. The precise cause of this change is the imminent creation by technology of entities with greater than human intelligence.”[1]

On the face of this paragraph alone, nothing here is particularly shocking. With the rise of discussion and debate about the increasing capability of AI, the possibility of an AGI (Artificial General Intelligence) and existential risk, this feels like one of many opening paragraphs that we’d see on social media.

Plot twist - this essay is from 1993.

Vernor Vinge, who is not only a prolific science fiction writer but a retired professor of computer science and mathematics, seems to be a perfect candidate for this speculative essay; a future thinker grounded in the practicalities and reasoning that comes with a STEM background. I had another look through this essay - I did a stream on Twitch discussing this and am planning to make a YouTube video on the topic in the future - and there are many points that still resonate today. This also needs to be made abundantly clear: when Vinge states that we are the “edge of change”, he clarifies this by declaring “I believe that the creation of greater than human intelligence will occur during the next thirty years”. The abstract to this paper was similarly shocking: “within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.” I’m taking the definition of human era to be a period of time where humans are no longer actively shaping the world with the power and control that is does now - rather, the end of the Anthropocene. However, the most striking detail here is when it was written. 1993. In other words, he predicted this to occur around 2023. This year. (when this was originally published)

Your mileage, of course, may vary on this - with conspiracy theories that GPT5 has been achieved, that even GPT4 shows sparks of sentience, the “meme” posted by Sam Altman himself (CEO of OpenAI) that AGI had been internally achieved - this seems to be a matter of opinion whether the sparks of possibility are present, or that it is actually here already but lurking in the shadows, or simply just a ...
The Ideal Speech Situation as a Tool for AI Ethical Reflection: A  Framework for Alignment
by kenneth myers — 2024-02-09T18:40:55.881Z

Please be merciful, this is my first post. This is a potentially idealistic, but theoretically intriguing conceptual avenue for thinking about aligning AI systems. This is all extremely high level and doesn't propose any kind of concrete solutions. Rather, this is an attempt to reframe the alignment problem in what I think is a novel way. It's with some fear and trembling that I offer this reframing to the LessWrong community.

The challenge of ensuring powerful AI adheres to complex human values often centers on imposing behaviors or pre-programming specific goals. These approaches may miss the dynamic, fundamentally social context in which ethics unfolds. Jürgen Habermas' Ideal Speech Situation (ISS) offers a powerful alternative grounded in communication governed by ethical principles. This concept could revolutionize AI alignment by shifting the focus from external constraint to the emergence of ethical communication capabilities within the AI itself.

ISS Core Principles

The ISS envisions a conversational environment where the most justified outcomes prevail – not via force, but due to reason and the openness of all participants:

Equal Voice & Opportunity: All parties capable of communication should be free to participate without systematic limitation due to background, knowledge, or status. This poses a core challenge for AI, with its inherent power disparity relative to humans.

Sincerity & Transparency: Participants aim to express genuine beliefs in ways others can understand. Misdirection, withholding information, or deliberate manipulation erode the possibility of genuine agreement.

Freedom from Coercion: Arguments must be accepted or rejected based on their inherent strength, not through threats, bribes, or appeals to emotion that bypass rational evaluation.

Focus on Reasoned Agreement: Instead of seeking pre-programmed 'truths,' the emphasis is on identifying the most defensible position at a given moment through shared evidence and logical justifications others could acknowledge, regardless of initial stance.

Internalizing the ISS: Ethical Reflection Within AI

The most radical shift with this approach is the goal of equipping AI with the tools to critically evaluate itself in this framework:

Formalizing 'Good Faith': Could an AI discern the difference between its own 'sincere' communication attempts and those intended to mislead? It needs

...
What’s ChatGPT’s Favorite Ice Cream Flavor? An Investigation Into Synthetic Respondents
by Greg Robison — 2024-02-09T18:38:56.830Z

Introduction

Welcome to a deep dive into the fascinating world of artificial intelligence (AI), where we are comparing synthetic responses from advanced large language models (LLMs) with real human survey responses about a universally beloved topic: ice cream flavors. AI use is spreading so quickly that understanding how well it can replicate or predict human choices, especially in something as subjective as taste preferences, is both fascinating and vital to understanding the strengths and weaknesses of LLMs as substitutes for human survey respondents.

Ice cream, with its endless varieties of flavors, serves as the perfect topic for this exploration. Everyone has a favorite – be it classic vanilla, rich chocolate, or something more complex like salted caramel – but can an AI, devoid of taste buds and any real personal experiences, accurately predict or simulate these preferences? This question is at the heart of our study.

Exploring the capabilities of Large Language Models (LLMs) in predicting human preferences is crucial for survey research on such topics as product development and marketing messaging. These models offer the potential to complement human feedback, providing quick, iterative insights that can inform marketing strategies and product innovations. However, a key aspect of this exploration is assessing the validity of the insights generated by LLMs. We aim to clarify the credibility of the data derived from these models, addressing the critical question: is the information provided by LLMs valid? Are some LLMs to be trusted more than others? Let’s find out.

Background

Large Language Models (LLMs) like ChatGPT, which achieved the milestone of being the fastest app to reach 100 million users, represent a significant breakthrough in the field of artificial intelligence. These models are gaining immense popularity due to their ability to understand and generate human-like text, making them incredibly versatile tools for a wide range of applications.

At the core of LLMs is a deep learning architecture known as the transformer which enables the model to process and generate text by recognizing the context and relationships between words in a sentence. LLMs like ChatGPT are trained on vast amounts of text data and through such training, the models learn to predict the next words in a sentence given a specific text input. They acquire an “understanding” of ...
OpenAI wants to raise 5-7 trillion 
by O O — 2024-02-09T16:15:00.421Z
Link: https://decrypt.co/216731/openai-considers-7-trillion-round-thinks-it-can-double-revenue-by-2025-reports?amp=1

This 5-7 trillion is to enhance GPU production and he also sees revenue doubling from current $2 billion to $4 billion next year.

I’m taking a guess GPT-5 training is going well?
Constituency-sized AI congress?
by Nathan Helm-Burger — 2024-02-09T16:01:09.592Z

I just had the idea for a constituency-sized AI congress. Each member of the constituency would have their personal debate agent trained on their preferences and values. The agents would debate tirelessly at superhuman speed to develop proposals which represented the best available win-win compromises given the issues on the table.

The final proposals after a set amount of debate would be presented to the constituency and voted on.

I haven't researched this yet or thought about it for long. I'd love for your feedback on the idea and links to related work.
One True Love
by Zvi — 2024-02-09T15:10:05.298Z

We have long been waiting for a version of this story, where someone hacks together the technology to use Generative AI to work the full stack of the dating apps on their behalf, ultimately finding their One True Love.

Or at least, we would, if it turned out he is Not Making This Up.

Fun question: Given he is also this guy, does that make him more or less credible?

Alas, something being Too Good to Check does not actually mean one gets to not check it, in my case via a Manifold Market. The market started trading around 50%, but has settled down at 15% after several people made strong detailed arguments that the full story did not add up, at minimum he was doing some recreations afterwards.

Which is a shame. But why let that stop us? Either way it is a good yarn. I am going to cover the story anyway, as if it was essentially true, because why should we not get to have some fun, while keeping in mind that the whole thing is highly unreliable.

Discussion question throughout: Definitely hire this man, or definitely don’t?

With that out of the way, I am proud to introduce Aleksandr Zhadan, who reports that he had various versions of GPT talk to 5,240 girls on his behalf, one of whom has agreed to marry him.

I urge Cointelegraph, who wrote the story up as ‘Happy ending after dev uses AI to ‘date’ 5,239 women, to correct the error – yes he air quotes dated 5,239 other girls, but Karina Imranovna counts as well, so that’s 5,240. Oops! Not that the vast majority of them should count as dates even in air quotes.

Aleksandr Zhadan (translated from Russian): I proposed to a girl with whom ChatGPT had been communicating for me for a year. To do this, the neural network re-communicated with other 5239 girls, whom it eliminated as unnecessary and left only one. I’ll share how I made such a system, what problems there were and what happened with the other girls.

For context

• Finding a loved one is very difficult

• I want to have time to work, do hobbies, study and communicate with people

• I could go this route myself without ChatGPT, it’s just much longer and more expensive

In 2021 I broke up with my girlfriend after 2 years. She influenced me a lot, I still appreciate her greatly. After a few months, I realized that I wanted a new relationship. But I also realized that I didn’t want to waste my time and feel uncomfortable with a new girl.

Where did the relation

...
Executive function advice from people who are good at it?
by TeaTieAndHat — 2024-02-09T10:11:54.546Z

Status: just a quick question that probably can’t get a good answer

LessWrong has a lot of advice on how to beat akrasia, most of which is written by people who had good ideas for the sort of things that might work, tested them on themselves, and found out that they did work really well. But, of course, advice is disproportionately written by defective people: pretty much everyone who investigates how to improve executive functioning does so because they start out being terrible at it. And, to a large extent, it makes sense: if a successful person gives advice, why would they be particularly good at telling apart what mattered to their success and what didn’t, and to what extent they just got lucky?

Still: I’m curious what would be the executive function advice given by the sort of people who manage to pull off ten-hour workdays, or from people doing their medical residency, etc.
Skills I'd like my collaborators to have
by Raemon — 2024-02-09T08:20:37.686Z

I originally wrote this in 2021, when Lightcone was exploring a vision of building "a rationalist campus" or maybe "a rationalist society."

Right now, neither I, nor Lightcone as a whole, are focused on this in the same way. You *might* make a guild that taught these skills on purpose, or a barrier-to-entry that filtered for them. That’s not where I’m currently focusing my plans.

But, I mostly am publishing it now because I still roughly like it as an aspirational world I'd like to be a part of some day. I like these skills, and I'd like my collaborators to have them, however that ends up being mediated. I haven't edited the post since 2021.

At the LW Team, we’ve been tossing around the idea of founding some kind of rationalist/x-risk guild, society, or other closed barrier social/professional network.

There’s a bunch of reasons I think a guild might be good. But one key thing I’m interested in is “Have a cluster of people who reliably have a particular set of skills, and a particular orientation towards x-risk, such that one can reliably find allies.”

In the current status quo, there is a vaguely defined rationalist/EA-landscape where people are filtered for some basic level of IQ, understanding of some foundational essays (but, not any particular foundational essay), and some vague pressure to be successful on at least some dimensions. It’s amorphous and unpredictable.

I’d like to have a network of people I can reliably trust. Reasons this seems important to me:

I want to find collaborators
I want to feel safe investing in social connections.
I want people I can trust to “think out loud with”
I think

changing social environment

is one of the most promising ways to grow as a person. Unfortunately, most of the time the best option is “randomly re-roll your job or roommates.” A pool of people with important rationality skills and attitudes might be helpful for having some control over how to reroll your social environment.

I think having a guild would provide a framework for both making sure people meet some basic bar of “be good at thinking, and good at collaborating”, as well as training people in new, deeper skills over time. Ideally, the bar for guild members goes up over time as we figure out better skills and better ways of teaching them.

In this Very Rough Draft of a plan, I think there should be some skills that are core requiremen...
Transfer learning and generalization-qua-capability in Babbage and Davinci (or, why division is better than Spanish)
by RP — 2024-02-09T07:00:45.825Z

Tl/Dr:

Generalisation is a capability in its own right, and in our experiments it scales less quickly than specific capabilities.
Our experiments show that on a range of tasks, finetuning Babbage on task A transfers more to task B than finetuning Davinci on task A transfers to task B. This is relevant for identifying the nature of the alignment problem we might encounter (for things in this vague cluster see

here

,

here

, and

here

).
Of independent interest is the fact that a transfer learning metric can give us a fairly non-arbitrary way to measure how similar any two tasks are.

Epistemic status: written quickly, but oh look we did 4×8^2 experiments and we have a lovely graphic

This post makes two contributions. First, we compare results of transfer learning experiments from Babbage and Davinci and suggest that, at least in these experiments, the ability to generalize scales less quickly than do specific capabilities. Then, we suggest and implement a measure of task-to-task similarity by using results from transfer learning. The first is more relevant to alignment; the second is of independent philosophical interest.

Along the way, we also comment on various interesting results, such as that good performance on division helps far more with multiplication than good performance on subtraction does with addition. Our code is available at github.com/ag8/transfer-scaling.

Setup

We picked 8 somewhat random tasks: five mathematical problems and three text-based multiple-choice questions given in trivia style. For every pair, we first finetuned on task A, then briefly finetuned on task B, and compared performance on task B to the case where we had only briefly finetuned on task B. For example, we finetuned on 200 examples of addition followed by 10 of subtraction, and compared the model’s subsequent performance on subtraction to its performance after only being finetuned on 10 examples of subtraction. We did this for each pair of tasks both for Babbage and Davinci.

Results

The table below presents our findings in each case. Results are given as percentage point improvement between each model’s performance on the column dataset after being fine-tuned on 10 examples from the column dataset, and the performance after being fine-tuned first on 200 examples from the row dataset, followed by 10 examples from the column dataset.

The left heat...
Biden-Harris Administration Announces First-Ever Consortium Dedicated to AI Safety
by Ben Smith — 2024-02-09T06:40:44.427Z
Link: https://www.nist.gov/news-events/news/2024/02/biden-harris-administration-announces-first-ever-consortium-dedicated-ai

"Today, U.S. Secretary of Commerce Gina Raimondo announced the creation of the

U.S. AI Safety Institute

Consortium (AISIC), which will unite AI creators and users, academics, government and industry researchers, and civil society organizations in support of the development and deployment of safe and trustworthy artificial intelligence (AI)"

It's not clear to me whether this is also the first announcement of the "US AI Safety Institute" as well as the Consortium as a possible separate thing, but either way, this seems like great news.
Running the Numbers on a Heat Pump
by jefftk — 2024-02-09T03:00:04.920Z

I recently looked into installing a heat pump mini-split system to
heat and cool our house. While it's expensive to install, I had
thought a combination of it being nicer than what we currently have
and cheaper to operate would make it worth it. Unfortunately, after
running the numbers on our utility costs it turns out it would be
substantially more expensive to power than our current gas system.

I got excited about a potential heat pump system after learning that
over the past few years their cold-weather performance has improved
dramatically and then that MA would provide a $20k subsidy ($10k for
each of two units) for us to switch over. I liked that it would allow
us to adjust the level of heating we had in each room, provide heat to
one room that currently is only heated passively through the walls,
replace the small amount of resistive heat we use in another room, and
provide cooling (where we currently use a whole-house fan, plus window
units during the hottest part of the summer). Friends who had
installed them were happy, and then I found someone putting together a
buying
collective to negotiate pricing. I got a MassSave energy audit,
to qualify for the $20k rebate, talked to the contractor (great experience!)
and got a quote. Since it was a lot of money I wanted to run the
numbers first, and then that's when my excitement drained.

The quote was for Mitsubishi
MXZ-SM42NAMHZ mini-split condensers, which are pretty efficient,
and have a Region IV [1] HSPF2 rating of
10.55 (10,550 BTU/kWh) [2] . This means that averaging over the
heating season, to produce 100k BTU of heat
we'd need 9.48 kWh of electricity. Burning one therm of natural gas
also gives you 100k BTU, but our condensing boiler is rated 96%
efficient so we'd need 1.04 therms for that 100k BTU.

Massachusetts has some of the highest
energy costs in the country, and our marginal costs are
$1.999/therm [3] for gas and $0.316/kWh for electric. That means heating
our house by 100k BTU would cost $2.08 with gas vs $3.00 for electric
(44% more). What's the scale of this?

In 2023 we burned 1,199 therms of natural gas. During the summer when
the heat was off we burned ~16 therms a month, for hot water, so I'll
estimate we used 1,007 therms for heating and 192 for hot water.

Doing the same calculation for more years, it sounds like 1,000 therms
is typical:

Year
Th

...
How do high-trust societies form?
by Shankar Sivarajan — 2024-02-09T01:11:24.201Z

First, it's clear to me they're not just myths bolstered by propaganda, and they really are qualitatively different from their low-trust counterparts, which I understand, and consider to be "natural," requiring no explanation.

I think it's a different concept than the WEIRD (Western, Educated, Industrialized, Rich, Democratic) thing, though of course, there's a lot of overlap.

A related question: how do they unravel, and can this be arrested?
How do health systems work in adequate worlds?
by mukashi — 2024-02-09T00:54:38.443Z

I'm confused about Health Systems in general, the same way I was confused about the Education System until I read "The case against education". I don't have a good model of what are the different agents interacting in this system, nor the differences among different countries. Is there any reading you would recommend to be a bit less confused about this?