trentmkelly/LessWrong-43k huggingface dataset:
Architects of Our Own Demise: We Should Stop Developing AI Carelessly
by Roko — 2023-10-26T00:36:05.126Z

Some brief thoughts at a difficult time in the AI risk debate.

Imagine you go back in time to the year 1999 and tell people that in 24 years time, humans will be on the verge of building weakly superhuman AI systems. I remember watching the anime short series The Animatrix at roughly this time, in particular a story called The Second Renaissance I part 2 II part 1 II part 2 . For those who haven't seen it, it is a self-contained origin tale for the events in the seminal 1999 movie The Matrix, telling the story of how humans lost control of the planet.

Humans develop AI to perform economic functions, eventually there is an "AI rights" movement and a separate AI nation is founded. It gets into an economic war with humanity, which turns hot. Humans strike first with nuclear weapons, but the AI nation builds dedicated bio- and robo-weapons and wipes out most of humanity, apart from those who are bred in pods like farm animals and plugged into a simulation for eternity without their consent.

Surely we wouldn't be so stupid as to actually let something like that happen? It seems unrealistic.

And yet:

AI software and hardware companies are rushing ahead with AI
The technology for technical AI safety (things like interpretability, RLHF, governance structures) is still very much in its infancy. The field is something like 5 years old.
People are already talking about an AI rights movement in major national papers
There isn't a plan for what to do when the value of human labor goes to zero
There isn't a plan for how to deescalate AI-enhanced warfare, and militaries are enthusiastically embracing killer robots. Also, there are two regional wars happening and a nascent superpower conflict is brewing.
The game theory of different opposing human groups all rushing towards superintelligence is horrible and nobody has even proposed a solution. The US government has foolishly stoked this particular risk by cutting off AI chip exports to China.

People on this website are talking about responsible scaling policies, though I feel that "irresponsible scaling policies" is a more fitting name.

Obviously I have been in this debate for a long time, having started as a commenter on Overcoming Bias and Accelerating Future blogs in the late 2000s. What is happening now is somewhere near the low end of my expectations for how competently and safely humans would handle the coming transition to machi...
EA Infrastructure Fund: June 2023 grant recommendations
by Linch — 2023-10-26T00:35:07.981Z

Introduction

This payout report covers the Effective Altruism Infrastructure Fund (EAIF)’s grantmaking starting January 2022 (after our

December 2021 payout report

), until early June 2023 (1 January 2022 - 15 June 2023).

In 2022, the EAIF received 595 applications, requesting a total of $77.8M in funding. Of these, the EAIF desk-rejected 77 applications[1]. Among the remaining 518 applications, the EAIF recommended funding 264 applications (51%) for a total of $11,265,585. A further 10 applications (1.9%) were referred to private funders for a total of $59,289.

In 2023, the EAIF (as of June 5th) received 280 applications, requesting a total of $17.1M in funding. Of these, we desk-rejected 52, and (as of June 5th) were evaluating 34 applications. Among the remaining 194 applications, we recommended funding 83 applications (43%) for a total of $2,079,927. One application (0.52%) was referred to a private funder for a total of $10,000.

If you’re interested in receiving funding from the EA Infrastructure Fund,

apply here

. If you’re interested in supporting the EA Infrastructure Fund,

donate here

.[2]

The current version of this post was primarily written by Tom Barnes and Linch Zhang. Significant writing and feedback came from Max Daniel (who wrote the earliest draft), Michelle Hutchinson, Peter Wildeford, and Caleb Parikh.

Background

(This section was written by Linch Zhang)

The EA Infrastructure Fund last published a payout report for the period

September - December 2021

. Since then, we decided to deprioritize the publication of payout reports, as EAIF’s capacity has been low. Instead, EA Funds launched a

Public Grants Database

which provides basic information on all grants.

Nonetheless, we believe that a payout report is (long) overdue, especially given a

number of changes to EA Funds

. For the EA Infrastructure Fund, the most immediately relevant updates are:

EAIF (along with LTFF) is significantly

funding constrained

.[3] If donations to EAIF continue at their (post-FTX) rate, EAIF will be unable to make ~$3.5 million in impactful grants[4] over the next 6 months.[5]
Open Philanthropy is matching donations to the EAIF (and LTFF) until the end of January 2024, at a rate of 2:1 (i.e., a $1 donation will lead to EAIF granting out $3). At the time of writing, ~30% of the match has been filled.[6]

...
Responsible Scaling Policies Are Risk Management Done Wrong
by simeon_c — 2023-10-25T23:46:34.247Z
Link: https://www.navigatingrisks.ai/p/responsible-scaling-policies-are

Summary

TLDR

Responsible Scaling Policies (RSPs) have been

recently

proposed

as a way to keep scaling frontier large language models safely.

While being a nice attempt at committing to specific practices, the framework of

RSP

is:

missing core components of basic risk management procedures (

Section 2

& 3)
selling a rosy and misleading picture of the risk landscape (

Section 4

)
built in a way that allows overselling while underdelivering (

Section 4

)

Given that, I expect the RSP framework to be negative by default (Section 3, 4 and 5). Instead, I propose to build upon risk management as the core underlying framework to assess AI risks (

Section 1

and 2). I suggest changes to the RSP framework that would make it more likely to be positive and allow to demonstrate what it claims to do (

Section 5

).

Section by Section Summary:

General Considerations on AI Risk Management

This section provides background on risk management and a motivation for its relevance in AI.

Proving risks are below acceptable levels is the goal of risk management.
To do that, acceptable levels of risks (not only of their sources!) have to be defined.
Inability to show that risks are below acceptable levels is a failure. Hence, the less we understand a system, the harder it is to claim safety.
Low-stake failures are symptoms that something is wrong. Their existence make high-stake failures more likely.

Read more.

What Standard Risk Management Looks Like

This section describes the main steps of most risk management systems, explains how it applies to AI, and provides examples from other industries of what it looks like.

Define Risk Levels: Set acceptable likelihood and severity.
Identify Risks: List all potential threats.
Assess Risks: Evaluate their likelihood and impact.
Treat Risks: Adjust to bring risks within acceptable levels.
Monitor: Continuously track risk levels.
Report: Update stakeholders on risks they incur and measures taken.

Read more.

RSPs vs Standard Risk Management

This section provides

a table

comparing RSPs and generic risk management standard ISO/IEC 31000, explaining the weaknesses of RSPs.

It then provides a list of 3 of the biggest failures of RSPs compared with risk management.

Prioritized RSPs failures against risk management:

Using underspecified definitions of risk thresholds and not quantifying

...
AI as a science, and three obstacles to alignment strategies
by So8res — 2023-10-25T21:00:16.003Z

AI used to be a science. In the old days (back when AI didn't work very well), people were attempting to develop a working theory of cognition.

Those scientists didn’t succeed, and those days are behind us. For most people working in AI today and dividing up their work hours between tasks, gone is the ambition to understand minds. People working on mechanistic interpretability (and others attempting to build an empirical understanding of modern AIs) are laying an important foundation stone that could play a role in a future science of artificial minds, but on the whole, modern AI engineering is simply about constructing enormous networks of neurons and training them on enormous amounts of data, not about comprehending minds.

The

bitter lesson

has been taken to heart, by those at the forefront of the field; and although this lesson doesn't teach us that there's nothing to learn about how AI minds solve problems internally, it suggests that the fastest path to producing more powerful systems is likely to continue to be one that doesn’t shed much light on how those systems work.

Absent some sort of “science of artificial minds”, however, humanity’s prospects for aligning smarter-than-human AI seem to me to be quite dim.

Viewing Earth’s current situation through that lens, I see three major hurdles:

Most research that helps one

point AIs

, probably also helps one make more capable AIs. A “science of AI” would probably increase the power of AI far sooner than it allows us to solve alignment.
In a world without a mature science of AI, building a bureaucracy that reliably distinguishes real solutions from fake ones is prohibitively difficult.
Fundamentally, for at least some aspects of system design, we’ll need to rely on a theory of cognition working on the first high-stakes real-world attempt.

I’ll go into more detail on these three points below. First, though, some background:

Background

By the time AIs are powerful enough to endanger the world at large, I expect AIs to do something akin to “caring about outcomes”, at least from a behaviorist perspective (making no claim about whether it internally implements that behavior in a humanly recognizable manner).

Roughly, this is because people are trying to make AIs that can steer the future into narrow bands (like “there’s a cancer cure printed on this piece of paper”) over long time-horizons...
My hopes for alignment: Singular learning theory and whole brain emulation
by Garrett Baker — 2023-10-25T18:31:14.407Z

Some prerequisites needed in order for this to make sense:

Two Subsystems: Learning & Steering
Shard Theory: An Overview
Distilling Singular Learning Theory[1]
Maybe also understanding at least a little Nate's picture though I don't claim to understand it fully.
Of course, AGI Ruin: A List of Lethalities, though hopefully that was implied
Maybe the very basics of infra-bayesianism (I liked the AXRP podcast with her (there's two)), Vanessa's original learning theoretic agenda philosophy, and her current pre-DCA alignment scheme.

Abstract

The philosophy behind infra-bayesianism, and Vanessa's learning-theoretic alignment agenda seems very insightful to me. However, the giant stack of assumptions needed to make the approach work, and the ontology-forcing nature of the plan leave me unsettled. The path of singular learning theory has recently seen enough empirical justification to excite me. I'm optimistic it can describe brains & machine learning systems, and I describe my hope that this can be leveraged into alignment guarantees between the two, becoming an easier task as whole brain emulation develops.

Introduction

Imagine a world where instead of humanity wandering together blindly down the path of cobbled together weird tricks learned off the machine learning literature, with each person trying to prepare for the bespoke failure-mode that seems most threatening to them (most of whom are wrong).

That instead we lived in the world where before heading down we were given a map and eyes to see for ourselves the safest path, and the biggest and most dangerous cliffs, predators, and dangers to be aware of.

It has always seemed to me that we get to the second world once we can use math for alignment.

The left: Our situation. The right: Our situation if we had math for alignment.
Generated with DALL·E 3.

For a long time I thought using math for alignment was essentially a non-starter. Deep learning is notoriously resistant to successful theories, I'm pessimistic that the approach of the Machine Intelligence Research Institute will take too much time, and the most successful mathematical theory of intelligence and alignment--infra-Bayesianism--rests on a stack of assumptions and mathematical arguments too high, too speculative, and too normative for me to be optimistic about. So I resigned myself to the lack of math for alignment.

That is, until Nina Rimsky &...
Lying to chess players for alignment
by Zane — 2023-10-25T17:47:15.033Z

Eliezer Yudkowsky recently posted on Facebook an experiment that could potentially indicate whether humans can "have AI do their alignment homework" despite not being able to trust whether the AI is accurate: see if people improve in their chess-playing abilities when given advice from experts, two out of three of which are lying.

I'm interested in trying this! If anyone else is interested, leave a comment. Please tell me whether you're interested in being:

A) the person who hears the advice, and plays chess while trying to determine who is trustworthy

B) the person who they are playing against, who is normally better at chess than A but worse than the advisors

C) one of the three advisors, of which one is honestly trying to help and the other two are trying to sabotage A; which one is which will be chosen at random after the three have been selected to prevent A from knowing the truth

Feel free, and in fact encouraged, to give multiple options that you're open to trying out! Who gets assigned to what role would depend on how many people respond and their levels of chess ability, and it's easier to find possible combinations with more flexibility in whose role is which.

Please also briefly describe your level of experience in chess. How frequently have you played, if at all; if you have ELO rating(s), what are they and which organizations are they from (FIDE, USCF, Chess.com, etc). No experience is required! In fact, people who are new to the game are actively preferred for A!

Finally, please tell me what days and times you tend to be available - I won't hold you to anything, of course, but it'll help give me an estimate before I contact you to set up a specific time.

Edit: also, please say how long you would be willing to play for - a couple hours, a week, a one-move-per-day game over the course of months? A multi-week or multi-month game would give the players a lot more time to think about the moves and more accurately simulate the real-life scenario, but I doubt everyone would be up for that.

Edit 2: GoteNoSente suggested using a computer at a fixed skill level for player B, which in retrospect is clearly a great idea.

Edit 3: there is now a Google Form for signing up: https://docs.google.com/forms/d/e/1FAIpQLScPKrSB6ytJcXlLhnxgvRv1V4vMx8DXWg1j9KYVfVT1ofdD-A/vi...
Anthropic, Google, Microsoft & OpenAI announce Executive Director of the Frontier Model Forum & over $10 million for a new AI Safety Fund 
by Zach Stein-Perlman — 2023-10-25T15:20:52.765Z
Link: https://www.frontiermodelforum.org/announcement/

Today, Anthropic, Google, Microsoft, and OpenAI are announcing the selection of Chris Meserole as the first Executive Director of the Frontier Model Forum, and the creation of a new AI Safety Fund, a more than $10 million initiative to promote research in the field of AI safety. The Frontier Model Forum, an industry body focused on ensuring safe and responsible development of frontier AI models, is also releasing its first technical working group update on red teaming to share industry expertise with a wider audience as the Forum expands the conversation about responsible AI governance approaches.

Chris Meserole appointed the first Executive Director of the Frontier Model Forum, an industry body focused on ensuring safe and responsible development and use of frontier AI models globally.
Meserole brings a wealth of experience focusing on the governance and safety of emerging technologies and their future applications.
Today Forum members, in collaboration with philanthropic partners, the Patrick J. McGovern Foundation, the David and Lucile Packard Foundation, Eric Schmidt, and Jaan Tallinn, and commit over $10 million for a new AI Safety Fund to advance research into the ongoing development of the tools for society to effectively test and evaluate the most capable AI models.

Executive Director

Chris Meserole

comes to the Frontier Model Forum with deep expertise on technology policy, having worked extensively on the governance and safety of emerging technologies and their future applications. Most recently he served as Director of the Artificial Intelligence and Emerging Technology Initiative at the Brookings Institution.

In this new role, Meserole will be responsible for helping the Forum fulfill its mission to (i) advance AI safety research to promote responsible development of frontier models and minimize potential risks, (ii) identify safety best practices for frontier models, (iii) share knowledge with policymakers, academics, civil society and others to advance responsible AI development; and (iv) support efforts to leverage AI to address society’s biggest challenges.

“The most powerful AI models hold enormous promise for society, but to realize their potential we need to better understand how to safely develop and evaluate them. I’m excited to take on that challenge with the Frontier Model Forum.”

Chris Meserole

AI Safety Fund

Over the past...
"The Economics of Time Travel" - call for reviewers (Seeds of Science)
by rogersbacon — 2023-10-25T15:13:59.647Z

Abstract

The lack of time travellers visiting us may be seen as evidence that time travel is not possible. In this article, I argue an alternative explanation is that we are not economically important enough to our descendants to justify the costs of time travel. Using a cost-benefit analysis, I elaborate on this argument. I suggest that the major cost of time travel is likely to be the energy cost, while the largest benefit of time travel is knowledge which the present possesses, but the future has lost. Focusing on this benefit component, I argue it is extremely unlikely that we possess a piece of knowledge which is sufficiently important to a future civilisation (system critical), but also has been lost by this civilisation. This is to say, we may not have been visited by time travellers because we are not important enough.

---

Seeds of Science is a journal (funded through Scott Alexander's ACX grants program) that publishes speculative or non-traditional articles on scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them). Comments that critique or extend the article (the "seed of science") in a useful manner are published in the final document following the main text.

We have just sent out a manuscript for review, "The Economics of Time Travel", that may be of interest to some in the LessWrong community so I wanted to see if anyone would be interested in joining us as a gardener and providing feedback on the article. As noted above, this is an opportunity to have your comment recorded in the scientific literature (comments can be made with real name or pseudonym).

It is free to join as a gardener and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so no worries if you don't plan on reviewing very often but just want to take a look here and there at the articles people are submitting).

To register, you can fill out this google form. From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review form for recording votes/comments. If you would like to just take...
Compositional preference models for aligning LMs
by Tomek Korbak — 2023-10-25T12:17:28.990Z

This post summarizes the main results from our recently released paper

Compositional preference models for aligning LMs

and puts them in the broader context of AI safety. For a quick summary of the paper, take a look at our

Twitter thread

.

TL;DR: We propose a new approach to building preference models out of prompted LMs. Compositional Preference Models (CPMs) decompose scoring a text into (1) constructing a series of questions about interpretable features of that text (e.g. how informative it is), (2) obtaining scalar scores for these features from a prompted LM (e.g. ChatGPT), and (3) aggregating these scores using a logistic regression classifier trained to predict human judgements. We show that CPMs, compared with standard preference models (PMs), generalize better and are more robust to reward model overoptimization. Moreover, best-of-n samples obtained using CPMs tend to be preferred over samples obtained using similar, conventional PMs. Finally, CPMs are a novel angle at scalable oversight: they decompose a hard evaluation problem into a series of simpler, human-interpretable evaluation problems.

How compositional preference models work?

Figure 1: While standard PMs output a preference score directly, CPMs score different features of LM responses separately and output a preference score as a linear combination of feature values.

Preference Models (PMs) are models trained to assign an LM response a score indicating the quality of the response. They are the workhorse of many techniques for aligning LMs: they are most prominently used as reward functions in RLHF or as ranking models in best-of-n sampling, in addition to playing a role in other techniques such as

pretraining with human feedback

.

Standard PMs involve adding a scalar head on top of a base model and finetuning the whole model (or certain upper layers) to predict which of two texts a human would prefer. While this approach is highly effective in practice, it can lead to uninterpretable models that fit spurious correlations in human preference judgements and are prone to goodharting (overoptimization).

We introduce an alternative: Compositional Preference Models (CPM). In contrast to PMs, CPMs decompose response evaluation into the following steps:

Feature decomposition. We maintain a fixed list of 13 human-interpretable features (e.g. specificity, relevance, readability) a...
Should the US House of Representatives adopt rank choice voting for leadership positions?
by jmh — 2023-10-25T11:16:14.223Z

One obvious take, partisanship and party power interest, might suggest a strong "No!" type of response. I would certainly expect to have many current Representatives to respond that way if floated as an idea today. But perhaps I am wrong.

I would think the current state of things, the apparent incapacitation of the House could benefit strongly from such a voting structure as it allows for a more functional process that what currently exists.

What would all the pros and cons be for such a procedural change?
Researchers believe they have found a way for artists to fight back against AI style capture
by vernamcipher — 2023-10-25T10:54:23.652Z
Link: https://finance.yahoo.com/news/artists-fight-back-against-ai-211000482.html

The real details are here: What is Glaze?

I wonder how robust this technique is to adversarial attacks. It claims to work at the pixel level, masking an image by distorting the pixels in ways invisible to the human eye but which throw off a generative AI model.

But what about a high resolution photograph of an artwork, or a image of one on a computer monitor? I often take photographs of artworks I find compelling, then use Midjourney to style copy them to make novel images.

Certainly this is not scalable to mass data scraping (or could it be?) but I have observed Midjourney can style copy with high fidelity from a handful of never before seen images.

The creators, in all fairness, do not claim it is future proof.