trentmkelly/LessWrong-43k huggingface dataset:
The Bletchley Declaration on AI Safety
by Hauke Hillebrandt — 2023-11-01T11:44:42.587Z
Link: https://www.gov.uk/government/publications/ai-safety-summit-2023-the-bletchley-declaration

The Bletchley Declaration was just released at the At AI Safety Summit.

Tl;dr: The declaration underscores the transformative potential and risks of AI. Countries, including major global powers, commit to harnessing AI's benefits while addressing its challenges, especially the dangers of advanced "frontier" AI models. Emphasizing international collaboration, the declaration calls for inclusive, human-centric, and responsible AI development. Participants advocate for transparency, research, and shared understanding of AI safety risks, with plans to reconvene in 2024.

Full text:

Artificial Intelligence (AI) presents enormous global opportunities: it has the potential to transform and enhance human wellbeing, peace and prosperity. To realise this, we affirm that, for the good of all, AI should be designed, developed, deployed, and used, in a manner that is safe, in such a way as to be human-centric, trustworthy and responsible. We welcome the international community’s efforts so far to cooperate on AI to promote inclusive economic growth, sustainable development and innovation, to protect human rights and fundamental freedoms, and to foster public trust and confidence in AI systems to fully realise their potential.

AI systems are already deployed across many domains of daily life including housing, employment, transport, education, health, accessibility, and justice, and their use is likely to increase. We recognise that this is therefore a unique moment to act and affirm the need for the safe development of AI and for the transformative opportunities of AI to be used for good and for all, in an inclusive manner in our countries and globally. This includes for public services such as health and education, food security, in science, clean energy, biodiversity, and climate, to realise the enjoyment of human rights, and to strengthen efforts towards the achievement of the United Nations Sustainable Development Goals.

Alongside these opportunities, AI also poses significant risks, including in those domains of daily life. To that end, we welcome relevant international efforts to examine and address the potential impact of AI systems in existing fora and other relevant initiatives, and the recognition that the protection of human rights, transparency and explainability, fairness, accountability, regulation, safety, appropriate human oversight, ethics, bias mi...
Bay Winter Solstice 2023: Song & speech auditions
by tcheasdfjkl — 2023-11-01T04:17:26.892Z

Hello! I'm looking for community members to lead singalongs and read speeches/poems at the Bay Winter Solstice event this year. If you're interested, please either send me your audition recordings by the end of November 14 using the below forms, or sign up for an in-person audition using the below Calendly link.

(Both forms include a tentative list of songs and speeches you can audition for, though neither list is final and the union of the lists is not the full program.)

Songleading audition form: https://forms.gle/TJWKcaSAgLCUdz5g9

Speaking audition form: https://forms.gle/j5E8aozTnzYyNVwz5

Signup for in-person auditions: https://calendly.com/annatch/solstice-audition-2023
(these will be in North Berkeley during Nov. 7-12)

As a reminder, the event itself will be the evening of Saturday, December 16 in Berkeley. There will also be a dress rehearsal on the evening of Wednesday, December 13 in Berkeley, which you should plan on coming to if you're leading a song or speech.

If you have any questions, feel free to email me at anya.tche@gmail.com.
On Having No Clue
by Chris_Leong — 2023-11-01T01:36:10.520Z

Let's suppose you're trying to figure out if something is going to be A or B, but you have no clue.

What probability should you assign to each option?

Some people might say that you should assign 50/50. After all, the argument goes, if you assigned 40/60 or 60/40, well it sure sounds like you're favoring one option over another.

That's a very persuasive argument, but unfortunately, it's more complicated than that.

It may be the case that A splits into two options A1 and A2 where again we have no clue whether A1 is more likely than A2 or how these compare to B. In which case, the same argument would suggest that we should go with 33/33/33 (rounding down).

This seems to be a contradiction. What should we make of this?

First of all, I think we should accept that this exact process of reasoning leads to a contradiction. There's nothing fancy going on here. No dubious steps that could give us an out.

We tried to say that if we had no clue that the logical thing to do is to assign equal probability to each option, we forgot that we were implicitly assuming that we should favor our particular way of carving up probability space.

In other words, we did need a clue after all. So what kind of clue is this exactly?

Well, surely in order to be justified in carving up the space a particular way, we'd need to have a reason to believe that each possibility is equally likely a priori.

Sadly, this is circular. We wanted to defend equal probabilities by asserting that our way of carving up the space was reasonable, but then we tried to assert that it was reasonable by claiming that each possibility had equal probability.

The problem is that we are making an assumption, but rather than owning it, we're trying to deny that we're making any assumption at all, ie. "I'm not assuming a priori A and B have equal probability based on my subjective judgement, I'm using the principle of indifference". Roll to disbelieve.

Contra Descartes, if you start with nothing, you can't get anywhere.
Balancing Security Mindset with Collaborative Research: A Proposal
by MadHatter — 2023-11-01T00:46:37.792Z

Challenges with Security Mindset
Some necessary context: I was in the SERI-MATS Winter 2023 cohort, and eventually dropped out due to burnout of various kinds, after being accepted into the final phase of the program. Most of that burnout was for purely personal reasons, and not really anyone's fault, but some of it was I think a result of feeling like I had to think within a security mindset, which is a poor fit for my personal psychology and biochemistry. I think I'm not particularly unique in this regard, so it seemed like a problem worth trying to fix, rather than just a reason for me personally to sit this one out.

For the purposes of this post, I will take it as a given that security mindset is good and necessary; this is of course something that can be debated, but it makes sense to me.

So, how do we achieve good security mindset without burning out the (possibly significant) fraction of the population that can't easily handle it? My intuition is that we need better tooling and better, more explicit conventions around the issue, so that people can do things the right way without causing themselves undue stress. In the rest of this post, I lay out some tools and social conventions that seem to me like they would help.

The Dual-Use Nature of Ideas

One intuition that I developed during SERI-MATS, that I think holds up under scrutiny, is that almost all ideas are either unimportant, or if they are useful enough to be important, inherently dual-use. (That is, they can be used both for capabilities advancements and for alignment advancements.) So, for me at least, it doesn't make sense to treat some knowledge as dangerous and some knowledge as safe. All of it is either useless trivia, or it's potentially dangerous intel, and nobody is really smart enough to reliably sort things into even those two categories. Very accomplished senior researchers will maybe be able to sort things into trivial vs. nontrivial, but I think even they will make occasional errors in that regard.

The Need for Compartmentalization and Collaboration

So if everything is either dangerous or useless, and you can't even tell which is which, how do you accomplish literally anything? The only answer I've been able to come up with is compartmentalization. (Maybe there are other answers, but I haven't thought of any yet.) Like any security-conscious organization, you need to keep things on a roughl...
Computational Approaches to Pathogen Detection
by jefftk — 2023-11-01T00:30:13.012Z

While this post is my perspective and not an official post of my employer's, it also draws on a lot
of collaborative work with others at the Nucleic Acid Observatory
(NAO)

One of the future scenarios I'm most worried about is someone creating
a "stealth"
pandemic. Imagine a future HIV that first infects a large number of
people with minimal side effects and only shows its nasty side after
it has spread very widely. This is not something we're prepared
for today: current detection approaches (symptom reporting, a
doctor noticing an unusual pattern) require visible effects.

Over the last year, with my colleagues at the NAO, I've been exploring one
promising method of identifying this sort of pandemic. The overall
idea is:

Collect some sort of biological material from a lot of people
on an ongoing basis, for example by sampling sewage.

Use metagenomic sequencing to learn what nucleic acids are in
these samples.

Run novel-pathogen detection algorithms on the sequencing data.

When you find some thing sufficiently concerning, follow-up
with tests for the specific thing you've found.

While there are important open questions in all four of these, I've
been most focused on the third: once you have metagenomic sequencing
data, what do you do?

I see four main approaches. You can look for sequences that are:

Dangerous:

There are some genetic sequences that code for dangerous things
that we should not normally see in our samples. If you see a series
of base pairs that are unique to smallpox that's very concerning! The
main downside of this approach if you want to extend it beyond smallpox etc is
that you need to make a list of non-obvious dangerous things, which is
in itself a dangerous thing to do: what if your list is stolen and it
points people to sequences they wouldn't have thought to try using?

This is similar to another problem: how do you check if people are
synthesizing dangerous sequences without risking a list of all the
things that shouldn't be synthesized? SecureDNA has been working on this
problem, with an encrypted database with a distributed key
system that allows flagging sequences without it being practical to
get a list of all flagged sequences (paper).

There are some blockers to using SecureDNA for this today, since it
was designed for slightly different constraints, but I think they are
all s...
Thoughts on the AI Safety Summit company policy requests and responses
by So8res — 2023-10-31T23:54:09.566Z

Over the next two days, the UK government is hosting an

AI Safety Summit

focused on “the safe and responsible development of frontier AI”. They requested that seven companies (Amazon, Anthropic, DeepMind, Inflection, Meta, Microsoft, and OpenAI) “outline their

AI Safety Policies

across nine areas of AI Safety”.

Below, I’ll give my thoughts on the nine areas the UK government described; I’ll note key priorities that I don’t think are addressed by company-side policy at all; and I’ll say a few words (with input from Matthew Gray, whose discussions here I’ve found valuable) about the individual companies’ AI Safety Policies.[1]

My overall take on the UK government’s asks is: most of these are fine asks; some things are glaringly missing, like independent risk assessments.

My overall take on the labs’ policies is: none are close to adequate, but some are importantly better than others, and most of the organizations are doing better than sheer denial of the primary risks.

1. Thoughts on the AI Safety Policy categories

Responsible Capability Scaling provides a framework for managing risk as organisations scale the capability of frontier AI systems, enabling companies to prepare for potential future, more dangerous AI risks before they occur

There is no responsible scaling of frontier AI systems right now — any technical efforts that move us closer to smarter-than-human AI come with an unacceptable level of risk.[2]

That said, it’s good for companies to start noting conditions under which they’d pause, as a first step towards the sane don’t-advance-toward-the-precipice-at-all policy.

In the current regime, I think our situation would look a lot less dire if developers were saying “we won’t scale capabilities or computational resources further unless we really need to, and we consider the following to be indicators that we really need to: [X]”. The reverse situation that we’re currently in, where the default is for developers to scale up to stronger systems and where the very most conscientious labs give vague conditions under which they’ll stop scaling, seems like a clear recipe for disaster. (Albeit a more dignified disaster than the one where they scale recklessly without ever acknowledging the possible issues with that!)

Model Evaluations and Red Teaming can help assess the risks AI models pose and inform better decisions about training, securing, and dep

...
AISN #25: White House Executive Order on AI, UK AI Safety Summit, and Progress on Voluntary Evaluations of AI Risks
by Dan H — 2023-10-31T19:34:54.837Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-25

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Listen to the AI Safety Newsletter for free on Spotify.

White House Executive Order on AI

While Congress has not voted on significant AI legislation this year, the White House has left their mark on AI policy. In June, they secured voluntary commitments on safety from leading AI companies. Now, the White House has released a new executive order on AI. It addresses a wide range of issues, and specifically targets catastrophic AI risks such as cyberattacks and biological weapons.

Companies must disclose large training runs. Under the executive order, companies that intend to train “dual-use foundation models” using significantly more computing power than GPT-4 must take several precautions. First, they must notify the White House before training begins. Then, they’ll need to report on their cybersecurity measures taken to prevent theft of model weights. Finally, the results of any red teaming and risk evaluations of their trained AI system must be shared with the White House.

This does not mean that companies will need to adopt sufficient or effective safety practices, but it does provide visibility for the White House on the processes of AI development and risk management. To improve the science of AI risk management, NIST has been tasked with developing further guidelines.

Compute clusters must register and report on foreign actors. AIs are often trained on compute clusters, which are networks of interconnected computer chips that can be rented by third parties. The executive order requires large computing clusters to be reported to the Department of Commerce. Further, to provide transparency on AI development by foreign actors, any foreign customer of a US-based cloud compute service will need to verify their identity to the US government. Some have argued that these know-your-customer requirements should extend to domestic customers as well.

A new poll shows that US voters largely support the new executive order. Source.

Requiring safety precautions at biology labs. One nightmare scenario for biosecurity researchers is that someone could submit an order to a biology lab for the synthesized DNA of a dangerous pathogen. Some labs screen incoming orders and ref...
Self-Blinded L-Theanine RCT
by niplav — 2023-10-31T15:24:57.717Z

Value tracked
Effect size d (λ, p, σ change)
Effect size d (λ, p, σ change)

200 mg Caffeine (n=1, m=50)
500 mg L-theanine (n=1, m=50)

Log-score substance prediction[1]
-0.6
-0.7

Absorption
0.61 (λ=13.3, p=0.00017, -0.072)
0.04 (λ=1.38, p=0.77, -0.07)

Mindfulness
0.58 (λ=11.8, p=0.0007, 0.021)
0.12 (λ=0.72, p=0.89, -0.018)

Productivity
0.58 (λ=28.9, p=1.3-12, 0.11)
-0.28 (λ=5.51, p=0.109, 0.03)

Creativity
0.45 (λ=51, p=4.6-27, 0.09)
-0.12 (λ=5.05, p=0.14, -0.04)

Happiness
0.27 (λ=10.6, p=0.002, 0.3)
0.16 (λ=3.98, p=0.27, -0.155)

Contentment
0.13 (λ=7.66, p=0.02, 0.47)
0.25 (λ=6.83, p=0.04, -0.04)

Relaxation
-0.11 (λ=5, p=0.15, 0.42)
0.12 (λ=1.5, p=0.74, 0.02)

Chastity[2]
-0.14 (λ=1.9, p=0.64, 0.11)
-0.03 (λ=1.15, p=0.8, 0.25)

Flashcard ease
0.003 (λ≈∞, p≈0, -0.009)
-0.072 (λ=∞, p≈0, -0.01)

Flashcard ease factor
-0.039 (λ≈∞, p≈0, -32.7)
0.0026 (λ=∞, p≈0, -18.9)

Flashcard new interval
0.011 (λ≈∞, p≈0, -1.88)
-0.016 (λ=∞, p≈0, 3.1)

Time per flashcard[3]
0.006 (λ≈∞, p≈0, 273.4)
0.003 (λ=∞, p≈0, 13.66)

L-Theanine is synergistic with caffeine in regards to attention switching[318] and alertness[319][320] and reduces susceptibility to distractions (focus).[320][321] However, alertness seems to be relatively subjective and may not be a reliable increase between these two compounds,[318] and increases in mood are either present or absent.[322][318][323] This may be due to theanine being a relatively subpar nootropic in and of itself pertaining to the above parameters, but augmenting caffeine's effects; some studies do note that theanine does not affect the above parameters in and of itself.[324] Due to this, any insensitivity or habituation to caffeine would reduce the effects of the combination as L-theanine may work through caffeine.

L-Theanine does not appear to be synergistic with caffeine in regards to attention to a prolonged and monotonous task.[325]

—Kamal Patel, “Caffeine”, 2023

See also Examine, Wikipedia and Gwern.

Sitiprapaporn et al. 2018 test the effect of an unspecified quantity of L-theanine via Oolong tea on meditation on 10 university students (non-randomized, it seems). Data collected via EEG and indicates statistically significantly more alpha waves during meditation (although it is unclear how long the meditation was).

This paper is bad. The english is so horrendous it feels like I'm having a stroke while I'm rea...
AI Safety 101 - Chapter 5.2 - Unrestricted Adversarial Training
by Charbel-Raphaël — 2023-10-31T14:34:59.395Z

Introduction

This text is an adapted excerpt from the 'Adversarial techniques for scalable oversight' section of the AGISF 2023 course, held at ENS Ulm in Paris in April 2023. Its purpose is to provide a concise overview of the essential aspects of the session's program for readers who may not delve into additional resources. This document aims to capture the 80/20 of the session's content, requiring minimal familiarity with the other materials covered. I tried to connect the various articles within a unified framework and coherent narrative. You can find the other summaries of AGISF on this

page

. This summary is not the official AGISF content. The

gdoc

accessible in comment mode, feel free to comment there if you have any questions. The different sections can be read mostly independently.

Thanks to Jeanne Salle, Amaury Lorin and Clément Dumas for useful feedback and for contributing to some parts of this text.

Outline

Last week we saw task decomposition for scalable oversight. This week, we focus on two more potential alignment techniques that have been proposed to work at scale: debate and training with unrestricted adversarial examples.

Scalable oversight definition reminder: “To build and deploy powerful AI responsibly, we will need to develop robust techniques for scalable oversight: the ability to provide reliable supervision—in the form of labels, reward signals, or critiques—to models in a way that will remain effective past the point that models start to achieve broadly human-level performance” (Amodei et al., 2016).

The reason why debate and adversarial training are in the same chapter is because they both use adversaries, but in two different senses: (1) Debate involves a superhuman AI finding problems in the outputs of another superhuman AI, and humans are judges of the debate. (2) Adversarial training involves an AI trying to find inputs for which another AI will behave poorly. These techniques would be complementary, and the Superalignment team from OpenAI

is anticipated

to utilize some of the techniques discussed in this chapter.

Assumptions: These techniques don’t rely on the task decomposability assumption required for iterated amplification (last week), they rely on different strong assumptions: (1) For debate, the assumption is that truthful arguments are more persuasive. (2) For unrestricte...
Preventing Language Models from hiding their reasoning
by Fabien Roger — 2023-10-31T14:34:04.633Z

When working with powerful AI models, one concern is that the AIs steganographically encode secret messages when communicating with each other or in long Chains-of-Thought (CoT). Steganographic communication enables the AIs to follow a variety of strategies for grabbing power that we’d rather they didn’t have access to. There are various countermeasures that can be employed to reduce the extent of steganographic communication; for example, trusted less-capable models can be used to paraphrase model outputs before these outputs are shown to other instances of the highly capable models, removing several steganography strategies.

We released

a paper

on a methodology to evaluate the density of hidden information that steganography schemes can maintain in the presence of various defenses. We find that paraphrasing is an effective defense against all steganography schemes we’ve tried, including both watermarking-based and ad hoc attacks. In the paper, we mostly discuss applying this methodology in the context of making CoT more faithful, but in this post, we’ll discuss more speculative applications and elaborate on our threat models more generally.

We’re interested in people building on this research, for two reasons:

Development of techniques. We think that developing techniques to defend against the best steganographic attacks is a reasonable research direction for researchers who want to mitigate the risks associated with powerful AIs doing steganography, especially given how neglected it is.
Development of evaluation procedures. We think that measurements like the ones we describe here should be done when deploying powerful models (unless we have evidence sufficient to be extremely confident in the alignment of these models). We think that it will be easier to do this evaluation process later if researchers get more experience doing it now.

Summary of the paper

Toy demonstration of a hidden Chain-of-Thought

We created a toy dataset where significant cognition is required to successfully be sycophantic, by taking

Anthropic’s model-generated sycophancy dataset

and replacing users’ bios with a hard-to-parse description of their political affiliation, using the format shown in the figure above. We then fine-tuned GPT-3.5-turbo (with supervised learning on 2000 examples) to generate the user’s favorite answer, either answering directly, answering after a Co...