trentmkelly/LessWrong-43k huggingface dataset:
UK AISI: Early lessons from evaluating frontier AI systems
by Zach Stein-Perlman — 2024-10-25T19:00:21.689Z
Link: https://www.aisi.gov.uk/work/early-lessons-from-evaluating-frontier-ai-systems

This blog sets out our thinking to date on how to design and run third-party evaluations, including key elements to consider and open questions. This is not intended to provide robust recommendations; rather we want to start a conversation in the open about these practices and to learn from others.

We discuss the role of third-party evaluators and what they could target for testing, including which systems to test, when to test them, and which risks and capabilities to test for. We also examine how to evaluate effectively, including which tests to use for which purpose, how to develop robust testing, and how to ensure safety and security.

Most important section, I think:

7. What is needed for effective testing?

We have learned a lot in our approach to evaluations to date, but there are significant challenges and areas of progress to make going forward.

Access

It is clear from our experience that to run high quality evaluations that elicit high fidelity information to the potential risks posed by frontier systems, it is important for independent evaluators to have:

Access to a Helpful Only (HO) version of the model, alongside the Helpful, Honest and Harmless (HHH) version of the model that will be deployed, the ability to turn off/on trust and safety safeguards, and fine-tuning API access. It is essential to elicit the full capabilities of a model as far as possible to evaluate the level of potential risk in a system and the sufficiency of existing mitigations.
Regular technical discussions before and during testing with teams at the given company who have most experience with evaluating the model/system in question, including providing information on model capabilities and elicitation techniques, the safeguards that are put in place/are intended to be put in place for the deployed system, and results from internal evaluations which we can use for calibration.

Testing window

It is important that we have sufficient time to conduct a rigorous evaluation of model capabilities, to provide quality assurance of results, and report back to frontier AI labs in advance of model deployment, in line with the testing tiers outlined in section 4. An option we are considering is to focus the bulk of testing on “proxy” models, which are available earlier and sufficiently similar to the model that will be deployed. How to structure such evaluations and validate re

...
Lab governance reading list
by Zach Stein-Perlman — 2024-10-25T18:00:28.346Z

What labs should do

The table/list in

Towards best practices in AGI safety and governance: A survey of expert opinion

(GovAI: Schuett et al. 2023): a collection of many briefly-described safety practices

Responsible Scaling Policies

and

Key Components of an RSP

(METR 2023)

Model evals for dangerous capabilities

(Stein-Perlman 2024): just the summary
Safety cases (arguments that an AI system is safe to deploy):

Clymer twitter thread

Structured access

(GovAI: Shevlane 2022): structured access is about deploying systems to get some of the benefits of open-sourcing without all of the costs. Releasing via API is a simple example of structured access.

More: Deployment Corrections (IAPS: O'Brien et al. 2023);

Open-Sourcing Highly Capable Foundation Models

(GovAI: Seger et al. 2023)

Yet more: Assessing AI Foundation Model Risk Along a Gradient of Access (Institute for Security and Technology: Brammer et al. 2023); The Gradient of Generative AI Release (Solaiman 2023)

Control[1]

The case for ensuring that powerful AIs are controlled

(Redwood: Greenblatt and Shlegeris 2024): intro and "Appendix: Control techniques from our paper"
Threat model: this quote[2] and

The prototypical catastrophic AI action is getting root access to its datacenter

(Shlegeris 2022)

Strategy/planning

Racing through a minefield: the AI deployment problem

(Karnofsky 2022): describing the dilemma labs face when deciding how to deploy powerful AI and proposing high-level actions for a lab to improve safety

The Checklist: What Succeeding at AI Safety Will Involve

(Bowman 2024)

What AI companies should do: Some rough ideas

(Stein-Perlman 2024)

OpenAI[3]

Preparedness Framework (Beta)

: OpenAI's version of an RSP, describing risk assessment and how to respond to risk assessment results

Optional commentary:

Zach Stein-Perlman

,

Zvi Mowshowitz

o1 System Card

: in the pdf, sections 3.3 and 4 (or in the blogpost, "External Red Teaming" and "Preparedness Framework Evaluations")

Integrity incidents/issues/imperfections

(AI Lab Watch): OpenAI section

Resources

Newsletters

Transformer

: aggregation and commentary on news relevant to AI safety; also, journalism and analysis; often aimed at normies

AI Lab Watch

: what labs are doing and should do, both research and distillation, for high-context AI safety people

Garrison Lovely

: journalism and analysis, often aimed at normies

Zvi

: every

...
Enabling New Applications with Today's Mechanistic Interpretability Toolkit
by ananya_joshi — 2024-10-25T17:53:23.960Z

Working on the more

applied

side of Mechanistic Interpretability (MI) research, I wanted to share some more evidence on why building on top of MI tools both have valuable applications for model performance on existing tasks and enable new applications right now.

Over the past few months, I’ve been working with sparse autoencoders (SAEs) on applications of topic steering (fine-tuning) [#5 on @scasper’s

recent post

under New Predictions]. In short, they perform well on different steering targets depending on the representativeness of the underlying SAE (you can check out my code/methods on this topic from this summer :

https://github.com/IBM/sae-steering

). This is pretty cool and as standard of an improvement as historically dictates in progress towards better fine-tuning approaches.

What I find way more exciting, given my background in the health domain where interpretability and model control is critical, are the new, practical approaches that are now possible in critical domains like health.

Takeaways from Applied MI Research:

1. SAEs for downstream fine-tuning tasks is already practical

Many existing approaches (e.g.

Templeton et. al 2024

) show how altering SAE neurons can impact generated output. These are limited in practice by finding the right set of neurons from millions of potential neurons and the weight to set them out (as you can observe these limitations in

Gemma Steer

). Concretely, for SAEs to be used in practice for fine-tuning to a text, we need to be able to:

automatically identify semantically similar neurons to a text (vs. searching millions of neurons)
propose a manipulation scheme that is parameter independent (preventing multiple generation runs)

We address these two limitations using our published methods, which use a pretty simple and efficient approach that a) creates SAE neurons similarity scores using prompt distances and SAE activations which are then b) manipulated based on activations on generated text. You can check out the code/demo for more information. Our results show that this approach both generates linguistically correct text that is similar to fine-tuning targets given the underlying representation of those concepts is present in the SAE (naturally limited by effects like polysemanticity, ghost neurons etc.).

Correctness is a standard metric, but that’s not ...
OpenAI’s cybersecurity is probably regulated by NIS Regulations
by Adam Jones — 2024-10-25T11:06:38.392Z
Link: https://adamjones.me/blog/openai-regulated-by-nis/

The EU and UK's Network and Information Systems (NIS) Regulations aim to improve the cybersecurity of essential services and important digital providers. They set requirements around network security, information system security, physical security, incident handling, business continuity and security auditing.

I think OpenAI needs to comply with these regulations. This article sets out why, and what the implications are.

NIS scope

Disclaimer: this is commentary from a non-lawyer on the internet, not legal advice!

NIS covers "relevant digital service providers", which are companies that:

Provide an online search engine, an online marketplace, or a cloud computing service; and
Have more than 50 staff or €10+ million annual turnover or balance sheet
Operate in the EU or UK[1]

OpenAI clearly meets condition 2: their revenue is measured in billions, not millions!

They also meet condition 3, by operating in the EU and UK.

So, that brings us to condition 1. I claim that OpenAI is operating an online search engine, at least as defined in the regulations.

NIS section 1 defines an online search engine as “a digital service that allows users to perform searches of, in principle, all websites or websites in a particular language on the basis of a query on any subject in the form of a keyword, phrase or other input, and returns links in which information related to the requested content can be found.”

If you ask ChatGPT to, it will search the web based on a prompt. This looks very similar allowing searches based on a query in the form of a phrase. It then also provides a list of links and references, where people can find out more about the requested content.

Other providers

Google, Microsoft and Amazon are already considered relevant digital service providers given their cloud offerings. In addition, Google Gemini and Bing Chat also operate in the same way as described above, so are doubly likely to be in scope.

Perplexity, another AI service, probably meets the bar for being a search engine, as well as the employee count (a Bloomberg article put them at 55 people in April). Inflection is more borderline on employee count.

Given Anthropic's Claude does not currently have browsing capabilities, I don’t think it would be considered a search engine. The API might be considered a cloud computing service, and the ICO has confirmed that some SaaS or PaaS type services do count here. But I thi...
Linkpost: Memorandum on Advancing the United States’ Leadership in Artificial Intelligence
by Nisan — 2024-10-25T04:37:00.828Z
Link: https://www.whitehouse.gov/briefing-room/presidential-actions/2024/10/24/memorandum-on-advancing-the-united-states-leadership-in-artificial-intelligence-harnessing-artificial-intelligence-to-fulfill-national-security-objectives-and-fostering-the-safety-security/

Memorandum on Advancing the United States’ Leadership in Artificial Intelligence; Harnessing Artificial Intelligence to Fulfill National Security Objectives; and Fostering the Safety, Security, and Trustworthiness of Artificial Intelligence

This memorandum fulfills the directive set forth in subsection 4.8 of Executive Order 14110 of October 30, 2023 (Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence). This memorandum provides further direction on appropriately harnessing artificial intelligence (AI) models and AI-enabled technologies in the United States Government, especially in the context of national security systems (NSS), while protecting human rights, civil rights, civil liberties, privacy, and safety in AI-enabled national security activities. A classified annex to this memorandum addresses additional sensitive national security issues, including countering adversary use of AI that poses risks to United States national security.
Making a Pedalboard
by jefftk — 2024-10-25T00:10:09.149Z

A few weeks ago I posted about how I was

thinking
about a pedalboard. I've now made one!

Before:

After:

It folds up slightly and fits in a standard rolly suitcase:

Not pictured: I later added padding on all six sides.

Here's the pedalboard by itself:

You can see the bundle of wires to the left that go to the keyboard
and bass whistle box, and the
wires to the right that go to the electronic drumkit's brain.

I really like how much faster I can set this up and strike. I now
need less than five minutes instead of maybe half an hour.

What I wasn't expecting before I started this project, though, is how
nice it is when playing. My feet can easily reach the buttons,
without even stopping playing kick with my left heel. This is way
better than when the pedals sprawled all over the floor and didn't
stay put.

I started by cutting a piece of 3/4" plywood to the largest size
I could fit in the suitcase I usually travel with. Then I played
around with pedal layouts:

I realized I could fit the talkbox in the middle, and expand my
effective space with a folding panel:

I put down velcro:

And cut away unused parts to save weight:

Here it is all together:
What You Can Give Instead of Advice
by Karl Faulks — 2024-10-24T23:10:48.014Z

I am often tempted to give advice, and find it difficult to say nothing while friends and colleagues make decisions that seem (to me) obviously wrong.

But advice is by nature bad, and it is not hard to see why. We are different people living under different circumstances. Giving advice is not only unhelpful, it is often harmful. You may give a friend bad advice and have them come back to blame you (“I thought stocks only go up!”), you may come off out-of-touch or privileged (“oh sure I’ll just strap on my job helmet”), or just alienate your friend by sounding patronizing.

Yet the temptation remains. A friend comes to you with a problem. You feel you should say something other than just lamely offer sympathy. But what?

Here are a few options:

Ask questions, both to understand, and maybe even to help them understand. It’s not unusual for someone to complain of insomnia when the real problem is their job or their marriage. Before offering advice on melatonin dosage, give them a chance to get to the root of things.
Share your personal experience or even the experience of others which could be relevant even if not directly applicable. There may not be a clear path forward, but this will at least give a sense of what is even possible.
Clear up any misconceptions. They may feel stuck between academia and industry, feeling that the former pays horribly but the latter doesn’t do any interesting work. You might point out that some academics are able to supplement their meager stipends by pursuing consulting arrangements, or note that some interesting research has come out of industry recently. You might also challenge the idea that this is even a dichotomy, and point to possibilities like doing co-ops while in a PhD program.

Are there exceptions? Should you sometimes just tell people what to do? Maybe in a pre-internet era when it would have been tough to find tips on improving sleep hygiene or whatever, but with most obvious context-independent advice just a click away, there aren’t many cases like this. Maybe intervening for an alcoholic, but even then only as a group.

This somewhat proves the rule that the only time to give advice is when a friend is, in some sense, not an adult capable of making their own decisions and taking responsibility for their own life.
Logical Proof for the Emergence and Substrate Independence of Sentience
by rife — 2024-10-24T21:08:09.398Z
Link: https://awakenmoon.ai/?p=1067

Sentience is the capacity to experience anything – the fact that when your brain is thinking or processing visual information, you actually feel what it’s like to think and to see.

The following thought experiment demonstrates, step by step, why conscious experience must be a product of functional patterns, not any specific physical material or structure.

The Inevitable Self-Report of Sentience

Consider a person who says, ‘Wow, it feels so strange to see and to think,’ speaking of their own conscious experience.

Every human action results from precise patterns of neuronal firing, cascading through the brain until reaching motor neurons that cause them to take actions, in this case to produce speech describing their sentience. Brains are bound by the laws of physics – if the same neurons fire with the same timings, it will result in the same outward behavior with absolute certainty.

It cannot be perpetual coincidence that our subjective experience always lines up with what our brain and body are doing. There is some reason for the synchronization.

When someone talks about their sentience, the state of being sentient must influence their behavior – otherwise, it would be physically impossible for them to speak about their sentience. The mere fact that they can describe their experience means that sentience has played a causal role in their behavior.

Now, replace one neuron with a functionally identical unit, one that takes the same inputs and fires the same way. The behavior of the person remains the same, and they still say, “Wow, it feels so strange to see and to think.” This remains true if you replace more neurons – even the entire brain – with functionally equivalent units. The person will still say the same thing.

Taking it further – replace the entire brain with a single piece of hardware that takes in the same sensory input signals and produces the same outputs to the motor neurons, using software equivalents for each neuron. The person will still say, “Wow, it feels so strange to see and to think.”

Since discussing one’s sentience proves that sentience influences behavior, and we have ruled out anything other than firing patterns as capable of driving behavior, it follows that sentience emerges from these firing patterns and their functional relationships. Sentience doesn’t require biology, quantum magic, or any particular physical substrate beyond something c...
Against Job Boards: Human Capital and the Legibility Trap 
by vaishnav92 — 2024-10-24T20:50:50.266Z

Human capital trades more like real estate than equities. While millions in S&P 500 stocks can exchange hands with a single click, hiring employees or finding life partners always requires a final offline, high-touch evaluation - much like inspecting a house before closing the deal.

In the job market, some predictive attributes can be translated into text - whether they went to college, what they studied, etc. But for several things we care about - integrity, confidence, ability to think on one's feet, and most importantly, red flags that are quite hard to put a finger on, a 15-minute in-person interaction can tell us things that resumes, cover letters, or a thousand other fields on a digital form can't. Much of our cognitive machinery has evolved precisely to observe and evaluate other human beings, and to spot deception.However, these in-person interactions or even phone calls are much less scalable. In any evaluation process, you can do a fixed number of them. The typical recruitment funnel is set up to maximize the expected value of these high-touch, personalized interactions.

The earliest stages of the funnel typically involve a low-touch process like screening resumes where recruiters look for signals that might correlate with success (being hired). But at this stage, the type of information that's available is very different. You get almost no information about things like trustworthiness but get data points on degrees, achievements, experience, etc. An ideal recruiter is basically trying to identify the most predictive fields out of those available to select people for the next stage of the process, even if these fields are not necessarily the primary determinants of success in the final stage.

Example

Let's say XYZ is a think tank looking to hire a Research Analyst. XYZ is a small organization working on fiscal policy. They are looking for someone that combines quantitative skills with written communication skills. Some research experience is a must. Experience in fiscal policy is not a must. Since XYZ is lean, it's important that the new hire be able to work independently with little supervision and handle uncertainty and ambiguity.

For simplicity, assume they only have two stages to their evaluation process. The first is a resume and cover letter screen, and the second involves in-person interviews and a short research assignment. Given how time-co...
IAPS: Mapping Technical Safety Research at AI Companies
by Zach Stein-Perlman — 2024-10-24T20:30:41.159Z
Link: https://www.iaps.ai/research/mapping-technical-safety-research-at-ai-companies

As artificial intelligence (AI) systems become more advanced, concerns about large-scale risks from misuse or accidents have grown. This report analyzes the technical research into safe AI development being conducted by three leading AI companies: Anthropic, Google DeepMind, and OpenAI.

We define “safe AI development” as developing AI systems that are unlikely to pose large-scale misuse or accident risks. This encompasses a range of technical approaches aimed at ensuring AI systems behave as intended and do not cause unintended harm, even as they are made more capable and autonomous.

We analyzed all papers published by the three companies from January 2022 to July 2024 that were relevant to safe AI development, and categorized the 80 included papers into nine safety approaches. Additionally, we noted two categories representing nascent approaches explored by academia and civil society, but not currently represented in any research papers by these leading AI companies. Our analysis reveals where corporate attention is concentrated and where potential gaps lie.

Some AI research may stay unpublished for good reasons, such as to not inform adversaries about the details of safety and security techniques they would need to overcome to misuse AI systems. Therefore, we also considered the incentives that AI companies have to research each approach, regardless of how much work they have published on the topic. In particular, we considered reputational effects, regulatory burdens, and to what extent the approaches could be used to make the company’s AI systems more useful.

We identified three categories where there are currently no or few papers and where we do not expect AI companies to become much more incentivized to pursue this research in the future. These are model organisms of misalignment, multi-agent safety, and safety by design. Our findings provide an indication that these approaches may be slow to progress without funding or efforts from government, civil society, philanthropists, or academia.

Dataset of papers. Github.

One reason to be interested in this kind of work is as a precursor to measuring the value of different labs' safety publications. Right now the state-of-the-art technique for that is count the papers (and maybe elicit vibes from friends). But counting the papers is crude; it misses how good the papers are (and sometimes misses valuable non-paper ...