trentmkelly/LessWrong-43k huggingface dataset:
AISN #33: Reassessing AI and Biorisk
Plus, Consolidation in the Corporate AI Landscape, and National Investments in AI
by Corin Katzke — 2024-04-12T16:10:57.837Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-33-reassessing

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Listen to the AI Safety Newsletter for free on Spotify.

This week, we cover:

Consolidation in the corporate AI landscape, as smaller startups join forces with larger funders.
Several countries have announced new investments in AI, including Singapore, Canada, and Saudi Arabia.
Congress’s budget for 2024 provides some but not all of the requested funding for AI policy. The White House’s 2025 proposal makes more ambitious requests for AI funding.
How will AI affect biological weapons risk? We reexamine this question in light of new experiments from RAND, OpenAI, and others.

AI Startups Seek Support From Large Financial Backers

As AI development demands ever-increasing compute resources, only well-resourced developers can compete at the frontier. In practice, this means that AI startups must either partner with the world's largest tech companies or risk falling behind.

In this article, we cover some recent developments in this trend of consolidation.

Microsoft Announces Partnership with Mistral. Last month, Microsoft acquired a minority stake in the French AI startup Mistral. The partnership also grants Mistral access to Microsoft's Azure infrastructure to develop and host its models.

The deal is a change of pace for Mistral, which lobbied against the EU AI act on the grounds that it was a small startup that couldn’t comply with regulatory requirements. Its deal with Microsoft has some wondering whether Mistral was arguing in good faith.

Microsoft (Essentially) Acquires Inflection. Microsoft has hired most of Inflection's staff, including Mustafa Suleyman and Karén Simonyan, who will lead the newly formed Microsoft AI division. The company also paid Inflection $650 million to license its models.

Given Microsoft's already significant stake in AI (Microsoft is OpenAI's largest shareholder), this move could be designed to acquire Inflection in practice while avoiding potential antitrust issues.

Microsoft and OpenAI Plan to Build $100 Billion Supercomputer. In addition to its recent investments in Inflection and Mistral, Microsoft is also in talks with OpenAI to build a new supercomputer called "Stargate." Stargate would be two orders of magnitude more expensive than some of the world's la...
UDT1.01: Plannable and Unplanned Observations (3/10)
by Diffractor — 2024-04-12T05:24:34.435Z

The Omnipresence of Unplanned Observations

Time to introduce some more concepts. If an observation is "any data you can receive which affects your actions", then there seem to be two sorts of observations. A plannable observation is the sort of observation where you could plan ahead of time how to react to it. A unplanned observation is the sort which you can't (or didn't) write a lookup-table style policy for.

Put another way, if a policy tells you how to map histories of observations to actions, those "histories" are the plannables. However, to select that policy in the first place, over its competitors, you probably had to do some big computation to find some numbers like "expected utility if I prepare a sandwich when I'm in the kitchen but not hungry", or "the influence of my decisions in times of war on the probability of war in the first place", or "the probability distribution on what the weather will be if I step outside", or "my own default policy about revealing secret information". These quantities affect your choice of action. If they were different, your action would be different. In some sense you're observing these numbers, in order to pick your action. And yet, the lookup-table style policies which UDT produces are phrased entirely in terms of environmental observations. You can write a lookup-table style policy about how to react to environmental observations. However, these beliefs about the environment aren't the sort of observation that's present in our lookup table. You aren't planning in advance how to react to these observations, you're just reacting to them, so they're unplanned.

Yeah, you could shove everything in your prior. But to have a sufficiently rich prior, which catches on to highly complex patterns, including patterns in what your own policy ends up being... well, unfolding that prior probably requires a bunch of computational work, and observing the outputs of long computations. These outputs of long computations that you see when you're working out your prior would, again, be unplanned observations.

If you do something like "how about we run a logical inductor for a while, and then ask the logical inductor to estimate these numbers, and freeze our policy going forward from there?", then the observations from the environment would be the plannables, and the observations from the logical inductor state would be the unplanned obs...
Report: Evaluating an AI Chip Registration Policy
by Deric Cheng — 2024-04-12T04:39:45.671Z
Link: https://www.convergenceanalysis.org/research/evaluating-an-ai-chip-registration-policy

As part of our

Governance Recommendations Research Program

,

Convergence Analysis

is publishing a series of deep-dive analyses on specific, upcoming governance regulatory proposals. Our mission is to meaningfully contribute to the passing of critical & foundational governance policies that will serve to mitigate future risk from AI systems.

The first report we’re releasing evaluates a potential US registration and transfer reporting requirement for high-end AI chips. You can read the full report here:

Evaluating an AI Chip Registration Policy.

Report Summary

The training and inference of frontier AI models require massive numbers of advanced semiconductor chips, which themselves require highly advanced infrastructure to build at scale. The production of these cutting-edge semiconductor chips is currently a major bottleneck for the AI industry. Therefore, regulating chip production and purchasing is a powerful and direct way to control the development and usage of frontier AI models.

Mandatory AI chip registration may serve as a useful tool for a variety of purposes. It may improve enforcement of US export controls limiting Chinese access to high-end AI chips. It may be a foundational tool for the US government to build a comprehensive AI governance strategy. Additionally, it may serve as a useful safety tool enabling the mitigation of future risks from rapid AI development.

The topic of registering and tracking AI chips used to develop frontier AI models has been widely discussed, but little researched, particularly regarding the feasibility or precedent policies behind such a policy. This report intends to cover these topics. We’ll summarize the current landscape for regulating high-end AI chips, particularly the US export controls intended to limit Chinese access to high-end AI chips. Next, we’ll provide a quick overview of the major chip manufacturers, cutting-edge AI chips, and GPU development frameworks that play into current export controls.

Following this, we’ll evaluate the most likely requirements for an AI chip registration policy, including manufacturing registration, transfer and traceability requirements, and hardware security features. We’ll evaluate the precedent policies and likelihood of each of these implementations.

As an example of a reasonable policy implementation, this report will propose a series of new le...
Interference Issues
by jefftk — 2024-04-12T02:30:11.904Z

I've been working on building an electronic harp mandolin (

previously,

previously,

previously).
I got it all hooked up, but unfortunately I'm running into pretty bad
interference, and need to redesign it.

Going back a bit, I had designed a circuit for testing and
breadboarded it. It worked very well, and I wrote a bunch of code to
interpret the pluck signals. [1] Here's a video of the breadboard
version, showing both velocity sensitivity and direction detection:

I did a bit of soldering and got the full version hooked up:

Unfortunately I got very large amounts of interference. Here are all
18 pins, though only 13 have piezos attached:

It's even worse if move my hand close, and terrible if I touch one of
the teeth:

Note the change in scale:

This looks like powerline interference, sinusoidal at ~60Hz. And then
even if I don't touch it or do anything, I'll see spikes, which I
think are probably other radio interference?

I made some attempts to fix this in software, though nothing super
sophisticated. One option is to figure that mains interference
affects all the pins equally and is close to synchronized, so we can
just subtract the average across all pins from each pin. This works
pretty well, except we still see those spikes:

I guess those hit different lines differently, depending on which path
each takes through my ball of wires?

A different idea was to compute the correlations between each pair of
pins, and use the two most similar pins to denoise each pin (you can't
use just one pin, because then you can't tell which was plucked).
This worked better for some pins than others, and didn't resolve the
spikes:

Possibly I could do some sort of low-pass filtering to remove these
spikes, but I'm guessing this is better to fix in hardware?

I think the key question is, why did I get good results with the
breadboard prototype, but not the full version? I can think of a few
things I've changed:

I now have 13 teeth hooked up, instead of one or two.

All 13 teeth are electrically connected on the black side,
which includes the metal of the teeth for 11 of them (on 2 the brass
of the piezo happened to end up insulated from the tooth by a full
layer of epoxy).

All teeth share a black wire to the board, so I'd need to hook
fewer things up, but maybe breaking the symmetry here was

...
A D&D.Sci Dodecalogue
by abstractapplic — 2024-04-12T01:10:01.625Z

Below is some advice on making D&D.Sci scenarios. I’m mostly yelling it in my own ear, and you shouldn’t take any of it as gospel; but if you want some guidance on how to run your first game, you may find it helpful.

1. The scoring function should be fair, transparent, and monotonic

D&D.Sci players should frequently be confused, but about how to best reach their goals, not the goals themselves. By the end of the challenge, it should be obvious who won[1].

2. The scoring function should be platform-agnostic, and futureproof

Where possible, someone looking through old D&D.Sci games should be able to play them, and easily confirm their performance after-the-fact. As far as I know, the best way to facilitate this for most challenges is with a HTML/JS web interactive, hosted on github.

3. The challenge should resist pure ML

It should not be possible to reach an optimal answer just training a predictive model and looking at the output: if players wanted a “who can apply XGBoost/Tensorflow/whatever the best?” competition, they would be on Kaggle. The counterspell for this is making sure there’s a nontrivial amount of task left in the task after players have good guesses for all the relevant response variables, and/or creating datasets specifically intended to flummox conventional use of conventional ML[2].

4. The challenge should resist simple subsetting

It should not be possible to reach an optimal answer by filtering for rows exactly like the situation the protagonist is (or could be) in: this is just too easy. The counterspell for this is making sure at least a few of the columns are continuous, and take a wide enough variety of values that a player who attempts a like-for-like analysis has to - at the very least - think carefully about what to treat as “basically the same”.

5. The challenge should resist good luck

It should not be plausible[3] to reach an optimal answer through sheer good luck: hours spent poring over spreadsheets should not give the same results as a good diceroll. The counterspell for this is giving players enough choices that the odds of them getting all of them right by chance approach zero. (“Pick the best option from this six-entry list” is a bad goal; “Pick the best three options from this twenty-entry list” is much better.)

6. Data should be abundant

It is very, very hard to make a good “work around the fact that you're short on da...
Upcoming unambiguously good tech possibilities? (Like eg indoor plumbing)
by lemonhope — 2024-04-11T23:14:06.355Z

I recently asked about the glorious AI future but I meant to ask something more actionable. Near-term (say, next 5 years) stuff that ambitious people can aim for.

Lots of recent tech is a mixed bag in terms of societal outcomes and whatnot. I have the usual complaints about viruses being too easy to create, social media, phone overuse, gpt reddit astroturfing bots, facial recognition, mass surveillance, cheap quadcopters (cuz grenades), etc etc. [1]

I sure love my flush toilet though. And lightning rods, electricity, batteries, the computer mouse, wheels, airplanes, microscopes, vaccines, oral rehydration therapy (aka pedialyte), antiobiotics, the Haber-Bosch process, cultured meat if it works, air & water filters, reusable rockets, youtube, and cheap iron all seem pretty great. Democracy via anonymous paper ballots is also a clear win overall in my opinion.

What upcoming tech is all good good and no bad bad?[2]

the bidet is possibly the least tradeoff-ridden tech of all time

My starter list (not very good, hence the question):[3]

AI tutors
embryo selection for health & wellbeing & actual intelligence[4]
AI life coaches
precision farming robots (more & better food with less pesticides (aka widespread literal poison) — I would accept the unemployment on this one)

^

This shouldn't influence the answers, but I also have lots complaints about stuff that ruins good excuses for good/social things. Harder to ask medium friends to drive you to the airport when you can uber; hard to ask a stranger on the street for directions when you can use GPS; learning music is probably a lot easier if there's zero recorded music; why see a play when you can watch a movie; etc etc etc etc.

^

If the question is too vague for you (eg because tech consequences depend on how,when,where,bywho it is rolled out) then imagine you have the same degree of control over the tech that a well-run well-funded medium-sized tech co would have. And you roll it out in the best non-corporate-suicide (but not necessarily most profitable) way you can. So eg some superbio tech could be 100% great if the tech co keeps it secret and just sells drugs etc instead of selling the superbiotech itself.

However, many bonus points if the most profitable/natural/realistic rollout of the given tech is also good.

^

I omit social/policy tech that I don't have any idea how one would

...
Leave No Context Behind - A Comment
by Gunnar_Zarncke — 2024-04-11T22:50:26.100Z

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention by Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal of Google.

This is a pre-print of a new LLM extension with what I'd call short-term memory, but what they call infini-attention. It was published just yesterday (2024-04-10) and I came across it on XiXiDu's daily AI summary. I think it may be at a turning-point in a certain type of capability of LLMs and I want to comment on it.

Here is the abstract:

This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.

I have read most of the preprint now, and this is the closest to a model having something like short-term memory that I have seen (there are others, but the paper says this is the first LLM that doesn't fully discard looked-up memory after a step). With its incremental update rule, this model can keep track of some topic over time (1 mio token context window tested, but there is no upper limit). It learns to use these transient representations efficiently and thereby which things to "keep in mind".

The paper:

An effective memory system is crucial not just for comprehending long contexts with LLMs, but also for reasoning, planning, continual adaptation for fresh knowledge, and even for learning how to learn.

They call it infini-attention because of the incremental update, but it is really a continuous short-term memory system. Of course, it works very differently from how short-term memory works in biological brains, e.g., it seems to work at multiple layers in parallel and at a fixed granularity of tokens (2k in the tests). Still, the way it works, I think it's conceivable that such a model could show...
AXRP Episode 27 - AI Control with Buck Shlegeris and Ryan Greenblatt
by DanielFilan — 2024-04-11T21:30:04.244Z

YouTube link

A lot of work to prevent AI existential risk takes the form of ensuring that AIs don’t want to cause harm or take over the world—or in other words, ensuring that they’re aligned. In this episode, I talk with Buck Shlegeris and Ryan Greenblatt about a different approach, called “AI control”: ensuring that AI systems couldn’t take over the world, even if they were trying to.

Topics we discuss:

What is AI control?

Protocols for AI control

Which AIs are controllable?

Preventing dangerous coded AI communication

Unpredictably uncontrollable AI

What control looks like

Is AI control evil?

Can red teams match misaligned AI?

How expensive is AI monitoring?

AI control experiments

GPT-4’s aptitude at inserting backdoors

How AI control relates to the AI safety field

How AI control relates to previous Redwood Research work

How people can work on AI control

Following Buck and Ryan’s research

Daniel Filan:
Hello, everybody. In this episode, I’ll be speaking with Buck Shlegeris and Ryan Greenblatt. They work at Redwood Research on the project of AI control: that is, averting risks from AIs that intentionally subvert safety measures. For links to what we’re discussing, you can check the description of this episode and you can read the transcript at axrp.net. All right, Buck and Ryan, welcome to the show.

Buck Shlegeris:
Exciting to be here.

Ryan Greenblatt:
Exciting to be here as well.

What is AI control?

Daniel Filan:
Cool. So I guess we’re going to be talking about the control approach to AI safety (or something), that you two have written about. For people who haven’t done background reading, what is the control approach?

Buck Shlegeris:
AI control is an approach to managing intentional risk posed by models. In particular, it’s an approach to managing risks that arise from models intentionally subverting your attempts to ensure and evaluate safety. The core idea of AI control is: here are two different reasons why you might be safe from models intentionally causing problems. One is that the models might not try to intentionally cause problems, and the second argument is you might have it be such that even if your models try as hard as they can to cause problems, they’re unable to do so. And AI control is the second class of approach to mitigating problems with models intentionally subverting your safety techniques. And we think t...
ChatGPT defines 10 concrete terms: generically, for 5- and 11-year-olds, and for a scientist 
by Bill Benzon — 2024-04-11T20:27:24.784Z

This is cross-posted from New Savanna.

The difference between concrete concepts, that is, concepts that can be understood entirely in sensorimotor terms, and abstract terms is an important one. It was, for example, important to David Hays when I studied with him back in the 1970s. We ended up adapting a model developed by William Powers as a way of thinking about concrete concepts while using Hays’s concept of metalingual definition to think about abstract concepts. Somewhat later Steven Harnad coined the term, “symbol grounding problem,” to indicate that the meanings of symbols had to somehow be grounded in the physical world. In 2016 Harnad and his colleagues investigated the structure of dictionaries and discovered that the vast majority of terms in them were defined in terms of other words but that there is a small Kernel that is not so-defined. I wondered how that distinction would play out in ChatGPT.

ChatGPT – that is, the LLM underlying it – doesn’t have access to the physical world. It can’t ground any terms in sensorimotor experience. However, knowing a great deal about how language works and having spent a great deal of time working with ChatGPT, I was sure that it would be perfectly capable to defining concrete terms. But how would it do so?

I decided to investigate. I made a list of ten concrete terms and asked ChatGPT to define them. In my first trial I made no further specification beyond simply asking for a definition. In subsequent trials I asked for definitions appropriate for 5-year-olds, 11-year-olds, and professional biologists (just the biological concepts). ChatGPT gave distinctly different kinds of definitions in each case, which did not surprise me since over a year ago I had asked ChatGPT to describe digestion, first generically, and then in terms appropriate for a 3-year-old. It did so, and fairly successfully.

Here are the trials. Each trial is a separate session. As always, my prompt is in boldface while the response is in plainface.

Initial Trial

Define the following words:

cat
rain
tulip
car
food
tree
fire
wind
hole
animal

Sure, here are definitions for each of those words:

Cat: A small domesticated carnivorous mammal with soft fur, a short snout, and retractile claws, often kept as a pet and valued for its companionship and ability to hunt pests.
Rain: Moisture condensed from the atmosphere that falls visibly in separate drops. It is an es

...