trentmkelly/LessWrong-43k huggingface dataset:
What is AI Safety’s line of retreat?
by Remmelt — 2024-07-28T05:43:05.021Z

Given we’re not on track to control fully autonomous systems to stay safe, what do we do?

Let’s say alignment research will not catch up with capability development. For any combination of reasons: corporations are scaling too fast, there are too many lethal-if-unsolved subproblems we have at most made partial process on, some subproblems can only be solved in sequential order, or there are hard limits capping the progress we can make on the control problem.

What do we do? With ‘we’, you can consider yourself, specific organisations, or the community coordinating roughly as a whole.
AXRP Episode 34 - AI Evaluations with Beth Barnes
by DanielFilan — 2024-07-28T03:30:07.192Z

YouTube link

How can we figure out if AIs are capable enough to pose a threat to humans? When should we make a big effort to mitigate risks of catastrophic AI misbehaviour? In this episode, I chat with Beth Barnes, founder of and head of research at METR, about these questions and more.

Topics we discuss:

What is METR?

What is an “eval”?

How good are evals?

Are models showing their full capabilities?

Evaluating alignment

Existential safety methodology

Threat models and capability buffers

METR’s policy work

METR’s relationship with labs

Related research

Roles at METR, and following METR’s work

Daniel Filan:
Hello everybody. In this episode I’ll be speaking with Beth Barnes. Beth is the co-founder and head of research at METR. Previously, she was at OpenAI and DeepMind, doing a diverse set of things, including testing AI safety by debate and evaluating cutting-edge machine learning models. In the description, there are links to research and writings that we discussed during the episode. And if you’re interested in a transcript, it’s available at axrp.net. Well, welcome to AXRP.

Beth Barnes:
Hey, great to be here.

What is METR?

Daniel Filan:
Cool. So, in the introduction, I mentioned that you worked for Model Evaluation and Threat Research, or METR. What is METR?

Beth Barnes:
Yeah, so basically, the basic mission is: have the world not be taken by surprise by dangerous AI stuff happening. So, we do threat modeling and eval creation, currently mostly around capabilities evaluation, but we’re interested in whatever evaluation it is that is most load-bearing for why we think AI systems are safe. With current models, that’s capabilities evaluations; in future that might be more like control or alignment evaluations. And yeah, [the aim is to] try and do good science there, be able to recommend, “Hey, we think if you measure this, then you can rule out these things. You might be still concerned about this thing. Here’s how you do this measurement properly. Here’s what assumptions you need to make,” this kind of thing.

Daniel Filan:
Gotcha. So, mostly evaluations. But it sounded like there was some other stuff as well, like threat modeling you mentioned.

Beth Barnes:
Yeah. We also do policy work recommending things in the direction of responsible scaling policies. So, saying what mitigations are needed based on the results of different evaluation...
AI existential risk probabilities are too unreliable to inform policy
by Oleg Trott — 2024-07-28T00:59:59.497Z
Link: https://www.aisnakeoil.com/p/ai-existential-risk-probabilities

Melanie Mitchell called the article "excellent". It also got a bit of discussion on HN.

(This is an FYI. I don't necessarily approve or endorse the links I post.)
Idle Speculations on Pipeline Parallelism
by DaemonicSigil — 2024-07-27T22:40:12.543Z
Link: https://pbement.com/posts/pipepara.html

Epistimic status: Pure speculation; I have 1 GPU, not dozens, so I have no reason to try and implement this stuff myself.

When training large models, we might have many GPUs we'd like to use to speed the process up. This requires a way to parallelize training somehow. Some options here are Data Parallelism and Pipeline Parallelism.

If our model is very large, we might have trouble fitting it into the memory of a single GPU. Pipeline parallelism is convenient in such a case, because it treats the model as a pipeline, where each GPU is responsible for implementing a part of the pipeline. Eg. if the model consists of 64 identical blocks, then we could assign a GPU to each block. Then the activations coming out of each block are passed to the next block. For inference, these activations are the only data that must be communicated between GPUs, which makes pipeline parallelism a very bandwidth-efficient optimization. In order to keep each GPU busy, we must feed in a continuous stream of data, adding more work before the work we previously added makes it through the pipeline.

Data parallelism is where we have a very large batch which we split up across multiple GPUs. Each GPU computes the output of the model on its own sub-batch. We'll mostly focus on pipeline parallelism today, though both methods can be used at once.

Though it's great for inference, pipeline parallelism has some issues for training. Check this article for details, but I'll give a quick overview. The problem is, we have to compute backwards passes, and also weights are continually updated by the optimization algorithm. The need to compute backwards passes introduces a bubble into the system, and the size of the bubble increases with the length of the pipeline. See this figure from the DAPPLE paper:

What we can see is that the Dapple method has a much friendlier memory usage than the method originally put forward in GPipe. In hindsight, it's clear that we should try and do the backwards pass as soon as we can after the forwards pass completes, in order to free up all the saved activations. This does get complicated because the amount of work in a backward() is not necessarily the same as the amount of work in a forward(). Let's neglect that issue for now though, and trust that the Dapple paper has provided good methods for dealing with it!

removing the bubble

Dapple doesn't get rid of the bubble, j...
Re: Anthropic's suggested SB-1047 amendments
by RobertM — 2024-07-27T22:32:39.447Z
Link: https://www.documentcloud.org/documents/25003075-sia-sb-1047-anthropic

Note: I received a link to the letter from elsewhere, but it's also cited in this SF Chronicle article, so I'm pretty confident it's real. Thanks to @cfoster0 for the SF Chronicle link.

If you're familiar with SB 1047, I recommend reading the letter in full; it's only 7 pages.

I'll go through their list of suggested changes and briefly analyze them, and then make a couple high-level points. (I am not a lawyer and nothing written here is legal advice.)

Major Changes

Greatly narrow the scope of pre-harm enforcement to focus solely on (a) failure to develop, publish, or implement an SSP[1] (the content of which is up to the company); (b) companies making materially false statements about an SSP; (c) imminent, catastrophic risks to public safety.

Motivated by the following concern laid out earlier in the letter:

The current bill requires AI companies to design and implement SSPs that meet certain standards – for example they must include testing sufficient to provide a “reasonable assurance” that the AI system will not cause a catastrophe, and must “consider” yet-to-be-written guidance from state agencies. To enforce these standards, the state can sue AI companies for large penalties, even if no actual harm has occurred. While this approach might make sense in a more mature industry where best practices are known, AI safety is a nascent field where best practices are the subject of original scientific research. For example, despite a substantial effort from leaders in our company, including our CEO, to draft and refine Anthropic’s RSP over a number of months, applying it to our first product launch uncovered many ambiguities. Our RSP was also the first such policy in the industry, and it is less than a year old. What is needed in such a new environment is iteration and experimentation, not prescriptive enforcement. There is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements.

While SB 1047 doesn't prescribe object-level details for how companies need to evaluate models for their likelihood of causing critical harms, it does establish some requirements for the structure of such evalutions (22603(a)(3)).

Section 22603(a)(3)

(3) Implement a written and separate safety and security protocol that does all of t

...
The problem with psychology is that it has no theory.
by Nicholas D. — 2024-07-27T19:36:44.601Z
Link: https://nicholasdecker.substack.com/p/psychology-is-bad-because-it-has

i. The problem with psychology

Psychology and economics are both attempts to study human behavior. While the precise focus may differ, we want to be able to make precise and accurate predictions about how humans respond to stimuli. We can use reasoning to make theoretical predictions about what will occur, and use empirical results to confirm or disconfirm them, or use empirical results to guide what our theory should be. You must have both, however.

I argue that economics is far better than psychology because it has a theory. Psychology does not. It is the fitting of epicycles to match observed empirical phenomena. If published research were an unbiased reflection of reality, this would be a distinction without a difference, but published research is distorted and obviously distorted. Psychology is a science without priors. They are left chasing the newest paper, without the slightest idea why it should be true or not true. Its only test of plausibility is whether it can tell a plausible story rationalizing the results. The funny thing about stories, though, is that you can spin a story to fit any particular set of facts. It is, moreover, in principle not possible to say what causes what without theory when looking at data sets after the fact. A randomized controlled trial (where people are assigned some treatment at random) can sort out that some treatment caused an effect, but for many interesting questions we can only observe data sets after the fact. As

this paper

proves, you cannot establish what causes what without a theory, without a prior.

Consider the example of priming. Priming, since debunked, is the idea that making people think about certain things will unconsciously affect their behavior. People who read a text about being old would supposedly walk slower after reading it. It is striking that it would be perfectly plausible to rationalize the exact opposite effect. You find that they walk faster – perhaps because they were reminded of their mortality, and walk faster to remind themselves of youth. You don’t have a sound sense of what would be confirming evidence, and what would be surprising evidence.Research will go on until a significant, publishable result is found – and so you end up with spurious claims on pointless questions.

This is not merely my complaint. Psychologists, surveying their own field with disappointment, agre...
Bryan Johnson and a search for healthy longevity
by NancyLebovitz — 2024-07-27T15:28:13.117Z

I've listened to some of his videos, and he strikes me as at least rationalist-adjacent.
https://www.bryanjohnson.com/
I was wondering what people think of him and his ideas, and whether they'd tried out any of his system.
I respect that he offers his research for free, and I don't see a problem with him charging for personal consultation.
I don't expect all of his ideas, even the ones he's been using for a while, to work for everyone, and I haven't seen anything about how much variation there is in what his clients seem to need.
I would like to see independent testing of his food and supplements. Are the heavy metals really that low? How good is the evidence that people are generally getting enough heavy metals from their food and supplements to be a problem?
I would appreciate it if people don't say that he's weird so his ideas must be bad.
What are matching markets?
by ohmurphy — 2024-07-27T15:05:28.647Z
Link: https://ohmurphy.substack.com/p/notes-on-reading-who-gets-what-and

{Epistemic Status: Just reviewing my thoughts on a book and its subject matter. I have done some independent study of the field (and have a degree in economics), but I did not try to check my impressions rigorously. Much or maybe most of what I say is extrapolation by me, rather than being in the book itself.}

I have recently finished the book Who Gets What and Why, by Alvin Roth and found it quite intellectually stimulating. The book itself was not as systematic as I originally hoped, but that was almost better since it forced me to reconstruct definitions and patterns in ways that made sense to me. Below I have included my thoughts on the first topic of the book, matching markets. The second topic, mechanism design, will appear in a later post.

What Are Matching Markets

The first model of a market that economics students learn is the perfectly competitive model. This is exceptionally useful to understand, but it can also limit the scope of what people consider as markets. In particular, this model emphasizes two characteristics of a market, price and quantity, neither of which are explicitly required to model a market. Matching markets include markets where there can be both no prices and no changes in quantity (i.e. no supply or demand curves).[1] They are also more than academic curiosities since they describe everything from dating markets to job markets to school admissions.

You can use supply and demand to get some insight into how the market for romantic relationships works, but if you start taking it too seriously then it stops making sense. For example: Is there a single market price? Are the surpluses of each side directly opposed? Do people always prefer lower ‘prices’? Is a person with a higher willingness-to-pay always going to do better?

The primary characteristic of a matching market is that both sides of an exchange need to actively choose each other in order for it to go through. How does this actually differ from a standard (commodities) market? The key distinction is that each exchange can now be dependent on the outcome of another exchange.

For example, imagine a group of tennis players that want to enter a doubles tournament. We can imagine a player Alice who wants to team up with Bob who wants to team up with Charlie. Since Bob would prefer Charlie to Alice, Alice’s ability to team up with Bob is determined by whether Charlie will choos...
Safety consultations for AI lab employees
by Zach Stein-Perlman — 2024-07-27T15:00:27.276Z

Many people who are concerned about AI x-risk work at AI labs, in the hope of doing directly useful work, boosting a relatively responsible lab, or causing their lab to be safer on the margin.

Labs do lots of stuff that affects AI safety one way or another. It would be hard enough to follow all this at best; in practice, labs are incentivized to be misleading in both their public and internal comms, making it even harder to follow what's happening. And so people end up misinformed about what's happening, often leading them to make suboptimal choices.

In my

AI Lab Watch

work, I pay attention to what AI labs do and what they should do. So I'm in a good position to inform interested but busy people.

So I'm announcing an experimental service where I provide the following:

Calls for current and prospective employees of frontier AI labs.

Book here
On these (confidential) calls, I can answer your questions about frontier AI labs' current safety-relevant actions, policies, commitments, and statements, to help you to make more informed choices.
These calls are open to any employee of OpenAI, Anthropic, Google DeepMind, Microsoft AI, or Meta AI, or to anyone who is strongly considering working at one (with an offer in hand or expecting to receive one).
If that isn't you, feel free to request a call and I may still take it.

Support for potential whistleblowers. If you're at a lab and aware of wrongdoing, I can put you in touch with:

Former lab employees and others who can offer confidential advice
Vetted employment lawyers
Communications professionals who can advise on talking to the media.

If you need this, email zacharysteinperlman at gmail or message me on Signal at 734 353 3975.

I don't know whether I'll offer this long-term. I'm going to offer this for at least the next month.

My hope is that this service makes it much easier for lab employees to have an informed understanding of labs' safety-relevant actions, commitments, and responsibilities.

If you want to help—e.g. if maybe I should introduce lab-people to you—let me know.

You can

give me anonymous feedback

.

Crossposted from AI Lab Watch. Subscribe on Substack.
The Case Against UBI
by Zero Contradictions — 2024-07-27T06:36:01.957Z
Link: https://thewaywardaxolotl.blogspot.com/2020/04/the-case-against-ubi.html

The PDF version can be read here.

UBI stands for “Universal Basic Income”. In its simplest form, UBI is a direct payment to every citizen every month. It would (at least in theory) replace other government programs that alleviate poverty, such as means-tested welfare. It is growing in popularity as a political proposal. In this essay, I’m going to make the case against UBI.

The basic argument for UBI is that we already agree on the existence of a social safety net, and UBI would be simpler and fairer than existing welfare schemes. As it is typically conceived, UBI would just be a direct payment from the government to all members of a society, without the complex bureaucracy that administers means-tested welfare schemes. Because it would be universal rather than means-tested, it would also eliminate or reduce certain perverse incentives of existing welfare schemes.

Let’s consider some of the problems with means-tested welfare scheme. They require means-testing on an individual basis, so they need a bureaucracy to administer and enforce. They can be abused in various ways, such as by working on the black market, using false identities to claim extra benefits, or lying about family relationships. They create a perverse incentive not to work. If a person on welfare gets a job, she loses her welfare benefits. This means that the net benefit from getting a job might be very small, especially considering the lost work at home and the costs of employment. Means-tested welfare schemes also create perverse sexual incentives. They disincentivize marriage and pair-bonding (because a husband’s income might disqualify a woman from receiving benefits). They create a financial incentive to have more children, because welfare typically increases with each child. The overall effect is a welfare-dependent culture of single mothers, short term sexual relationships, and black market employment (such as dealing illegal drugs). Means-tested welfare has a lot of negative consequences.

I agree that UBI would partially remedy some of those problems. However, it can be very misleading to compare an existing scheme to a hypothetical scheme. It is very easy to be fooled by such a comparison (or use it to fool others). It is much easier to imagine something than to make it real. People are often seduced by utopian dreams that turn into dystopian nightmares (e.g. communism).

UBI would still re...
Unlocking Solutions—By Understanding Coordination Problems
by James Stephen Brown — 2024-07-27T04:52:13.435Z
Link: https://nonzerosum.games/unlockingsolutions.html

The following is a post introducing coordination problems, using the examples of poaching, civilisational development, drug addiction and affirmative action. It draws on my experience as a documentary filmmaker. The post is available for free in its original format at nonzerosum.games.

When I was eleven, I disassembled the lock to our back door, and as I opened the housing… it exploded, scattering six tiny brass pellets on to the floor.

I discovered (too late) that a lock of this type contained spring-loaded cylinders of different heights corresponding to the teeth of the key.

I struggled for hours trying to get the little buggers back in, but it was futile—eventually, my long suffering parents called a locksmith.

The reason fixing the lock was so difficult was not only because it was spring-loaded but because I had to find the right combination and hold them all in balance as I put it back together. I just couldn’t coordinate everything.

Coordination Problems

We sometimes run into problems where a number of factors have to be addressed simultaneously in order for them to be effective at all. One weak link can ruin it for the rest. These are called Coordination Problems. The fact that they are so much more difficult to solve than other problems means that many of the problems remaining in the world today, end up being coordination problems.

Poaching

An example of a system requiring more than one problem to be solved at once, is poaching. If you police poaching behavior but don’t address the buyers you are left with the perpetual cost of policing, because the demand remains. If you address the buyers, the poachers, who are likely living in poverty may just move on to some other criminal behavior. Daniel Schmachtenberger tells the story of eliminating elephant poaching in one particular region in Africa:

“The first one I noticed when I was a kid was trying to solve an elephant poaching issue in one particular region of Africa that didn’t address the poverty of the people, that had no mechanism other than black market on poaching, didn’t address people’s mindset towards animals, didn’t address the macro-economy that created poverty at scale. So when the laws were put in place and the fences were put in place to protect those elephants in that area better, the poachers moved to poaching other animals, particularly in that situation, rhinos and gorillas that were both mor

...