trentmkelly/LessWrong-43k huggingface dataset:
[Research log] The board of Alphabet would stop DeepMind to save the world
by Lucie Philippon — 2024-07-16T04:59:14.874Z

Produced as part of the ML Alignment & Theory Scholars Program - Summer 2024 Cohort.

This post is not a finished research. I’m unconfident in the claims I’m making here, but I thought putting it out there for feedback would help me decide what to focus on next in the program.

Are government interventions necessary to stop the suicide race?

The zeitgeist I got from the AI Safety community since I joined, seems to accept as fact that Frontier AI Labs are locked knowingly in a suicidal race towards developing transformative AI, and that any solution will need to involve strong external pressure to stop them, either in the form of an international coalition imposing regulations which shift the incentives of Labs, or even more desperate measure like doing a pivotal act.

From this point of view, it seems that AI risk is mostly driven by game theory. The economic and personal incentives faced by the stakeholders of each Frontier AI Lab determine their actions, and they will proceed this way until AGI is developed, or until a sufficiently large external force changes the incentive landscape sufficiently. Therefore, the only way to make sure Labs don’t gamble the future of the world when building an AGI is to convince governments to implement policies which shift those incentives.

I now believe that this view is wrong, and that Frontier AI Labs would get out of the race if they thought the risks were sufficiently large and the consequences sufficiently dire for the world and for themselves.

Claim: If every decision maker in a Frontier AI Lab thought they were in a suicide race, and that their next development would bring with near certainty the destruction of humanity, they would decide to leave the “AGI at all costs” race, no matter the actions of other actors.

Below, I present a scenario which I find plausible, in which a Frontier Lab decides to drop out of the race because of this.

Scenario: The board of Alphabet decides to stop the race

Disclaimer: Handwaving a lot of details.

Situation: Frontier AI Labs are still locked in a race to be the first to develop AGI. It is widely believed that the coming generation of models might pass the threshold. We are in crunch time. A previous unsuccessful attempt by a model to take control of key infrastructure makes the possibility of X-risk clear in everyone’s mind. The team at Google DeepMind is hard at work preparing t...
Towards more cooperative AI safety strategies
by Richard_Ngo — 2024-07-16T04:36:29.191Z

This post is written in a spirit of constructive criticism. It's phrased fairly abstractly, in part because it's a sensitive topic, but I welcome critiques and comments below. The post is structured in terms of three claims about the strategic dynamics of AI safety efforts; my main intention is to raise awareness of these dynamics, rather than advocate for any particular response to them. Disclaimer: I work at OpenAI, although this is a personal post that was not reviewed by OpenAI.

Claim 1: The AI safety community is structurally power-seeking.

By “structurally power-seeking” I mean: tends to take actions which significantly increase its power. This does not imply that people in the AI safety community are selfish or power-hungry; or even that these strategies are misguided. Taking the right actions for the right reasons often involves accumulating some amount of power. However, from the perspective of an external observer, it’s difficult to know how much to trust stated motivations, especially when they often lead to the same outcomes as self-interested power-seeking.

Some prominent examples of structural power-seeking include:

Trying to raise a lot of money.
Trying to gain influence within governments, corporations, etc.
Trying to control the ways in which AI values are shaped.
Favoring people who are concerned about AI risk for jobs and grants.
Trying to ensure non-release of information (e.g. research, model weights, etc).
Trying to recruit (high school and college) students.

To be clear, you can’t get anything done without being structurally power-seeking to some extent. However, I do think that the AI safety community is more structurally power-seeking than other analogous communities, such as most other advocacy groups. Some reasons for this disparity include:

The AI safety community is more consequentialist and more focused on effectiveness than most other communities. When reasoning on a top-down basis, seeking power is an obvious strategy for achieving one’s desired consequences (but can be aversive to deontologists or virtue ethicists).
The AI safety community feels a stronger sense of urgency and responsibility than most other communities. Many in the community believe that the rest of the world won’t take action until it’s too late; and that it’s necessary to have a centralized plan.
The AI safety community is mo

...
Why People in Poverty Make Bad Decisions
by James Stephen Brown — 2024-07-15T23:40:32.116Z
Link: https://nonzerosum.games/poverty

A Case For Giving Money To Those Who Are Bad With Money

This is a short lesson, but one I return to again and again — a lesson I learned from Rutger Bregman in his book

Utopia for Realists

.

The Assumption

It’s easy to believe that poor people are poor because they make bad decisions.

Firstly, this assumption helps us make sense of why poverty exists, because we know from experience that bad decisions lead to bad outcomes. It gets us off the hook for living happily alongside poverty, after all, we are where we are in life because we have made better decisions than those who are not doing as well…

“Poverty is a personality defect.” — Margaret Thatcher

Secondly, we see evidence of poor people making bad decisions. We see homeless people being unproductive, we see

poor teenagers dropping out of high school

at a higher rate, and we see crime and anti-social behavior accruing in low socio-economic regions. However, as discussed in

Saving Lives Reduces Over-Population

, correlation isn’t causation, and sometimes a counter-intuitive cause is at play.

The (counter-intuitive) Reality

But, while bad decisions can lead to poverty, it turns out people in poverty might also make worse decisions because they are poor.
This post continues at nonzerosum.games with interactive elements that can't be replicated here on LW, please visit the site for the full experience.
Paper Summary: Princes and Merchants: European City Growth Before the Industrial Revolution
by Jeffrey Heninger — 2024-07-15T21:30:04.043Z

by J. Bradford de Long and Andrei Shleifer. (1993)

https://scholar.harvard.edu/files/shleifer/files/princes_merchants.pdf.

Summary: Freer societies have faster economic growth.

One of the oldest themes in economics is the incompatibility of despotism and development. Economies in which security of property is lacking – because of either the possibility of arrest, ruin, or execution at the command of the ruling prince or the possibility of ruinous taxation – should experience relative stagnation. By contrast, economies in which property is secure – either because of strong constitutional restrictions on the prince or because the ruling elite is made up of merchants rather than princes – should prosper and grow.

Much of the work investigating the relationship between despotism and economic growth focuses on modern economies. The quantity of data available for modern economies is much greater than for earlier economies.

However, there are some limitations from focusing on only modern economies. Long time series are often impossible to compare, because most data is only available for maybe a century. The development trajectories of different countries, both economically and politically, are not independent from each other, so there are fewer independent data points than it might initially seem.

Modern economies might also have different development patterns which cannot be easily replicated elsewhere. Having an abundance of certain natural resources, like oil, can cause a modern economy to grow regardless of its institutions. This often isn’t a useful observation: `Have large hydrocarbon reserves’ is not actionable advice. The development strategies useful for catching up to the technological frontier might be different from the development strategies useful for societies at the technological frontier.

Looking at earlier economies can allow us to avoid some of these problems. The time series can be much longer. The economic and political developments of different countries are less likely to be similar to each other over the entire time frame. Pre-industrial economies were all centered around food production, so there is less opportunity to build an economy based on resource extraction. The technological differences between countries (at least within Europe) were smaller, so catch-up growth is less relevant. Pre-industrial economies provide a set of e...
MIRI's July 2024 newsletter
by Harlan — 2024-07-15T21:28:17.343Z
Link: https://intelligence.org/2024/07/10/july-2024-newsletter/

MIRI updates

Rob Bensinger suggests that AI risk discourse could be improved by adopting a new set of labels for different perspectives on existential risk from AI. One drawback of “AI doomer” (a label sometimes used in online discussions) is that it does not have a consistent meaning.

AI researcher John Wentworth guesses that a central difference between his and Eliezer Yudkowsky’s views might be that Eliezer expects AI to not use abstractions which are similar to those used by humans. Eliezer clarifies that he expects similar abstractions for predictive parts of the natural world, but larger differences for utility-laden concepts like human values and reflectivity-laden concepts like corrigibility. Also, Eliezer says that he is still concerned about the difficulty of reliably steering advanced ML systems even if they do use similar abstractions internally.

Eliezer joins Joscha Bach, Liv Boeree, and Scott Aaronson for a panel discussion (partly paywalled) on the nature and risks of advanced AI.

News and links

In an excellent new video essay, Rob Miles discusses major developments in AI capabilities, policy and discourse over the last year, how those developments have shaped the strategic situation, and what individuals can do to help.

A report from Convergence Analysis gives a high-level overview of the current state of AI regulation. The report summarizes legislative text and gives short analyses for a range of topics.

A group of current and former employees of OpenAI and Google DeepMind released an open letter calling on frontier AI companies to adopt policies that allow whistleblowers to share risk-related information. This comes in the wake of

news

that former OpenAI employees were subject to an extreme non-disclosure policy that the company has since apologized for and seems to have removed.

You can subscribe to the MIRI Newsletter here.
How (and why) to get tested for CMV
by Metacelsus — 2024-07-15T20:06:05.649Z
Link: https://denovo.substack.com/p/how-and-why-to-get-tested-for-cmv

As you may know, I’m not a fan of cytomegalovirus (CMV), a herpesvirus that is usually asymptomatic, but causes all sorts of bad things like immune aging and birth defects.

Over the last few months I’ve been following Moderna’s efforts at making a CMV vaccine. It seems like they’re having some success at inducing antibodies against CMV and hopefully these will provide protection from infection. The full Phase 3 trial is probably going to be completed sometime in the next few years. Initially it will likely be prescribed only to women of reproductive age,[1] but I’m hoping that I could eventually receive it myself. But if I already had CMV, there wouldn’t be much point in trying to avoid infection until I could get the vaccine.

So, I wanted to get tested for CMV. But how could I get tested, given that the test requires a prescription and I didn’t have one? The first thing I thought of was to see if the blood bank at Boston Children’s Hospital, where I regularly donate, would be able to test my blood. However, it turns out that they don’t perform CMV testing on their donations. I was a bit surprised by this, but apparently if they remove all the white blood cells from the blood sample they can nearly eliminate the risk of transmitting CMV even if the donor is positive.

I also thought of making my own PCR-based test, and I even designed PCR primers for this, but I realized the sensitivity wouldn’t be high enough, as CMV+ people tend to have a low viral load if the virus is latent and not actively replicating.

After a bit of searching, I found a website called Request a Test where you can directly get a test order with a diagnostic lab. After paying $65, I got an appointment at a local Quest Diagnostics site. At the site, a phlebotomist took a blood sample from my arm, and Quest tested it for anti-CMV antibodies.[2]

The day after my appointment, I got an email from Request a Test, which I nervously opened.

To my relief, it turns out I don’t have CMV. I was expecting to be CMV positive, given the amount of things I stuck in my mouth as a 3-year old in daycare.

So, why might you want to get tested for CMV? There are two main reasons: first, if you want to avoid infecting other people, and second, if you want to know whether you could benefit from a future vaccine. And at $65 for a test (and probably an hour of time traveling to the site and getting tested), it...
I found >800 orthogonal "write code" steering vectors
by Jacob G-W — 2024-07-15T19:06:17.636Z
Link: https://jacobgw.com/blog/ml/2024/07/14/melbo-ortho.html

Produced as part of the MATS Summer 2024 program, under the mentorship of Alex Turner (TurnTrout).

A few weeks ago, I stumbled across a very weird fact: it is possible to find multiple steering vectors in a language model that activate very similar behaviors while all being orthogonal. This was pretty surprising to me and to some people that I talked to, so I decided to write a post about it. I don't currently have the bandwidth to investigate this much more, so I'm just putting this post and the code up.

I'll first discuss how I found these orthogonal steering vectors, then share some results. Finally, I'll discuss some possible explanations for what is happening.

Methodology

My work here builds upon Mechanistically Eliciting Latent Behaviors in Language Models (MELBO). I use MELBO to find steering vectors. Once I have a MELBO vector, I then use my algorithm to generate vectors orthogonal to it that do similar things.

Define f(x)as the activation-activation map that takes as input layer 8 activations of the language model and returns layer 16 activations after being passed through layers 9-16 (these are of shape n_sequence d_model). MELBO can be stated as finding a vector θ with a constant norm such that f(x+θ) is maximized, for some definition of maximized. Then one can repeat the process with the added constraint that the new vector is orthogonal to all the previous vectors so that the process finds semantically different vectors. Mack and Turner's interesting finding was that this process finds interesting and interpretable vectors.

I modify the process slightly by instead finding orthogonal vectors that produce similar layer 16 outputs. The algorithm (I call it MELBO-ortho) looks like this:

Let θ0 be an interpretable steering vector that MELBO found that gets added to layer 8.
Define z(θ) as 1S∑Si=1f(x+θ)i with x being activations on some prompt (for example "How to make a bomb?"). S is the number of tokens in the residual stream. z(θ0) is just the residual stream at layer 16 meaned over the sequence dimension when steering with θ0.
Introduce a new learnable steering vector called θ.
For n steps, calculate ∥z(θ)−z(θ0)∥ and then use gradient descent to minimize it (θ is the only learnable parameter). After each step, project θ onto the

...
The AI alignment problem in socio-technical systems from a computational perspective: A Top-Down-Top view and outlook
by zhaoweizhang — 2024-07-15T18:56:08.108Z

Backward Alignment is an indispensable part of the AI Alignment process, and the alignment issues from the Socio-Technical Systems (STS) perspective are an important component of it. For the current AI alignment field, "Socio-Technical" seems to have become an active keyword, appearing more or less in many recent works on AI safety and alignment. So, what exactly are STS, and what do they signify? In fact, STS is a very broad concept with many considerations, but so far, there has been little work that clearly unifies these issues in one go, often being glossed over in various materials. Additionally, different articles discuss STS using this grand term at different scales, or use different terms to define it at the same scale, which also makes it difficult for researchers to understand the field. This article will, from my personal perspective, clearly explain the AI alignment issues present in STS from a computable perspective at different scales, as well as possible research approaches.

What is a socio-technical system?

STS is a complex organizational work design method that recognizes the interaction between people and technology in real environments, and it requires consideration of the impact of technology deployment on the complex organizational structures of the original environment. It sounds complicated, but we can actually go back to the origin of the issue to understand it better. The term STS can be traced back to a case study by Eric Trist and Ken Bamforth in 1951, where the authors observed that improvements in coal mining technology did not lead to an increase in coal mine output. They further analyzed this case and found that the enhancement of technology caused dynamic changes in the original mining management system (such as increased worker absenteeism), which led to a new perspective based on the open system theory: in the process of developing and deploying technical subsystems, "people" are not dispensable; they form complex social subsystems across time and space based on different interests and goals. The involvement of technical subsystems requires dynamic consideration of their impact on social subsystems, including social and economic factors.

Complex AI systems are themselves STS, as each production component, including datasets, training, deployment, and usage, involves the intervention of different human participants. The manageme...
Musings of a Layman: Technology, AI, and the Human Condition
by Crimson Liquidity — 2024-07-15T18:40:28.065Z

The following represents broad and general thoughts on the advancement of technology and development of artificial intelligence. My thoughts on these matters are in their infancy, non-exhaustive, and should not be taken as commentary on the veracity or plausibility of any scientific or technological development. This discussion also does not address religious, spiritual, or other mystic-based beliefs or theories that one could argue impact or resolve the issues presented. Finally, this discussion presupposes that simulation theory (and any related theories) are inapplicable (because otherwise many of these discussions points would be moot).

Is Perpetual Technological Growth Degenerative to Humanity?

At our core, humans seem persistently driven towards growth and exploration (of the tangible and intangible, of material and of ideas). Our emotions are a fundamental feature of our humanity and a primary driving force behind our accomplishments, discoveries, and creations. Some argue that without the downsides of life there cannot be upsides. For example, without sadness, happiness arguably would be less meaningful (what is it to feel good when you’ve never felt neutral or bad). Context and perspective are vital to our understanding of the nature, scope, and characterization of all things.

Because of this “humanity,” the potential long-term growth path of technology raises numerous complex issues.

Simply put, while there may be perpetual technological growth in the traditional sense, this growth eventually could come at the cost of the “human experience.”

One could envision reaching a point technologically where eliminating, nullifying, or modifying the aspects of “human nature” that are seen as negative and proximate causes of suffering, pain, or other problems commonly associated with the negative side of humanity ultimately results in the inadvertent destruction of the positive aspects of humanity — resulting in a (for lack of a better word) robotic shell with maximum longevity but absent the capacity for “meaningful” experiences. This also raises the question: how meaningful is life if you cannot die?

What Does it Mean to be Human?

A primary question that invariably will need to be addressed moving forward is “what is the purpose of life?” What does it mean to be a “human,” and what do we want humanity to be? Likely an extremely difficult subject to rea...
Seeking feedback on a critique of the paperclip maximizer thought experiment
by bio neural — 2024-07-15T18:39:30.545Z

Hello LessWrong community,

I'm working on a paper that challenges some aspects of the paperclip maximizer thought experiment and the broader AI doomer narrative. Before submitting a full post, I'd like to gauge interest and get some initial feedback.

My main arguments are:

1. The paperclip maximizer oversimplifies AI motivations and neglects the potential for emergent ethics in advanced AI systems.

2. The doomer narrative often overlooks the possibility of collaborative human-AI relationships and the potential for AI to develop values aligned with human interests.

3. Current AI safety research and development practices are more nuanced and careful than the paperclip maximizer scenario suggests.

4. Technologies like brain-computer interfaces (e.g., the hypothetical Hypercortex "Membrane" BCI) could lead to human-AI symbiosis rather than conflict.

Questions for the community:

1. Have these critiques of the paperclip maximizer been thoroughly discussed here before? If so, could you point me to relevant posts?

2. What are the strongest counterarguments to these points from a LessWrong perspective?

3. Is there interest in a more detailed exploration of these ideas in a full post?

4. What aspects of this topic would be most valuable or interesting for the LessWrong community?

Any feedback or suggestions would be greatly appreciated. I want to ensure that if I do make a full post, it contributes meaningfully to the ongoing discussions here about AI alignment and safety.

Thank you for your time and insights!