trentmkelly/LessWrong-43k huggingface dataset:
How to avoid death by AI.
by Krantz — 2024-07-23T01:59:54.339Z

If we continue to scale ML systems, we will all perish.

If you disagree with that, you should probably be reading Eliezer's work instead.

If you are caught up on his work, then I have something new for you to think about.

One solution for solving this problem, would be to teach everyone on the planet the full set of reasons Eliezer holds this position. That would be the 'humanity grows up and decides not to build AI' possible future.

That seems like an intractable task. Most people do not care about those reasons. They don't care about learning what they can do to alter the trajectory of our future. They have no incentive to comprehend what you are wanting them to comprehend. We can't force knowledge into people heads.

How on Earth could we possible educate so many people about such a nuanced topic?

How could we verify that they really understand?

You give them something they do care about, money.

If you pay individuals to prove they understand something (in a way that works) you create a function that takes money as input and outputs public comprehension of the topics that are incentivized.

That's something we really need right now. It would serve the function of a 'fire alarm' not only for Eliezer's message, but for any information any person wants another person to consider.

This can be done by constructing a decentralized collective intelligence that rewards individuals for using it.

If you aren't familiar with collective intelligence, do not mistake it for artificial intelligence.

It is a completely different field in computer science. (https://cci.mit.edu/)

It is a paradigm shift from a self contained intelligent system that can evolve beyond us to a system that has humans as its parts and requires our interactions to grow.

It's a shift from trying to get a machine to twirl a pencil all by itself to getting a machine that can coordinate billions of people to solve much more complex problems.

That's intelligence also.

There are projects in this spirit (Community Notes, Wikipedia, Research Hub, predictive markets, Anthropic/collectiveintelligenceproject), but they fall short.

What we actually need is a place where people intentionally go to receive an education, read the news, verify the truth and get rewarded financially for it (without needing to invest capital).

I think people really underestimate the effect such a ...
ML Safety Research Advice - GabeM
by Gabe M — 2024-07-23T01:45:42.288Z
Link: https://open.substack.com/pub/mukobimusings/p/ml-safety-research-advice

This is my advice for careers in empirical ML research that might help AI safety (ML Safety). Other ways to improve AI safety, such as through AI governance and strategy, might be more impactful than ML safety research (I generally think they are). Skills can be complementary, so this advice might also help AI governance professionals build technical ML skills.

1. Career Advice

1.1 General Career Guides

Preventing an AI-related catastrophe - 80,000 Hours
A Survival Guide to a PhD (Andrej Karpathy)
How to pursue a career in technical AI alignment — EA Forum
AI safety technical research - Career review - 80,000 Hours
Beneficial AI Research Career Advice

2. Upskilling

2.1 Fundamental AI Safety Knowledge

AI Safety Fundamentals – BlueDot Impact
AI Safety, Ethics, and Society Textbook
Forming solid AI safety threat models helps you select impactful research ideas.

2.2 Speedrunning Technical Knowledge in 12 Hours

Requires some basic coding, calculus, and linear algebra knowledge
Build Intuition for ML (5h)

Essence of linear algebra - 3Blue1Brown (3h)
Neural networks - 3Blue1Brown (2h)

Backpropagation, the foundation of deep learning (3h)

Neural Networks: Backpropagation - CS 231N (0.5h)
The spelled-out intro to neural networks and backpropagation: building micrograd (2.5h)

Transformers and LLMs (4h)

[1hr Talk] Intro to Large Language Models (1h)
The Illustrated Transformer – Jay Alammar (1h)
Let's build GPT: from scratch, in code, spelled out. (2h)

2.3 How to Build Technical Skills

Traditionally, people take a couple of deep learning classes.

Stanford CS 224N | Natural Language Processing with Deep Learning (lecture videos)
Practical Deep Learning for Coders - Practical Deep Learning (fast.ai)

Other curricula that seem good:

Syllabus | Intro to ML Safety
Levelling Up in AI Safety Research Engineering [Public]
ARENA
Maybe also check out recent topical classes like this with public lecture recordings: CS 194/294-267 Understanding Large Language Models: Foundations and Safety

Beware of studying too much.

You should aim to understand the fundamentals of ML through 1 or 2 classes and then practice doing many manageable research projects with talented collaborators or a good mentor who can give you time to meet.
It’s easy to keep taking classes, but you tend to learn many more practical ML skills through practice doing real research projects.
You can also re

...
Ransomware Payments Should Require a Sin Tax
by Brian Bien — 2024-07-22T21:16:29.029Z

A tax could largely mitigate the growing ransomware problem. The following is a proposal for a scheduled, gradual increase in the tax rate on ransom payments:

The ransom demanded by an attacker is a function of the expected likelihood that the victim will pay that amount.
If a tax is legally enforced, the market will adapt: "optimal" ransoms will decrease to account for tax rates. For instance, a 100% tax would have the victim be legally bound to matching a $1M ransom payment with a $1M tax to account for negative externalities. Thus, an attacker, knowing this, could only reasonably expect to demand half the amount to get paid.
With each subsequent increase in tax rates, a market equilibrium will be reached, given sufficient time.
As tax rates approach the limit of effectiveness (in terms of compliance and enforceability), the funding of ransomware is significantly reduced and attacks are much less widespread.

Why not favor an outright ban?

A ban can also be considered in terms of immediately adopting an infinite (or very extreme) tax rate, and this argument focuses on the practical implications of a gradual rate increase. If immediate implementation of a one-billion percent tax rate seems unreasonable, then a ban is similarly unreasonable.
Consider those immediately-impacted after the policy takes effect. A tax can be gradually increased as the market adapts to each tier so as not to overly-penalize those who are immediately hit by ransomware following its implementation; each time the market adapts, a cost equilibrium is reached for the victim. By starting with a low tax, attackers will adopt their demands to the commensurate lower probability of payouts for higher amounts, while organizations would have time to adapt to this norm. The tax would be raised over time to avoid imposing a special burden to those victims who are immediately hit by the new tax after its initial rollout; the gradual raising of this tax would give time for attackers to absorb this new information and adopt to lower payments with their optimal demand amounts. At each tax rate increase, when the equilibrium is reached, the victim's cost is once-again equalized, yet the attacker's gain is again further-reduced.
Compliance: a tax, similar to a ban, accounts for the negative externalities. A ban would reduce negative externalities, but due to the increased severity for victims, complianc

...
The Elusive Root Cause of Schizophrenia - Thesis Introduction Only
by kareempforbes — 2024-07-22T20:24:43.155Z

This review paper aims to examine and explain the root cause of schizophrenia through a theoretical model based on Information Technology (IT) processing principles. The model conceptualizes the brain’s processing ability and capacity in terms of IT processing loads. Chronic trauma and stress degrade the brain’s processing capacity, leading to systemic neural overload. This sustained overload diminishes the brain’s ability to process information and sensory data effectively, resulting in the hallucinations, delusions, and psychosis characteristic of schizophrenia.

The likelihood of developing mental illness, including schizophrenia, can be described through an equation that compares the brain’s processing capacity to the load placed upon it. A value of 1 indicates a state of homeostasis, where the brain’s capacity is equal to the load it must handle. A value higher than 1 suggests an overload, while a value lower than 1 means the brain’s capacity exceeds the load.

When the load exceeds the brain’s capacity, mental illness occurs. If this excessive load is sustained over time, it can lead to schizophrenia.

Brain Computing Function Health/Capacity = Biological Age Risk + Brain Logical Organization + Brain Developmental Health + Brain Physical Health + Brain Neurochemical Health + Brain Cognitive Reserve

This should be less than:

Required Processing Load = Total Physiological Computing Demands or Stress (Sensory Ability + Sensitivity Factor + Cumulative Trauma Load) * (Current/Sustained Environment Sensory Load) * Time

We can describe the relationship of these two factors by their relative state of balance or imbalance. When the brain’s capacity is less than the required processing load, the risk of mental illness increases.

Optimal Well-being (capacity >>> load)
Healthy Balance (capacity > load)
Homeostasis/Equilibrium (capacity = load)
Mental Strain (capacity < load)
Severe Overload (capacity <<< load)

Categorization Explanation:

Optimal Well-being (capacity >>> load)

This state represents high mental healthiness and happiness, where the brain’s capacity far exceeds the processing load.
Individuals experience a sense of well-being, satisfaction, and contentment.
There’s ample cognitive reserve to handle stressors and challenges, leading to positive emotions and overall mental wellness.

2. Healthy Balance (capacity > load)

This state signifie

...
Is Chinese AGI a valid concern for the USA?
by sammyboiz — 2024-07-22T20:21:57.800Z

I) Introduction: The race as of today

The California AI bill SB-1047 has ignited

a fierce debate

among policymakers and the public. Safety concerns are at the forefront of arguments for regulation, with some urging caution before advancing AI technologies. Opponents of regulation argue that such measures could hinder AI progress, posing their own safety risks by potentially allowing China to overtake the US in AI development. The recognition of an AI race between the US and China is widespread among American policymakers, with a significant focus on maintaining AI supremacy as a matter of national security. A US government report highlights the potential dangers of AI-enabled capabilities, stating they could be used for threatening critical infrastructure, amplify disinformation campaigns, and wage war. The report warns that global stability and nuclear deterrence could be compromised by these AI applications. Beyond these immediate threats, the prospect of AGI (Artificial General Intelligence) or transformative AI, capable of revolutionizing society, also poses a significant concern for the US. However, less attention has been given to the existential risk of uncontrolled AI development within the US, exacerbated by competitive pressures. According to a

2023 Expert Survey

on Progress in AI conducted by AI Impacts, the median response from 655 AI experts indicated a 19.4% probability that human inability to control future advanced AI systems could lead to human extinction or severe disempowerment. To address these concerns, an accurate assessment of Chinese AGI efforts is crucial. Policymakers must determine whether to prioritize outpacing China in the race to AGI or focus on governing domestic AI labs and guarding against other AI threats from the PRC (People's Republic of China). This essay examines China's competitiveness in general intelligence models and its potential to compete in the AGI race. After analyzing the AI market, particularly the capabilities of Chinese large language models (LLMs), it becomes evident that while Chinese AI development is impressive, its LLMs lag behind. For the future, despite the challenges China faces in scaling up toward AGI, it remains a potential competitor in the race towards transformative AI. This analysis underscores the need for a balanced approach in US AI policy, weighing the importance of maintaining a lead...
Trying to understand Hanson's Cultural Drift argument
by Kemp — 2024-07-22T20:20:32.734Z

At 2024's Manifest, Robin Hanson gave a talk (in his usual sweeping polymathic style) on cultural drift - a phenomenon for which he thinks "there's going to be hell to pay", and about which he is "scared, because this is a really big fundamental problem".

Watch the recording of the 30:00 minute talk here (followed by Q&A).

His argument is roughly the following:

Evolution between species is more important (more adaptive) than evolution within species.
The same is true of cultural evolution.
In the last couple hundred years, there has been a great reduction of evolution between cultures. There are now far fewer cultures, and all of the changes are happening within these cultures.
There is less famine and war now, i.e. much less selection pressure.
Therefore, our cultural changes are not adaptive - they're not responsive to selection pressures. They're "drifting off the rails" (hence cultural drift).
Dropping fertility rates are a consequence of this drift: "I interpret this [dropping fertility] as our main macro-world culture becoming less adaptive..."

Hanson takes this cultural drift to be a problem. But here I become confused. From what perspective is cultural drift a problem? The obvious candidate is the POV of our culture - if it is not adaptive, it will likely die out. We don't want our culture to die out. But Hanson seems to want to appeal to a perspective outside of our culture:

"There are two levels of thinking about culture, one is from the inside and one is from the outside. I'm trying to get you to see it from the outside..."
"I'm here scared, because this is a really big fundamental problem. This is Humanity's main engine..."

It seems Hanson wants to say that this is a problem for Humanity, not just for our culture. However, cultures dying out and being replaced by more adaptive ones is exactly what we would expect of a healthy evolutionary system. If our liberal western culture (whatever that means) goes extinct and is replaced by the Amish or Mennonite cultures (as Hanson suggests it might), then this looks like selection working "as intended".

Despite Hanson saying that we should see things "from the outside", all of the specific worries he cites are quite clearly from inside our culture:

"We would then plausibly have several centuries of a declining population with very little innovation."
"...the world will become less liberal."
"the Amish and Mennonites

...
Efficient Dictionary Learning with Switch Sparse Autoencoders
by Anish Mudide — 2024-07-22T18:45:53.502Z

Produced as part of the ML Alignment & Theory Scholars Program - Summer 2024 Cohort

0. Summary

To recover all the relevant features from a superintelligent language model, we will likely need to scale sparse autoencoders (SAEs) to billions of features. Using current architectures, training extremely wide SAEs across multiple layers and sublayers at various sparsity levels is computationally intractable. Conditional computation has been used to scale transformers (Fedus et al.) to trillions of parameters while retaining computational efficiency. We introduce the Switch SAE, a novel architecture that leverages conditional computation to efficiently scale SAEs to many more features.

1. Introduction

The internal computations of large language models are inscrutable to humans. We can observe the inputs and the outputs, as well as every intermediate step in between, and yet, we have little to no sense of what the model is actually doing. For example, is the model inserting security vulnerabilities or backdoors into the code that it writes? Is the model lying, deceiving or seeking power? Deploying a superintelligent model into the real world without being aware of when these dangerous capabilities may arise leaves humanity vulnerable. Mechanistic interpretability (Olah et al.) aims to open the black-box of neural networks and rigorously explain the underlying computations. Early attempts to identify the behavior of individual neurons were thwarted by polysemanticity, the phenomenon in which a single neuron is activated by several unrelated features (Olah et al.). Language models must pack an extremely vast amount of information (e.g., the entire internet) within a limited capacity, encouraging the model to rely on superposition to represent many more features than there are dimensions in the model state (Elhage et al.).

Sharkey et al. and Cunningham et al. propose to disentangle superimposed model representations into monosemantic, cleanly interpretable features by training unsupervised sparse autoencoders (SAEs) on intermediate language model activations. Recent work (Templeton et al., Gao et al.) has focused on scaling sparse autoencoders to frontier language models such as Claude 3 Sonnet and GPT-4. Despite scaling SAEs to 34 million features, Templeton et al. estimate that they are likely orders of magnitude short of capturing all features. Furthermore, Gao et a...
Analyzing DeepMind's Probabilistic Methods for Evaluating Agent Capabilities
by Axel Højmark — 2024-07-22T16:17:07.665Z

Produced as part of the MATS Program Summer 2024 Cohort. The project is supervised by Marius Hobbhahn and Jérémy Scheurer
Update: See also our paper on this topic admitted to the NeurIPS 2024 SoLaR Workshop.

Introduction

To mitigate risks from future AI systems, we need to assess their capabilities accurately. Ideally, we would have rigorous methods to upper bound the probability of a model having dangerous capabilities, even if these capabilities are not yet present or easily elicited.

The paper “Evaluating Frontier Models for Dangerous Capabilities” by Phuong et al. 2024 is a recent contribution to this field from DeepMind. It proposes new methods that aim to estimate, as well as upper-bound the probability of large language models being able to successfully engage in persuasion, deception, cybersecurity, self-proliferation, or self-reasoning. This post presents our initial empirical and theoretical findings on the applicability of these methods.

Their proposed methods have several desirable properties. Instead of repeatedly running the entire task end-to-end, the authors introduce milestones. Milestones break down a task and provide estimates of partial progress, which can reduce variance in overall capability assessments. The expert best-of-N method uses expert guidance to elicit rare behaviors and quantifies the expert assistance as a proxy for the model's independent performance on the task.

However, we find that relying on milestones tends to underestimate the overall task success probability for most realistic tasks. Additionally, the expert best-of-N method fails to provide values directly correlated with the probability of task success, making its outputs less applicable to real-world scenarios. We therefore propose an alternative approach to the expert best-of-N method, which retains its advantages while providing more calibrated results. Except for the end-to-end method, we currently feel that no method presented in this post would allow us to reliably estimate or upper bound the success probability for realistic tasks and thus should not be used for critical decisions.

The overarching aim of our MATS project is to uncover agent scaling trends, allowing the AI safety community to better predict the performance of future LLM agents from characteristics such as training compute, scaffolding used for agents, or benchmark ...
The Garden of Eden
by Alexander Turok — 2024-07-22T16:07:42.509Z

[This is the text of the sermon given by Pastor James Windrow on Sunday, July 14, 2024.]

Greetings, brothers and sisters. Today, we reflect on humanity’s origin in the Garden of Eden and its fall and devolution thereafter.

When Christians used to think about the Garden of Eden, it was common to imagine it as an almost metaphysical place, existing on another plane of reality, permanently inaccessible to us “fallen” people. The Biblical text is clear that it is on Earth; it even tells us its approximate location. Yet many people couldn’t quite accept that. They were willing to accept the existence of God, Angels, the Tree of Knowledge of Good and Evil and the Tree of Life, but only when they existed “over there,” away from the “real” world. Even today, when it’s been scientifically proven that the Bible is true, some still think this way.

You know who didn’t think this way? Our ancestors who lived before the Flood. From their recovered artifacts, we know they could sail up the river to Garden and see for themselves the cherubim and the flaming sword. Some deranged people even tried to fight the cherubim so they could seize the fruit of the Tree of Life.

We know where the Garden is. We’ve excavated it and found … nothing. No cherubim, no flaming sword, no wall that pre-Flood texts state surrounded the Garden. These things were presumably destroyed or taken to heaven during the Flood. It is as if God did not want us to know anything more about the Garden than what he deigned to tell us in the Biblical text.

In the Garden were Adam, Eve, and all the animals, including the serpent. God granted mankind dominion over the animals, for he was made in God’s image. While only Adam, Eve, and the Serpent are shown to speak, we now know, based on the recovered remains of animals underneath the flood layer, that all animals larger than rats had brains large enough for intelligent thought and vocal tracts that enabled them to speak human language. Some see a contradiction between this evidence and the Biblical text, but no such contradiction exists. Genesis states that the serpent was “more crafty” than any other animal; this does not imply that the other animals lacked the ability to speak.

Then, as we all know, Adam, Eve, and the serpent rebelled against God. I will read out the verses in which these three receive their punishment.

[3:14] The LORD God said...