trentmkelly/LessWrong-43k huggingface dataset:
Jesse Hoogland on Developmental Interpretability and Singular Learning Theory
by Michaël Trazzi — 2023-07-06T15:46:00.116Z
Link: https://theinsideview.ai/jesse

Jesse Hoogland is a research assistant at

David Krueger's lab

in Cambridge studying AI Safety who has recently been publishing on LessWrong about how to apply Singular Learning Theory to Alignment, and even organized some workshop in Berkeley last week around this.

I thought it made sense to interview him to have some high-level overview of Singular Learning Theory (and other more general approaches like developmental interpretability).

Below are some highlighted quotes from our conversation (available on Youtube,

Spotify

,

Google Podcast

,

Apple Podcast

). For the full context for each of these quotes, you can find the accompanying

transcript

.

Interpreting Neural Networks: The Phase Transition View

Studying Phase Transitions Could Help Detect Deception

"We want to be able to know when these dangerous capabilities are first acquired because it might be too late. They might become sort of stuck and crystallized and hard to get rid of. And so we want to understand how dangerous capabilities, how misaligned values develop over the course of training. Phase transitions seem particularly relevant for that because they represent kind of the most important structural changes, the qualitative changes in the shape of these models internals.

Now, beyond that, another reason we’re interested in phase transitions is that phase transitions in physics are understood to be a kind of point of contact between the microscopic world and the macroscopic world. So it’s a point where you have more control over the behavior of a system than you normally do. That seems relevant to us from a safety engineering perspective. Why do you have more control in a physical system during phase transitions?" (context)

A Concrete Example of Phase Transition In Physics and an analogous example inside of neural networks

"Jesse: If you heat a magnet to a high enough temperature, then it’s no longer a magnet. It no longer has an overall magnetization. And so if you bring another magnet to it, they won’t stick. But if you cool it down, at some point it reaches this Curie temperature. If you push it lower, then it will become magnetized. So the entire thing will all of a sudden get a direction. It’ll have a north pole and a south pole. So the thing is though, like, which direction will that north pole or south pole be? And so it turns out that you only need an infinitesimally small perturba

...
Progress links and tweets, 2023-07-06:
Terraformer Mark One, Israeli water management, & more
by jasoncrawford — 2023-07-06T15:35:22.591Z
Link: https://rootsofprogress.org/links-and-tweets-2023-07-06

Opportunities

Free virtual course for people in tech who want to work on biology (via @kulesatony)

News

A background of gravitational waves pervades the cosmos (via @QuantaMagazine)
The Terraformer makes cheap natural gas from sunlight & air (via @TerraformIndies)

Obituaries

Henry Petroski, “America’s poet laureate of technology,” author of To Engineer is Human, The Evolution of Useful Things, and other books
John Goodenough, age 100 (!) “His discovery led to the wireless revolution and put electronic devices in the hands of people worldwide”

Links

Great SpaceX video about iterating towards reusable rockets (via @pronounced_kyle)
Scott Alexander on the “illusion of moral decline” (but see Matt Clancy’s pushback)
Three auto maintenance philosophies, from @stewartbrand’s book in progress
As the opportunity cost of time increases, so does our concern about wasting time
How to do great work (by @paulg)
The “undercurrent of tacit knowledge” in any scientific literature (by Tyler Cowen)
GLP-1 drugs are amazing for weight loss, says Derek Lowe (via @curiouswavefn)
Why you can ignore the “aspartame causes cancer” headlines (by @StuartJRitchie). Related, the IARC puts bacon in the same category as plutonium

Queries

What use cases is hydrogen best for? And what are the front-runners for long-term energy storage?
What are the best films about capitalism, business, and market economics?
Why does DNA use base 4 rather than base 2 for encoding information?
How to roll back as much bad regulation as possible in one tweet?
Any indicators of misuse threat of adversarially fine-tune-able 10^25 LLMs?
What intellectuals refocused New Left types towards remaking the physical world?

Quotes

Real history is not politics but the story of those who make a better life
Academics and nonprofits act as if someone else will translate their papers into policy

Tweets & threads

The saga of Israeli water management
A biodefense roadmap from Kevin Esvelt. Excellent work
Chicago went all in on railroads and Cincinnati went all in on canals
A path to prosperity for small cities within one hour of a major city/airport
Update from prediction market Kalshi on regulated election markets
I push back against Dustin Moskowitz’s list of AI safety “strawmen”
Many good ideas are killed for reasons unrelated to their promise
As we make progress on social problems, we broaden the definition of the problem. A case of the Shirky Princi

...
Towards Non-Panopticon AI Alignment
by Logan Zoellner — 2023-07-06T15:29:39.705Z

List of reasons to be concerned about AI are notoriously hard to read. Worse, most of them include an equal mix of bad arguments (Foom is likely, AI development will mimic human evolution, the AI will unbox itself by solving the protein folding problem) and good ones.

So, here is my very short (hopefully very easy to read) list of reasons why AI could be dangerous:

It might be hard to explain human values to AI
Even if AI understands Human values, it might do something bad by accident
Even if it is possible to build Safe AI, people might deploy an unsafe AI because we are all racing
Even if Safe AI is widely deployed, some people might do very bad things since AI gives us all superpowers

We can call the first 2 points the "individual alignment problem", because they are problems anyone building an AGI will have to solve. And we can call points 3. and 4. "cooperative AGI problems" because even if we know how to to build safe AGI, the outcome could still be bad if we fail to cooperate.

For this post, I want to focus on problems 3. and 4. If it helps, imagine that we live in a world where future variants of RLAIF and Corrigible AI have perfectly solved problems 1. and 2.

The Global Panopticon

There is a very common (but ultimately flawed) approach that frequently comes up when discussing solutions to 3. and 4.

The naive solution is:

Come up with a limit defining the boundary between "safe" and "unsafe" AI models
Set up a global monitoring/enforcement regime to ensure no one violates this boundary

naive cooperative alignment solution

We can see many examples of this "define and monitor" solution. For example calling for monitoring on large GPU clusters or drone strikes on rogue data centers. Indeed, these approaches are often explicitly models after controls on nuclear material, a regime which should be most famous for its complete failure to prevent nations from acquiring nuclear weapons. Define and monitor is the go-to solution for most global coordination problems such as CFCs, Greenhouse Gasses, missiles, military equipment and genocide.

In addition to failing to achieve its stated goals in almost all of these cases (CFCs being the notable exception), another downside of a global define-and-monitor strategy is its implications for privacy. As Moore's Law and algorithmic improvements inexorably move forward, the "safe" ...
A Defense of Work on Mathematical AI Safety
by Davidmanheim — 2023-07-06T14:15:21.074Z
Link: https://forum.effectivealtruism.org/posts/NKMxC2nA47uuhFm8x/a-defense-of-work-on-mathematical-ai-safety

AI Safety was, a decade ago, nearly synonymous with obscure mathematical investigations of hypothetical agentic systems. Fortunately or unfortunately, this has largely been overtaken by events; the successes of machine learning and the promise, or threat, of large language models has pushed thoughts of mathematics aside for many in the “AI Safety” community. The once pre-eminent advocate of this class of “agent foundations” research for AI safety, Eliezer Yudkowsky, has more recently said that timelines are too short to allow this agenda to have a significant impact. This conclusion seems at best premature.

Foundational research is useful for prosaic alignment

First, the value of foundational and mathematical research can be synergistic with both technical progress on safety, and with insight into how and where safety is critical. Many machine learning research agendas for safety are investigating issues identified years earlier by foundational research, and are at least partly informed by that research. Current mathematical research could play a similar role in the coming years, as more funding and research are increasingly available for safety. We have also repeatedly seen the importance of foundational research arguments in discussions of policy, from Bostrom’s book to policy discussions at OpenAI, Anthropic, and DeepMind. These connections may be more conceptual than direct, but they are still relevant.

Long timelines are possible

Second, timelines are uncertain. If timelines based on technical progress are short, many claim that we have years not decades until safety must be solved. But this assumes that policy and governance approaches fail, and that we therefore need a full technical solution in the short term. It also seems likely that short timelines make all approaches less likely to succeed. On the other hand, if timelines for technical progress are longer, fundamental advances in understanding, such as those provided by more foundational research, are even more likely to assist in finding or building more technical routes toward safer systems.

Aligning AGI ≠ aligning ASI

Third, even if safety research is successful at “aligning” AGI systems, both via policy and technical solutions, the challenges of ASI (Artificial SuperIntelligence) still loom large. One critical claim of AI-risk skeptics is that recursive self-improvement is speculative, so we do not...
Understanding the two most common mental health problems in the world
by spencerg — 2023-07-06T14:06:25.968Z
Link: https://www.clearerthinking.org/post/understanding-the-two-most-common-mental-health-problems-in-the-world

Co-authored with Amanda Metskas

This is a linkpost from ClearerThinking.org. We've included some excerpts of the article below, but you can read the full post here.

Chances are, your life has been impacted by anxiety disorders or depression, either through your direct experience or through the impact they have had on your loved ones. Anxiety and depression are the two most common mental health conditions in the world, but they are frequently misunderstood.

Previous EA Mental Health surveys (2023, 2021, and 2018) have also highlighted the importance of these topics to people in the community who take the surveys.

In this data-based essay, we aim to help you better understand anxiety and depression, as well as the hidden links between them. By improving your understanding of these disorders, you may find it easier to recognize anxiety and depression in yourself and be more effective at supporting people in your life who experience these conditions. This infographic summarizes some of our interesting findings about the differences and similarities between anxiety and depression (

click here to see the infographic at full size

).

The scale of both anxiety and depression is vast: the

World Health Organization

estimates that 301 million people worldwide suffer from an anxiety disorder, and 280 million people worldwide suffer from depression. Worldwide, depression ranks as the second largest cause of disability, and anxiety ranks eighth, according to

analyses of the most recent Global Burden of Disease study

. And yet, despite their prevalence and severe impacts, humanity’s scientific understanding has a substantial way to go to fully understand and highly reliably treat these conditions. Improved treatment and management techniques could make a huge difference in the quality of life of hundreds of millions of people around the world. These astounding statistics have motivated us to run our own studies investigating how these conditions work and how they relate to each other. This article will explain what we found!

Overlapping Disorders

A major obstacle to understanding anxiety and depression is that they often go together - many people who experience one also experience the other. Approximately

45% of people

who experience a depressive disorder in their lifetime also experience an anxiety disorder, and these often occur durin...
Announcing the EA Archive
by Aaron Bergman — 2023-07-06T13:49:17.387Z

About the EA Archive

The EA Archive is a project to preserve resources related to effective altruism in case of a sub-existential catastrophe such as nuclear war.

Its more specific, downstream motivating aim is to increase the likelihood that a movement akin to EA (i.e., one that may go by a different name and be essentially discontinuous with the current movement, but share the broad goal of using evidence and reason to do good) survives, reemerges, and/or flourishes without having to re-invent the wheel, so to speak.

It is a work in progress, and some of the subfolders at the referenced Google Drive are already slightly out of date.

Theory of Change

The theory of change is simple, if not very cheerful to describe: if copies of this information exist in many places around the world, on devices owned by many different people, it is more likely that at least one copy will remain accessible after, say, a war that kills most of the world's population.

Structure

As shown in the screenshot, there are three folders. The smallest one, "Main content," contains html, pdf, and other static, text-based files. It is by far the most important to download.

If for whatever reason space isn't an issue and you'd like to download the larger folders to, that would be great too.

I will post a shortform quick take (at least) when there's been a major enough revision to warrant me asking for people to download a new version.

How you can help

1) Download!

Click here to download the core content (.zip, 2GB)

Browse and download the complete archive (up to 51GB uncompressed)

This project depends on people like you downloading and storing the Archive on a computer or flash drive that you personally have physical access to, especially if you live in any of the following:

Southeast Asia + Pacific (esp. New Zealand)
South and Central Africa
Northern Europe (esp. Iceland)
Latin America, Mexico City and south (esp. Ecuador, Colombia, and Argentina)
Any very rural area, anywhere

If you live in one of the shaded areas (green or blue), I would love to buy you a flash drive to make this less annoying and/or enable you to store copies in multiple locations, so please get in touch via the Google Form, DM, or any other method.

2) Suggest/submit, and provide feedback

Currently, the limiting factor on the Archive's contents is my ability and willingness to find identify relevant resources and then scrape or ...
Agency begets agency
by Richard_Ngo — 2023-07-06T13:08:44.318Z

The second way of cultivating excitement-based motivation I'll discuss in this sequence is cultivating agency, by which I mean the ability to try to shape the world. The key idea I want to convey here is that the feeling of agency is in many cases a self-fulfilling prophecy, because the hardest part of many things is mustering the energy and initiative to get started. If you think of yourself as agentic, then you’ll continually be looking out for opportunities, and approach the world with a “why not?” attitude. That’ll give you more chances to do well, which will give you more evidence that you’re the sort of person who can make stuff happen, and reinforce your sense of agency.

Why don’t people have that sense by default? One reason is that most people have experienced a huge amount of social pressure towards conformity, especially as children. Doing unusual or novel things often leads to criticism or ridicule, which ingrains a sense of learned helplessness about defying consensus. This is especially strong for teenagers, most of whom spend a huge amount of effort trying to be liked and fit in. And it’s not just enough to be willing to stand out: your success might make others look bad, which creates incentives for them to

”cut you down to size”

, or deter you from trying in the first place (e.g. by mocking anyone who cares enough about something to try hard to achieve it).

A more intellectual justification for lacking agency comes from

epistemic modesty

: the view that you should pay relatively little attention to your own view on most topics, and defer instead to other people. One intuition that often motivates epistemic modesty is the idea that

markets are often efficient

, in the sense that you can’t consistently beat them. However, people often apply epistemic modesty to conclude that they shouldn’t trust their

inside view

even in domains that are very different from markets: founding a startup, or doing research, or reasoning about politics, or even finding a partner. In those domains, epistemic modesty often involves thinking that you’re not so special: if others disagree, then there’s little reason to think you’re mistaken rather than them; and if others haven’t taken an opportunity, then there’s probably some reason you shouldn’t (or can’t) either.

Certainly there are some people who should pay more attention to epistemic modesty. Howe...
AI #19: Hofstadter, Sutskever, Leike
by Zvi — 2023-07-06T12:50:05.037Z

The big news of the week is OpenAI introducing the Superalignment Taskforce. OpenAI is pledging 20% of compute secured to date towards a four year effort to solve the core technical challenges of superintelligence alignment. They plan to share the results broadly. Co-founder and OpenAI Chief Scientist Ilya Sutskever will lead the effort alongside Head of Alignment Jan Leike, and they are hiring to fill out the team.

That is serious firepower. The devil, as always, is in the details. Will this be an alignment effort that can work, an alignment effort that cannot possibly work, an initial doomed effort capable of pivoting, or a capabilities push in alignment clothing?

It is hard to know. I will be covering these questions in a post soon, and want to take the time to process and get it right. In the meantime, the weekly post covers the many other things that happened in the past seven days.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Commercial LLMs keep the edge.

Language Models Don’t Offer Mundane Utility. Can you?

Fun With Image Generation. Valve says: Keep all that fun away from Steam.

They Took Our Jobs. Then they came for America’s finest news source.

Introducing. If only it were as easy to forget.

Reinforcement Learning By Humans From Feedback. United we grok.

Costs are not Benefits. Easy mistake to make. Many such cases.

Quiet Speculations. Questions that are not so infrequently asked.

The Quest for Sane Regulation. Some small good signs.

Another Open Letter. EU CEOs call on EU to not act like the EU.

The Week in Audio. Odd Lots offers AI fun.

No One Would Ever Be So Stupid As To. Is that what you think?

Safely Aligning a Smarter than Human AI is Difficult. Formal verification?

Rhetorical Innovation. The latest crop of explanations and resources.

People Are Worried About AI Killing Everyone. Including Douglas Hofstadter.

The Lighter Side. Actual progress.

Language Models Offer Mundane Utility

Open source models often do well on open source benchmarks. So do foreign efforts like Falcon or Ernie. When it comes to actual mundane utility, or tests that were not anticipated in advance, the answer seems to come back somewhat differently.

Lmsys.org: Quick note – we’ve transitioned from the deprecated vicuna benchmark to a more advanced MT-bench,

...
Do you feel that AGI Alignment could be achieved in a Type 0 civilization?
by Super AGI — 2023-07-06T04:52:57.819Z

Or, would a higher level of Global coordination and cooperation be required?

E.g.

- Loosely Regulated Capitalism: Capitalist societies often prioritize profit and short-term gains, which might discourage thorough testing and careful deployment of AI systems. There could be a race to market, potentially leading to the deployment of AI systems that are not fully aligned with Human values. Furthermore, capitalism often encourages competition, which could discourage the kind of global cooperation that may be necessary for safe AI alignment.

- Non-Aligned Countries Leading to an Arms Race: If AI development and regulation are not globally aligned, it could lead to an arms race where countries or corporations try to create the most powerful AI first, potentially neglecting safety protocols and alignment principles. This could lead to the development of AI that is not adequately controlled or understood by Humans, threatening global stability.

- High Energy Cost: The high energy demands of advanced AI systems could strain existing resources, potentially leading to conflict over these resources and jeopardizing the alignment process.

- Data Bias and Privacy: Unregulated use of AI could lead to invasion of privacy and perpetuate bias in decision-making, from hiring to law enforcement.

- Climate Change: High-energy computational processes contribute to climate change. As AI models grow larger, the energy required to train them increases. This could exacerbate global warming and have profound societal impacts, which might in turn make AI alignment more challenging by adding to the complexity of Human values that AI systems need to understand and align with.

- Social Inequality: In a situation where the benefits of AI are not equally distributed, social inequality can be exacerbated. A misaligned AI system could lead to a further concentration of wealth and power in the hands of a few, potentially causing societal unrest. Moreover, increased inequality could lead to a larger, more desperate Human population seeking employment and resources, which could in turn be exploited for the mass production of robots with Human-like dexterity and skills. For more on this, see Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment
Open Thread - July 2023
by Ruby — 2023-07-06T04:50:06.735Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.