trentmkelly/LessWrong-43k huggingface dataset:
MIRI's May 2024 Newsletter
by Harlan — 2024-05-15T00:13:30.153Z
Link: https://intelligence.org/2024/05/14/may-2024-newsletter/

Update (5-15-2024): I wrote that “it appears that not all of the leading AI labs are honoring the voluntary agreements they made at [AI Safety Summit],” citing a Politico article. However, after seeing more discussion about it (e.g. here), I am now highly uncertain about whether the labs made specific commitments, what those commitments were, and whether commitments were broken. These seem like important questions, so I hope that we can get more clarity.
MIRI updates:

MIRI is shutting down the Visible Thoughts Project.

We originally

announced

the project in November of 2021. At the time we were hoping we could build a new type of data set for training models to exhibit more of their inner workings. MIRI leadership is pessimistic about humanity’s ability to solve the alignment problem in time, but this was an idea that seemed relatively promising to us, albeit still a longshot.
We also hoped that the $1+ million bounty on the project might attract someone who could build an organization to build the data set. Many of MIRI’s ambitions are bottlenecked on executive capacity, and we hoped that we might find individuals (and/or a process) that could help us spin up more projects without requiring a large amount of oversight from MIRI leadership.
Neither hope played out, and in the intervening time, the ML field has moved on. (ML is a fast-moving field, and alignment researchers are working on a deadline; a data set we’d find useful if we could start working with it in 2022 isn’t necessarily still useful if it would only become available 2+ years later.) We would like to thank the many writers and other support staff who contributed over the last two and a half years.

Mitchell Howe and Joe Rogero joined the comms team as writers. Mitch is a longtime MIRI supporter with a background in education, and Joe is a former reliability engineer who has facilitated courses for

BlueDot Impact

. We’re excited to have their help in transmitting MIRI’s views to a broad audience.
Additionally, Daniel Filan will soon begin working with MIRI’s new Technical Governance Team part-time as a technical writer. Daniel is the host of two podcasts:

AXRP

, and The Filan Cabinet. As a technical writer, Daniel will help to scale up our research output and make the Technical Governance Team’s research legible to key audiences.
The Technical Governance Team submitted responses to the

...
The Greater Goal: Sharing Knowledge with the Cosmos
by pda.everyday — 2024-05-14T22:46:26.737Z

Our planet faces numerous risks, from natural disasters to technological threats. By transmitting AI models as data into space now, we could proactively safeguard our intellectual heritage against potential future catastrophes that could wipe out intelligent life on Earth. We could enable potential existing extraterrestrial civilizations to decode and utilize our knowledge, contributing to the collective intelligence of the universe. This altruistic vision recognizes that the value of knowledge increases when shared, transcending the boundaries of our planet and species. This approach not only safeguards our intellectual heritage but also contributes to the preservation and dissemination of intelligence across the cosmos.

A Speculative Proposal to Safeguard Our Knowledge and Intelligence

Sending AI models into the Kosmos ensures that human knowledge and intelligence are preserved for potentially billions of years. Unlike static archives, AI models can adapt, learn, and interact with future civilizations if they mange to run them, making them a dynamic and invaluable resource.

Ethics

It is crucial to address several ethical considerations. Firstly, the decision to transmit such information should be made with global consensus. This means that the project should involve input and approval from a wide range of stakeholders around the world, ensuring that the transmitted knowledge accurately represents diverse cultures and knowledge systems. This inclusivity is essential to truly encapsulate the breadth of human civilization. Secondly, we must carefully evaluate the risks associated with revealing our technological capabilities to unknown civilizations. While the intention is to preserve and share knowledge, it is important to consider the potential consequences of disclosing advanced technological information to entities whose intentions and capabilities are unknown. This risk assessment should guide the selection of information that is transmitted and the manner in which it is encoded.

Lastly, it is vital to clearly define the purpose and intent behind the transmission. The goal is to create a lasting legacy of human intellect that can contribute to the collective intelligence of the universe, rather than to initiate interactions that could be misinterpreted or lead to unintended consequences.
Despite these considerations, the potential benefits of this projec...
Teaching CS During Take-Off
by andrew carle — 2024-05-14T22:45:39.447Z

I stayed up too late collecting way-past-deadline papers and writing report cards. When I woke up at 6, this anxious email from one of my g11 Computer Science students was already in my Inbox.

Student: Hello Mr. Carle, I hope you've slept well; I haven't.

I've been seeing a lot of new media regarding how developed AI has become in software programming, most relevantly videos about NVIDIA's new artificial intelligence software developer, Devin.

Things like these are almost disheartening for me to see as I try (and struggle) to get better at coding and developing software. It feels like I'll never use the information that I learn in your class outside of high school because I can just ask an AI to write complex programs, and it will do it much faster than I would.

I'd like to know what your thoughts on this are. Do you think AI will replace human software developers, as NVIDIA claims it will?

My response: Buddy, that is a big question for 5:15 am.

First AI horizon thoughts:

Software development as a field will look incredibly different in 10 years.
My priors say that MOST of human intellectual+economic activity will ALSO be radically different in 10 years.
I have a very small p(doom) for the 10 year horizon. That means I don't expect human-equivalent AGIs to completely disrupt human civilisation within 10 years.
The delta between how fast AI will affect software engineering and how fast AI will transform other (roughly speaking) white collar careers is relatively small. That means I think the AI affect on say, hedge fund management and software engineering to be similar.

Then some priors I have for teaching IB Computer Science in the middle of this take-off:

I don't think becoming a software engineer is the modal outcome for IBCS students
I believe that most long term personal utility from IBCS (or any other intro CS exposure) comes from shifting a student's mental model of how the modern social and economic system interacts with / depends on these technologies.
While the modern Ai tools are light years beyond the simple Von Neumann CPU models and intro Python we're studying, it does address the foundations of those systems. Similarly, HL Analysis and HL Physics don't cover anything about the math and physics that underpin these huge ML systems, but that foundation IS there. You can't approach the superstructure without it.

So, in summary, if your concern is "the world seems...
A Narrative History of Environmentalism's Partisanship
by Jeffrey Heninger — 2024-05-14T16:51:01.029Z
Link: https://blog.aiimpacts.org/p/a-narrative-history-of-environmentalisms

This is the second in a sequence of four posts taken from my recent report: Why Did Environmentalism Become Partisan?

Many of the specific claims made here are investigated in the full report. If you want to know more about how fossil fuel companies’ campaign contributions, the partisan lean of academia, or newspapers’ reporting on climate change have changed since 1980, the information is there.

Introduction

Environmentalism in the United States today is unusually partisan, compared to other issues, countries, or even the United States in the 1980s. This contingency suggests that the explanation centers on the choices of individual decision makers, not on broad structural or ideological factors that would be consistent across many countries and times.

This post describes the history of how particular partisan alliances were made involving the environmental movement between 1980 and 2008. Since individual decisions are central to understanding why this happened, this history is best presented as a narrative following the key people and organizations.

Environmentalism in the Reagan Era

In the wake of the New Deal, the Republican Party acquiesced to the government having a larger role in society than it had had before the Great Depression.[1] Republican presidents would sometimes support increased government spending and regulation. This is apparent in environmental policy: Nixon was involved in several major pieces of environmental legislation and created the EPA in the executive branch.

The election of Reagan in 1980 reoriented the Republican Party. It would now advocate for a smaller government: less (non-military) spending, lower taxes, and less regulation. The free market would provide many of the services that had previously been done by the government. Thatcher’s election as Prime Minister had similar results for the Conservative Party in the UK.

This might seem like it would cause a deep ideological conflict: Environmentalists advocated for regulations on private enterprise and international cooperation on policy, while the Republican Party preferred private and local action. However, this is not what we observe with either Reagan or Thatcher.

The conservative leaders in the US and UK supported environmentalism, even when it involved international regulations. The clearest example of this is the Montreal Protocol on Substances That Deplete the Ozone Layer ...
How much AI inference can we do?
by Benjamin_Todd — 2024-05-14T15:10:58.539Z
Link: https://benjamintodd.substack.com/p/how-much-ai-inference-can-we-do

Suppose you have a bunch of GPUs. How many LLM forward passes can you do with them?[1]

This is relevant to figuring out how profitable AI will be in the short-term, how powerful AI systems might be able to come in the near future, how large the compute overhang will be and other strategic questions.

Here’s my attempt to understand this topic as a non-specialist. I’ve had it checked over by some technical advisors, but I don’t claim any special expertise. I wrote it because I haven’t been able to find an accessible explainer elsewhere. I appreciate corrections.

The most obvious approach – the one I often see people in the community taking – is to look up how many FLOP per second your GPU can process, then how many FLOP it takes to run a forward pass, and then divide the two.

For example, Nvidia’s A100 GPU is listed at 312 teraflop per second (3e14) on its spec sheet (FP16 tensor), a forward pass of GPT-4 requires 5.6e11 FLOP per forward pass.[2] So that would imply a single GPU can do about 560 forward passes per second.

But this turns out to be much too high.

Even if it were possible to achieve spec sheet FLOP in a real life application (it’s not), this wouldn’t be the relevant figure because, in practice, inference is limited more by memory than by FLOP:

Each forward pass requires all the parameters to also pass through the GPU’s memory. If 280 billion parameters are activated, and each parameter requires 16-bits = 2 bytes to encode it, then 560 gigabytes must pass through memory.[3]

But the A100’s memory bandwidth is 2000 gigabytes per second – only enough for 4 forward passes.

However, 4 forward passes per second is also not right.

In practice, GPUs are parallelised, so multiple forward passes are processed in batches and many other optimisations are applied, allowing real world efficiency to be much higher than the memory bandwidth of an individual GPU would suggest.

So, the first FLOP-based method is an upper bound, the second memory-based method a lower bound, and the real world lies somewhere in between.

Figuring out where real world efficiency lies is tricky. Not only is it an area of active research, but it also depends on many factors, such as acceptable latency, context length, batch size, size of the model, etc.

The best estimate I’ve seen so far is from Semianalysis. In

their article on GPT-4 architecture

, they estimate that a cluster of 128 ...
How to do conceptual research: Case study interview with Caspar Oesterheld
by Chi Nguyen — 2024-05-14T15:09:30.390Z

Caspar Oesterheld

came up with two of the most important concepts in my field of work:

Evidential Cooperation in Large Worlds

and

Safe Pareto Improvements

. He also came up with a potential implementation of evidential decision theory in boundedly rational agents called

decision auctions

, wrote a comprehensive

review of anthropics and how it interacts with decision theory

which most of my anthropics discussions built on, and independently decided to work on AI some time late 2009 or early 2010.

Needless to say, I have a lot of respect for Caspar’s work. I’ve often felt very confused about what to do in my attempts at conceptual research, so I decided to ask Caspar how he did his research. Below is my writeup from the resulting conversation.

How Caspar came up with surrogate goals

The process

Caspar had spent six months FTE thinking about a specific bargaining problem between two factions with access to powerful AI, spread over two years.
A lot of the time was spent on specific somewhat narrow research projects, e.g. modelling the impact of moral advocacy in China on which bargaining problems we’ll realistically encounter in the future. At the time, he thought those particular projects were important although he maybe already had a hunch that he wouldn’t think so anymore ten years down the line.
At the same time, he also spent some time on most days thinking about bargaining problems on a relatively high level, either in discussions or on walks. This made up some double digit percentage of his time spent researching bargaining problems.
Caspar came up with the idea of surrogate goals during a conversation with Tobias Baumann. Caspar describes the conversation leading up to the surrogate goal idea as “going down the usual loops of reasoning about bargaining” where you consider just building values into your AI that have properties that are strategically advantaged in bargaining but then worrying that this is just another form of aggressive bargaining. The key insight was to go “Wait, maybe there’s a way to make it not so bad for the other side.” Hence, counterpart-friendly utility function modifications were born which later on turned into surrogate goals.
Once he had the core idea of surrogate goals, he spent some time trying to figure out what the general principle behind “this one weird trick” he found was. Thus, with Vincent Conitzer as hi

...
In the context of AI interp. What is a feature exactly?
by f3mi — 2024-05-14T13:46:21.645Z

As I read more about previous interpretability work, I've noticed this trend that implicitly defines a feature in this weird human centric way. It's this weird prior that expects networks to automatically generate features that correspond with how we process images/text because... why exactly?

Chris Olah's team at Anthropic thinks about features as "Something a large enough neural network would dedicate a neuron to". Which doesn't have the human-centric bias, but just begs the question of what is a thing a large enough network will dedicate an neuron to? They admit that this is flawed, but say it's their best current definition. This never felt like a good enough answer, even to go off of.

I don't really see the alternative engaged with. What if these features aren't robust? What if these features don't make sense from a human point of view? It feels like everyone is engaging with an alien brain and expecting it to process things in the same way we do.

Also, I'm confused about the Linear Representation Hypothesis. It makes sense when thinking about categorical features like gender or occupation, but what about quantitative features? Is there a length direction? Multiple?

I hope there's a paper or papers I'm missing, or maybe I'm blowing this out of proportion.
Announcing the AI Safety Summit Talks with Yoshua Bengio
by otto.barten — 2024-05-14T12:52:41.553Z

At the Existential Risk Observatory, we focus on reducing AI existential risk by informing the public debate. One way we do this is by organizing events, such as before the AI Safety Summit. We did this last time in Bletchley Park, and we're now doing it again, fully remote, between the two days (21-22 May) of the Seoul AI Safety Summit.
Our speakers will be:
Opening:
The Rt. Hon. Michelle Donelan MP, Secretary of State for Science, Innovation and Technology of the United Kingdom will open our event.
Keynote:
Yoshua Bengio is professor at the University of Montreal (MILA institute). He is recipient of the Turing Award and generally considered one of the fathers of AI. He is also globally the second-most cited AI scientist.
Panel:
Max Tegmark is physics professor at MIT. His current research focuses on the intersection of physics and AI. He is also president and cofounder of the Future of Life Institute (FLI).
Jaan Tallinn is cofounder of Skype, CSER, and FLI, investor in DeepMind and Anthropic, and a leading voice in AI Safety.
Holly Elmore is AI activist and Executive Director of PauseAI US. She holds a PhD in Organismic & Evolutionary Biology from Harvard University.
Stijn Bronzwaer is AI and technology journalist at leading Dutch newspaper NRC Handelsblad. He co-authored a best-selling book on Booking and is recipient of investigative journalism award De Loep.
Will Henshall is editorial fellow at TIME Magazine. He covers tech, with a focus on AI. One recent piece he wrote details big tech lobbying on AI in Washington DC.
Arjun Ramani writes for The Economist about economics and technology. His writings on AI include a piece on what humans might do in a world of superintelligence.
David Wood, chair of the London Futurists, will be our moderator.
Date/time: 21 May 20:00-21:30 Seoul time (13:00 CET, 12:00 UK time, 7:00 ET)
Register here (free): https://lu.ma/1ex04fuw
Since the event is fully remote and free, anyone interested in AI Safety and AI Governance is can attend. Spreading this announcement is much appreciated!
Can AI partners make human ones pale in comparison?
by false — 2024-05-14T10:57:19.664Z
Link: https://arminbagrat.com/ailuv/

We normally view AI as cold, pragmatic, and, maybe, even socially awkward.
The risks we identify reflect this stereotype.
Just consider the paperclip maximizer;
a superintelligent AI is asked to manage a paperclip factory.
As it is trying to maximize the paperclip production, it quickly realizes that humans waste a whole lot of resources that would be better allocated to paperclip production.
Therefore, it wipes out all humans and relocates the resources they would've consumed to paperclip production.

Although paperclip AI is superintelligent in managing factory operations, it has the social intelligence of a 3 year old.
Historically, this view of AI was technically justified.
Deep learning models are built to optimize a single, well-define measure (e.g. paperclips produced per dollar).
As a result, they tend to be exceptional at tasks where performance can be clearly measured (e.g. image classification, traffic prediction, speech transcription), but unable to perform tasks where performance is not as easily quantifiable.
It was natural to imagine superintelligent versions of such models to inherit this flaw.

However, something surprising has happened.
We have been able to train models on narrow tasks in a way that they end up learning an impressive amount of commonsense knowledge and social intuition.
Of course, I'm talking about language models here.
Even though language models are trained to generate the next word in a sequence of words, they end up also learning a fair bit of knowledge captured in that text.
When training data includes human conversations, models learn the social intuition from them.

Consequently, I'm more concerned with socially superintelligent AI than the stereotypical emotionless cerebrally superintelligent AI.
In this essay, I will explore the implications of such AI on human romantic relationships.

I believe AI partners will become better than their human counterparts in romantic conversations.
As a result, interhuman intimate conversations will pale in comparison to the ones they can have with their artificial partners.

I know it's quite a bold statement to make.
I'll spend the rest of this essay unpacking it.
In particular, I'll focus on three questions:

What makes a good romantic conversation?

Why are AI partners well equipped to excel at romantic conversations?

What are the implications for our society?

What makes a good romantic

...
Why do we enjoy music?
by metachirality — 2024-05-14T08:29:16.803Z

Enjoying music doesn't seem to have any obvious purpose. Sure you can argue it strengthens social bonds, but why specifically sounds arranged in patterns through time over anything else?

At least with humor you can say it's about identifying the generating function of some observation which is sort of like reducing prediction error in predictive coding (and I suspect something like this is the basis for aesthetics) but I can't fit music into being anything like this.
Emergence Is a Universal Non-Zero-Sum Phenomenon.
by James Stephen Brown — 2024-05-14T08:06:51.503Z
Link: https://nonzerosum.games/conwaysgame.html

Against a torrent of entropy, eddies of complexity arise. We have the non-zero-sum phenomenon of emergence to thank for our very existence. While I concede the problematic nature of the term "Emergence" as detailed by Eliezer Yudkowsky in The Futility of Emergence, this series makes a case for emergence not as an explanation, but as a description of a particular type of phenomenon, one that produces qualitatively distinct and more complex entities than those from which they emerged. This six part series explores:

Conway's Game of Life—explore the concept of emergence with a simple simulation.
Entropy vs Emergence—understanding the seemingly paradoxical nature of the second law of thermodynamics.
Emergent Gravity—how can a universe arise from pure chaos?
On the Shoulders of Substrates—how one phenomenon lays the foundation for the next.
Replicators—the dangerous children of emergence; genes, memes, temes, snenes, grenes and of course quenes.
Emergence is Beautiful—beauty and meaning in an entropic universe.

I'm link-posting because these posts in particular involve interactive simulations to help illustrate the principle of emergence—which won't function on LW.

Please discuss, I'm particularly interested in criticisms by physicists, as I am not a trained physicist and I would like to become less wrong.