trentmkelly/LessWrong-43k huggingface dataset:
What I've been reading, September 2023
by jasoncrawford — 2023-09-06T09:32:56.224Z
Link: https://rootsofprogress.org/reading-2023-09

A quasi-monthly feature. Recent blog posts and news stories are generally omitted; you can find them in my links digests. I’ve been busy helping to choose the first cohort of our blogging fellowship, so my reading has been relatively light. All emphasis in bold in the quotes below was added by me.

Books

Joel Mokyr, The Lever of Riches: Technological Creativity and Economic Progress (1990). I’ve been a big fan of Mokyr ever since the start of this project; his book A Culture of Growth was part of my initial motivation. I’m only a few chapters in to Lever of Riches, but it’s excellent so far. Most intriguing so far is his comment that classical civilization was “not particularly technologically creative” even though it was “relatively literate and mobile, and ideas of all kinds disseminated through the movement of people and books.” In contrast:

Early medieval Europe, sometimes still referred to as a “dark” age, managed to break through a number of technological barriers that had held the Romans back. The achievements of early medieval Europe are all the more amazing because many of the ingredients that are usually thought of as essential to technological progress were absent. Particularly between 500 and 800 A.D., the economic and cultural environment in Europe was primitive compared to the classical period. Literacy had become rare, and the upper classes devoted themselves to the subtle art of hacking each other to pieces with even greater dedication than the Romans had. Commerce and communications, both short- and long-distance, declined to almost nothing. The roads, bridges, aqueducts, ports, villas, and cities of the Roman Empire fell into disrepair. Law enforcement and the security of life and property became precarious, as predators from near and afar descended upon Europe with a level of violence and frequency that Roman citizens had not known. And yet toward the end of the Dark Ages, in the eighth and ninth centuries, European society began to show the first signs of what eventually became a torrent of technological creativity. Not the amusing toys of Alexandria’s engineers or the war engines of Archimedes, but useful tools and ideas that reduced daily toil and increased the material comfort of the masses, even when population began to expand after 900 A.D., began to emerge. When we compare the technological progress achieved in the seven centuries betwe

...
Decision Theory: A (Normative) Introduction
by Pareto Optimal — 2023-09-06T08:22:57.877Z
Link: https://paretooptimal.substack.com/p/decision-theory-a-normative-introduction

I. Introduction

Let's define

decision theory

as the study of decisions, specifically their effects on outcomes.

There are three main branches of decision theory: descriptive decision theory (how real agents make decisions), prescriptive decision theory (how real agents should make decisions), and normative decision theory (how ideal agents should make outcomes).

Since decision theory as a field is too broad to be summarized in one post, I'll primarily focus on normative decision theory and only two-thirds of it.

Decisions under ignorance,

rational choice theory

,

bounded rationality

,

prospect theory

,

heuristics

, and the

VNM axioms of rational choice

all deserve separate posts.

II. Terms and Definitions

Before we begin with specific procedures in decision theory, let's start with defining some important terms.

We can define a decision as an act or choice an agent has made and an outcome as a result of such decisions.

Utility should represent an agent's preference over said outcomes and while may be assigned a cardinal value (such as when the agent is VNM-rational), is still a representation of ordinal preferences.

Decisions can be made under certainty, risk, or ignorance. The latter two represent when an agent is uncertain of the outcome corresponding to a decision, however, the former in contrast with the latter allows one to assign subjective probabilities to the outcomes.

In this post, only decisions under certainty and risk will be analyzed.

III. CDT, EDT, and FDT

Finally, we can now discuss three types of decision theory algorithms:

Causal Decision theory (CDT)

,

Evidential Decision Theory (EDT)

, and

Functional Decision Theory (FDT)

.

To briefly define each procedure, we can say that CDT recommends choosing decisions that cause the best-expected outcome, EDT recommends choosing which decision "one would prefer to know one would have chosen", and FDT recommends treating a decision as the output of a ﬁxed mathematical function that answers the question, “Which output of this very function would yield the best outcome?”.

Let's give out some counterexamples to the decision algorithms. First is

Newcomb’s Problem

:

Omega, a being with near-perfect prediction accuracy, shows you two boxes designated A and B. You are given a choice between taking only box B or taking both boxes A and B. You know the following:

Box A is transparent and always contains a visible $1,000.

Box B is opaqu...
What's the easiest way to make a luminator?
by  — 2023-09-06T00:07:08.397Z

My room is really dark. I just realized this after being outside in a car for something and then coming back to my room. (I usually don't go outside.)

I wouldn't have motivation to string a whole bunch of bulbs together, I just want something I can order and plug in. How can I do this while still getting a lot of light?

Thanks in advance
Ordinary claims require ordinary evidence
by blake8086 — 2023-09-05T22:09:16.044Z

Ordinary claims require ordinary evidence

(this is a post version of a YouTube video I made)

One common argument is that making extraordinary claims, such as AI being harmful, requires extraordinary evidence. However, I believe that asserting AI's potential to be harmful is not an extraordinary claim at all. Rather, it's grounded in several key axioms that, when examined, are hard to refute.

Why It's Not an Extraordinary Claim

I think the AI Optimist imagines a particular scenario or set of scenarios (perhaps "Terminator" or [insert fictional franchise here]) and says "that seems improbable". Perhaps Eliezer comes along and posits one additional scenario, and the Optimist says "all of those combined are improbable". "Do you have any proof that this [particular tiny set of scenarios] will happen!?" But the space of AI ruin is vast, any failure scenario would ruin everything.

To me, AI ruin seems to a natural consequence of five simple processes and conditions:

The Five Core Axioms

1. AI gets better, never worse: AI's intelligence, however you define it, is increasing. As new research emerges, the knowledge becomes a permanent part of the record. Like other technological advances, we build on it rather than regress. People constantly throw more resources at AI, training up bigger and bigger models without any regard to safety.

2. Intelligence always helps: Being more intelligent always aids success in the real world. A slight edge in intelligence has allowed humans to dominate the Earth. There is no reason to expect a different outcome with an entity more intelligent than humans.

3. No one knows how to align AI: No one can precisely instruct AI to align with complex human values or happiness. We can optimize for likely prediction of a data point, but no one has written a Python function to rank outcomes by how positive they are for humanity.

4. Resources are finite: Any AI acting in the real world will inevitably compete with humans for resources. These resources, once consumed by AI, won’t be available for human use, leading to potential conflicts.

5. AI cannot be stopped: Once an AI becomes more intelligent and possibly harmful, it's impossible to halt it. Stopping an unaligned AI requires human decision-making to defeat more-intelligent-than-human decision making, which isn't possible.

Combined, these axioms point towards AI: becoming smarter, outcompeting humans, bein...
Conversation about paradigms, intellectual progress, social consensus, and AI
by Ruby — 2023-09-05T21:30:17.498Z

I have a number of thoughts here that I haven't gotten around to writing up, and sharing via a public conversation seems like a decent way to try.
A few opening thoughts:

People talk about the field of AI Alignment being "pre-paradigmatic". I don't have a sense that people have shared precise models behind this, or at least I don't know what other people mean when they say it. Most people don't seem to have read Kuhn for example..
I used to think the goal was to "become paradigmatic". I now think a better goal is to become the kind of field that can go through cycles of paradigm creation, paradigm crisis, and then recreation of a new paradigm.
I think there are ways to increase our ability to do that, and I'd like to have people thinking more about them.

Happy to launch off in any direction that seems interesting or productive to you.
What I would do if I wasn’t at ARC Evals
by LawrenceC — 2023-09-05T19:19:36.830Z

In which: I list 9 projects that I would work on if I wasn’t busy working on safety standards at ARC Evals, and explain why they might be good to work on.

Epistemic status: I’m prioritizing getting this out fast as opposed to writing it carefully. I’ve thought for at least a few hours and talked to a few people I trust about each of the following projects, but I haven’t done that much digging into each of these, and it’s likely that I’m wrong about many material facts. I also make little claim to the novelty of the projects. I’d recommend looking into these yourself before committing to doing them. (Total time spent writing or editing this post: ~8 hours.)

Standard disclaimer: I’m writing this in my own capacity. The views expressed are my own, and should not be taken to represent the views of ARC/FAR/LTFF/Lightspeed or any other org or program I’m involved with.

Thanks to Ajeya Cotra, Caleb Parikh, Chris Painter, Daniel Filan, Rachel Freedman, Rohin Shah, Thomas Kwa, and others for comments and feedback.

Introduction

I’m currently working as a researcher on the Alignment Research Center Evaluations Team (ARC Evals), where I’m working on lab safety standards. I’m reasonably sure that this is one of the most useful things I could be doing with my life.

Unfortunately, there’s a lot of problems to solve in the world, and lots of balls that are being dropped, that I don’t have time to get to thanks to my day job. Here’s an unsorted and incomplete list of projects that I would consider doing if I wasn’t at ARC Evals:

Ambitious mechanistic interpretability.
Getting people to write papers/writing papers myself.
Creating concrete projects and research agendas.
Working on OP’s funding bottleneck.
Working on everyone else’s funding bottleneck.
Running the Long-Term Future Fund.
Onboarding senior(-ish) academics and research engineers.
Extending the young-EA mentorship pipeline.
Writing blog posts/giving takes.

I’ve categorized these projects into three broad categories and will discuss each in turn below. For each project, I’ll also list who I think should work on them, as well as some of my key uncertainties. Note that this document isn’t really written for myself to decide between projects, but instead as a list of some promising projects for someone with a similar skillset to me. As such, there’s not ...
The Illusion of Universal Morality: A Dynamic Perspective on Genetic Fitness and Ethical Complexity
by  — 2023-09-05T17:47:01.454Z

The belief in a universal, independent standard for altruism, morality, and right and wrong is deeply ingrained in societal norms. However, when scrutinized through the lens of dynamic genetic fitness, these concepts reveal their inherent complexities. The idea of group selection suggests that a group rich in altruists may have a survival advantage over a group mainly composed of selfish organisms. Yet, even within such groups, altruists find themselves at a disadvantage compared to their selfish counterparts. This paradox challenges the straightforward interpretation of altruism and suggests that the concept is not as simple as it seems.

Crucially, the application of genetic fitness extends to all levels of gene pools, from our own offsprings to species-level offspring. This perspective reframes altruism and the notions of right and wrong as not merely moral or ethical choices but as strategies for optimizing genetic fitness across varying contexts. In this light, what is often considered 'altruistic' or 'right' may be better understood as actions that contribute to the genetic fitness of a group or species, rather than individual moral virtues.

Environmental factors also significantly shape our understanding of genetic fitness. In certain bird species, for example, some individuals stay near nesting sites to assist in rearing related offspring. This behavior is influenced by various factors such as food availability, mate attraction, and predation. Similarly, our moral and ethical choices are not made in isolation; they are molded by our immediate environment, societal norms, and the gene pool under consideration.

As we venture into new frontiers like deep space exploration and potential coexistence with other intelligent species, our understanding of genetic fitness will need to evolve. The ethical and moral frameworks we have built may not be applicable in these new contexts, especially when the gene pool extends beyond our own species. This raises questions about how we will navigate ethical dilemmas in the future.

The choices we make are deeply rooted in our genetic predispositions and environmental influences. For example, the choice to save one's own child over a stranger's, even when the latter is healthier, is not merely a moral dilemma but a complex interplay of genetic fitness, emotional bonds, and societal expectations.

As technology advances, partic...
Benchmarks for Detecting Measurement Tampering [Redwood Research]
by ryan_greenblatt — 2023-09-05T16:44:48.032Z
Link: https://arxiv.org/abs/2308.15605

TL;DR: This post discusses our recent empirical work on detecting measurement tampering and explains how we see this work fitting into the overall space of alignment research.

When training powerful AI systems to perform complex tasks, it may be challenging to provide training signals that are robust under optimization. One concern is measurement tampering, which is where the AI system manipulates multiple measurements to create the illusion of good results instead of achieving the desired outcome. (This is a type of reward hacking.)

Over the past few months, we’ve worked on detecting measurement tampering by building analogous datasets and evaluating simple techniques. We detail our datasets and experimental results in

this paper

.

Detecting measurement tampering can be thought of as a specific case of

Eliciting Latent Knowledge (ELK)

: When AIs successfully tamper with measurements that are used for computing rewards, they possess important information that the overseer doesn't have (namely, that the measurements have been tampered with). Conversely, if we can robustly elicit an AI's knowledge of whether the measurements have been tampered with, then we could train the AI to avoid measurement tampering. In fact, our best guess is that this is the most important and tractable class of ELK problems.

Measurement tampering as discussed in the ELK report: the robber has taken the diamond, but measurements have been tampered with using a picture of a diamond (image by María Gutiérrez-Rojas)

We also think that measurement tampering detection is a natural application for alignment work such as creating better inductive biases, studying high-level model internals, or studying generalization. We’ll discuss what these applications might look like in the Future work section.

In this post:

We explain what measurement tampering detection is;
We summarize the results of our paper;
We argue that there are structural properties of measurement tampering that might make it considerably easier to detect than arbitrary cases of eliciting knowledge from models;
We argue that detecting measurement tampering is very useful for avoiding catastrophic outcomes from powerful and misaligned AI systems;
We explain why we think the dataset structure we chose reflects real-world difficulties of measurement tampering, and how techniques developed for our datasets could be used

...
Strongest real-world examples supporting AI risk claims?
by rosehadshar — 2023-09-05T15:12:11.307Z

[Manually cross-posted to EAForum here]

There are some great collections of examples of things like specification gaming, goal misgeneralization, and AI improving AI. But almost all of the examples are from demos/toy environments, rather than systems which were actually deployed in the world.

There are also some databases of AI incidents which include lots of real-world examples, but the examples aren't related to failures in a way that makes it easy to map them onto AI risk claims. (Probably most of them don't in any case, but I'd guess some do.)

I think collecting real-world examples (particularly in a nuanced way without claiming too much of the examples) could be pretty valuable:

I think it's good practice to have a transparent overview of the current state of evidence
For many people I think real-world examples will be most convincing
I expect there to be more and more real-world examples, so starting to collect them now seems good

What are the strongest real-world examples of AI systems doing things which might scale to AI risk claims?

I'm particularly interested in whether there are any good real-world examples of:

Goal misgeneralization
Deceptive alignment (answer: no, but yes to simple deception?)
Specification gaming
Power-seeking
Self-preservation
Self-improvement

This feeds into a project I'm working on with AI Impacts, collecting empirical evidence on various AI risk claims. There's a work-in-progress table here with the main things I'm tracking so far - additions and comments very welcome.
AISN #21:
Google DeepMind’s GPT-4 Competitor, Military Investments in Autonomous Drones, The UK AI Safety Summit, and Case Studies in AI Policy
by Dan H — 2023-09-05T15:03:00.177Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-21

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Google DeepMind’s GPT-4 Competitor

Computational power is a key driver of AI progress, and a new report suggests that Google’s upcoming GPT-4 competitor will be trained on unprecedented amounts of compute.

The model, currently named Gemini, may be trained by the end of this year with 5x more computational power than GPT-4. By the end of next year, the report projects that Google will have the ability to train a model with 20x more compute than GPT-4.

For reference, the compute difference between GPT-3 and GPT-4 was 100x. If these projections are true, Google’s new models could create a meaningful spike relative to current AI capabilities.

Google’s position as an AI leader. The recent boom in large language models has been driven by several innovations pioneered at Google. For example, the Transformer architecture used by most advanced language models was first described in a 2017 paper from Google.

OpenAI has led Google in language modeling for several years now. But after the release of ChatGPT, Google significantly increased their AI investments. They merged Google Brain and DeepMind into a single research lab with increased resources, and invested in Anthropic.

Google has tremendous financial resources, with $118 billion cash on hand. In contrast, OpenAI’s last investment round in January raised only $10 billion. Perhaps it’s no surprise that Google can quickly ramp up spending to compete with other leading AI labs.

Yet it seems that Gemini will be only one member of the next generation of frontier models, as Inflection AI CEO Mustafa Suleyman says his company will also soon surpass the compute used to train GPT-4.

Inflection AI CEO on compute growth. Mustafa Suleyman, CEO of the rapidly growing Inflection AI, estimated that his company will be training models “100x larger than the current frontier models in the next 18 months.” Notably, this is not based on predictions that they’ll gain access to new compute, but rather estimated using the compute that Inflection already owns.

Three years from now, he predicts the industry will be “training models that are 1,000x larger than they currently are.” This positions Suleyman as one of the m...
Who Has the Best Food?
by Zvi — 2023-09-05T13:40:07.593Z

It is a fun question going around the internet this past week, so here we go.

In particular, people focused on the question of France vs. America. As one would expect, those on the French side think those on the American side are crazy, it is insulting to even consider this a question. Those on the American side like food.

All of this is always just, like, your opinion, man, or at least that’s the story.

Checking the Survey Data

YouGov asked back in 2019, got the following answers across nations, which we were reminded of during current debate on Twitter of American versus French food.

I will quibble, but I was impressed how good this list was for nationally identified cuisine, as opposed to in-country experience.

Where do I see obvious mistakes, ignoring the unfamiliar ones?

Everyone is underrating Brazilian because meat on swords is awesome, Pão de queijo is awesome, and the accompaniments are optional, if not diverse enough for the true top tier.

Mongolian is not even listed, and I am aware that what I call Mongolian is not in any way an authentic experience, where you fill a bowl with meats, noodles and sauces (and for some, vegetables) that they then grill for you, but that format is indeed excellent, and I will take advantage of it every chance I get. Which is not often. Somehow you cannot get Mongolian BBQ anywhere in New York City, although a comment did point me to one in the East Bay which helps.

I have two very different places I like to go for Lebanese, one low end and one high end. I’m not sure if this is a coincidence or not and what distinguishes them from other Middle Eastern cuisines, except perhaps their rice style. Either way, underrated.

America’s biggest mistake is underrating Indian quite a lot. Indian provides a unique set of flavors and experiences, done mediocre it is fine and done well it is very good. Only China and Thailand have it lower than America, and I am guessing that opinion is mostly not about the food.

Spanish and British seem clearly overrated here, although perhaps Spanish gets a lot better when Italian isn’t available locally. I have never not felt Spanish cuisine was an inferior good.

Thai food is very good when they don’t overdo the chili oil as a cheat code, but is likely higher than it should be due to its absurdly excellent marketing.

Korean is alien, they serve you all these things my brain refuses to agre...