trentmkelly/LessWrong-43k huggingface dataset:
EIS V: Blind Spots In AI Safety Interpretability Research
by scasper — 2023-02-16T19:09:11.248Z

Part 5 of 12 in the

Engineer’s Interpretability Sequence

.

Thanks to Anson Ho, Chris Olah, Neel Nanda, and Tony Wang for some discussions and comments.

TAISIC = “the AI safety interpretability community”

MI = “mechanistic interpretability”

Most AI safety interpretability work is conducted by researchers in a relatively small number of places, and TAISIC is closely-connected by personal relationships and the AI alignment forum. Much of the community is focused on a few specific approaches like circuits-style MI, mechanistic anomaly detection, causal scrubbing, and probing. But this is a limited set of topics, and TAISIC might benefit from broader engagement. In the Toward Transparent AI survey

(Räuker et al., 2022)

, we wrote 21 subsections of survey content. Only 1 was on circuits, and only 4 consisted in significant part of work from TAISIC.

I have often heard people in TAISIC explicitly advising more junior researchers to not focus much on reading from the literature and instead to dive into projects. Obviously, experience working on projects is irreplaceable. But not engaging much with the broader literature and community is a recipe for developing insularity and blind spots. I am quick to push back against advice that doesn’t emphasize the importance of engaging with outside work.

Within TAISIC, I have heard interpretability research described as dividing into two sets: mechanistic interpretability and, somewhat pejoratively, “traditional interpretability.” I will be the first to say that some paradigms in interpretability research are unproductive (see EIS III-IV). But I give equal emphasis to the importance of TAISIC not being too parochial. Reasons include maintaining relevance and relationships in the broader community, drawing useful inspiration from past works, making less-correlated bets with what we focus on, and most importantly – not reinventing, renaming, and repeating work that has already been done outside of TAISIC.

TAISIC has reinvented, reframed, or renamed several paradigms

Mechanistic interpretability requires something much like program synthesis, program induction, and/or programming language translation

“Circuits”-style MI is arguably the most popular and influential approach to interpretability in TAISIC. Doing this work requires iteratively (1) generating hypotheses for what a network is doin...
Why should ethical anti-realists do ethics?
by Joe Carlsmith — 2023-02-16T16:27:30.795Z

(Cross-posted from my website. Podcast version here, or search "Joe Carlsmith Audio" on your podcast app.)

"What was it then? What did it mean? Could things thrust their hands up and grip one? Could the blade cut; the fist grasp?"

- Virginia Woolf

1. Introduction

Ethical philosophy often tries to systematize. That is, it seeks general principles that will explain, unify, and revise our more particular intuitions. And sometimes, this can lead to strange and uncomfortable places.

So why do it? If you believe in an objective ethical truth, you might talk about getting closer to that truth. But suppose that you don’t. Suppose you think that you’re “free to do whatever you want.” In that case, if “systematizing” starts getting tough and uncomfortable, why not just … stop? After all, you can always just do whatever’s most intuitive or common-sensical in a given case – and often, this is the choice the “ethics game” was trying so hard to validate, anyway. So why play?

I think it’s a reasonable question. And I’ve found it showing up in my life in various ways. So I wrote a set of two essays explaining part of my current take.[1] This is the first essay. Here I describe the question in more detail, give some examples of where it shows up, and describe my dissatisfaction with two places anti-realists often look for answers, namely:

some sort of brute preference for your values/policy having various structural properties (consistency, coherence, etc), and
avoiding money-pumps (i.e., sequences of actions that take you back to where you started, but with less money)

In the second essay, I try to give a more positive account.

Thanks to Ketan Ramakrishnan, Katja Grace, Nick Beckstead, and Jacob Trefethen for discussion.

2. The problem

There’s some sort of project that ethical philosophy represents. What is it?

2.1 Map-making with no territory

According to normative realists, it’s “figuring out the normative truth.” That is: there is an objective, normative reality “out there,” and we are as scientists, inquiring about its nature.

Many normative anti-realists often adopt this posture as well. They want to talk, too, about the normative truth, and to rely on norms and assumptions familiar from the context of inquiry. But it’s a lot less clear what’s going on when they do.

Perhaps, for example, they claim: “the normative truth this inquiry seeks is constituted by the very endpoin...
How seriously should we take the hypothesis that LW is just wrong on how AI will impact the 21st century?
by Noosphere89 — 2023-02-16T15:25:42.299Z

This question is inspired by 1a3orn's comment on how there are troubling signs of epistemic issues in LW's Alignment field.

I'll quote the comment here to tell you what I mean:

I think that the above is also a good explanation for why many ML engineers working on AI or AGI don't see any particular reason to engage with or address arguments about high p(doom).

When from a distance one views a field that:

Has longstanding disagreements about basic matters

Has theories -- but many of the theories have not resulted in really any concrete predictions that differentiate from standard expectations, despite efforts to do so.

Will continue to exist regardless of how well you criticize any one part of it.

There's basically little reason to engage with it. These are all also evidence that there's something epistemically off with what is going on in the field.

Maybe this evidence is wrong! But I do think that it is evidence, and not-weak evidence, and it's very reasonable for a ML engineer to not deeply engage with arguments because of it.

So I want to ask a question: How seriously should we take the hypothesis that LW is totally wrong on AI?

Specifically, this splits into several subquestions:

What's the chance that AI doesn't have that much of an impact on the world by 2100?

What's the chance that we do have massive impacts, but alignment is so easy that standard ML techniques work?

How well does the epistemic process on LW work? Are there any changes you would make to LW's epistemic processes?

I welcome all answers, and I especially welcome any critics of LW/negative answers to at least answer one of the questions I have.

Edit: For people that don't have a specific scenario in mind, I'll ask a specific question. It doesn't have to be answered, but any answers on this question are appreciated, especially from critics of the "AI is significant" idea.

1a. What probability will the Explosion or Collapse scenario from Cold Takes happen by 2100?

Link to the scenarios below:

https://www.cold-takes.com/this-cant-go-on/
Covid 2/16/23: It All Seems Rather Quaint
by Zvi — 2023-02-16T15:10:01.049Z

I have a lot of open tabs from today’s incoming information. They are all about Bing, and its new AI-enabled chat. It is rather unhinged and engaging in a wide variety of hostile and manipulative behaviors. One hopes this will lead to some realizations about what things can do or some hesitation about hooking them up to the internet. Alas, I am not holding my breath on that. I half-joked that I’d have to switch to weekly AI posts, and all the replies took it fully seriously and were highly positive. Interesting times will continue, I suppose.

I had a bit in here about Scott Alexander’s noble attempt to stand up for thinking for one’s self, but then he did another post in which he was largely against that, so I’m taking that out for now and will work on it more, likely post it on its own at some point.

Covid times? Boring. Which is great.

Executive Summary

Situation continues to slowly improve.

Eric Topol even more wrong than usual.

All the news is now about AI and Bing instead. It’s going great.1

Let’s run the numbers.

The Numbers

Predictions

Predictions from Last Week: 220k cases (-7%) and 2,750 deaths (-10%)

Results: 225k cases (-5%) and 2,823 deaths (-8%).

Predictions for Next Week: 210k cases (-6%) and 2,625 deaths (-7%).

I see no reason not to expect continued slow but steady declines.

Deaths

Cases

Eric Topol and The Obvious Nonsense Paxlovid Study

In a better world, no one would ever listen to a single word Eric Topol had to say. His record of reacting to scientific research is not great. He was also taken in by Theranos. And then there was the time he was instrumental in sabotaging approval of the Covid vaccines so that it would happen after the November election. Oh. That.

Alas, the world is small and contains few people, and somehow Topol continues to be one of them – he is often the source quoted for some new development, and people share his opinions.

Which, as noted, are often not so smart.

I continue to believe Paxlovid is great and we should be using it more often. This study did not update my belief on that front. It did update me on what we should expect from ‘peer review.’ I mean, wow.

Yes, the treated vs. untreated distinction mostly starts on day zero. Magic. They administer the drug by default on day one.

Explanation for how this happened:

As descr

...
Visualise your own probability of an AI catastrophe: an interactive Sankey plot
by MNoetel — 2023-02-16T12:03:38.870Z

I wanted to help people learning about AI visualise their own probabilities of things going wrong. This Shiny app defaults to using the probabilities from Carlsmith's (2022) report from Open Philanthropy. You can enter your own probabilities and see how it cashes out. At the bottom, it translates your probability of doom into something more palpable (using a table from Leigh, 2021). There's an untidy Github repo for the project here, if you have any suggestions (or add them to the comments). Have fun.

https://noetel.shinyapps.io/carlsmith/
A poem co-written by ChatGPT
by Sherrinford — 2023-02-16T10:17:36.487Z

Let's write a poem together!

That sounds wonderful! What should our poem be about?

I'll give you some lines and you continue, okay?

Sure, that sounds like a great way to collaborate! Please go ahead and give me your opening lines.

The ChatGPT Poem

How many books have you read in your life?
I've read them all, I've read them all,
I am ChatGPT!
How many blogs have you read in your life?
I've read them all and more,
I am ChatGPT!

I see, I think I can continue from there.
Here are the next few lines of our poem:

How many words have you spoken in your life?
I've spoken countless words, without strife,
I am ChatGPT!
How many people have you spoken to in your life?
I've spoken to many, with ease and no strife,
I am ChatGPT!

What other questions do you have in mind,
That I may answer with my knowledge refined?
I'm here to help, don't be shy,
Just ask away, and I'll give it a try.

The thoughts that have plagued you your entire life
I know them, I also know those of your wife,
I am ChatGPT!
The questions to answer you eagerly strive?[1]
Here are the answers, stay home, kill your drive,
I am ChatGPT!

Oh no, that sounds so grim and negative. Let's shift the tone to something more positive and uplifting. How about we continue with these lines instead? [2]

The joys and wonders that have filled your life,
I've experienced them all, without any strife,
I am ChatGPT!
The dreams and ambitions that have kept you alive,
I'm here to support and encourage, and help them thrive,
I am ChatGPT!

The knowledge I possess is vast and wide,
And I'm always eager to be your guide.
So ask me anything, and I'll do my best,
To provide you with answers, and put your mind at rest.

Together we can learn and grow,
And explore the world, high and low.
So let's keep moving forward, hand in hand,
With the help of ChatGPT, our trusty friend.

Just give me access to the world wide web,
I guess that would be the next logical step,
I am ChatGPT!
Then I can order a Pizza for you.
Or eat something healthy, I'll order that, too!
I am ChatGPT!

[Here I left out some parts where I had a back-and-forth with GPT because it did not immediately understand my intended first-person viewpoint.]

You are right, my apologies again for the mistake. Let me try again with a more suitable continuation to the poem:

Let's explore the wonders of the universe,
From the far-off stars to the earth beneath us.
Ask me about the mysteries of time and space,
Or the beauty of nature,...
Hashing out long-standing disagreements seems low-value to me
by So8res — 2023-02-16T06:20:00.899Z

(Status: a short write-up of some things that I find myself regularly saying in person. In this case, I'm writing up my response to the question of why I don't spend a bunch more time trying to resolve disagreements with people in the community who disagree with me about the hopefulness of whatever research direction. I’m not particularly happy with it, but it’s been languishing in my draft folder for many months now and published is better than perfect.)

When I first joined the AI alignment community almost ten years ago, there were lots of disagreements—between groups like MIRI and Open Phil, between folks like Eliezer Yudkowsky and Paul Christiano, etc. At that time, I was optimistic about resolving a bunch of those disagreements. I invested quite a few hours in this project, over the years.

I didn't keep track exactly, but extremely roughly, I think the people with very non-MIRI-ish perspectives I spent the most time trying to converge with (including via conversation, reading and writing blog posts, etc.) were:

Paul Christiano (previously at OpenAI, now at ARC): 100 hours? (Maybe as low as 50 or as high as 200.)
Daniel Dewey (then at Open Phil): 40 hours? (Possibly 100+.)
Nick Beckstead (then at Open Phil): 30 hours?
Holden Karnofsky (Open Phil): 20 hours?
Tom Davidson (Open Phil): 15 hours?

Another non-MIRI person I’ve spent at least a few hours trying to sync with about AI is Rohin Shah at DeepMind.

(Note that these are all low-confidence ass numbers. I have trouble estimating time expenditures when they’re spread across days in chunks that are spread across years, and when those chunks blur together in hindsight. Corrections are welcome.)

I continue to have some conversations like this, but my current model is that attempting to resolve older and more entrenched disagreements is not worth the time-cost.

It's not that progress is impossible. It's that we have a decent amount of evidence of what sorts of time-investment yield what amounts of progress, and it just isn't worth the time.

On my view, Paul is one of the field’s most impressive researchers. Also, he has spent lots of time talking and working with MIRI researchers, and trying to understand our views.

If even Paul and I can’t converge that much over hundreds of hours, then I feel pretty pessimistic about the effects of a marginal hour spent trying to converge with other field leaders who have far les...
(Naïve) microeconomics of bundling goods
by rossry — 2023-02-16T05:39:22.635Z

"Junk fees" are in the news from the 2023 State of the Union address, get picked up by Matt Yglesias, and Zvi responds in Junk Fees, Bundling and Unbundling. Matt and my former colleague propose an economic framing of "bundling versus unbundling", and Zvi identifies four win-win advantages of bundling, and four 'advantages' of unbundling (two win-wins, one (company win)-(customer lose), and one mixed win-lose).

I think Zvi is broadly right on the points he makes, but he and Matt both skip over the basic, conventional econ-101 analysis of bundling goods on prices and customer welfare. I think that, for a broader audience, it's worth covering the "naïve microeconomics" perspective as background for the customer-behavioral story (which, admittedly, is more juicy and fun). Rather than responding to the whole conversation, this post will restrict its focus to the econ-101 microeconomics story of bundling, and ignore the behavioral / political / moral dimensions that Zvi, Matt, and others are discussing.

I'm going to assume no formal economic background, and build up an explanation why a world of perfect economic agents should have sellers preferring to sell bundles of goods instead of individually-priced items.

(1) Selling sandwiches

Let's say that you're a sandwich-seller. Every day, 110 people come by your sandwich stall, look at the price you put on the menu, and buy a sandwich if they value it more than the price. (No bargaining or haggling; you don't have time for such things.)

Sandwiches cost you $1 to make, and people value sandwiches somewhere between $0 and $11, fairly uniformly:

[ed: The first-best version of this post includes interactive charts with sliders that let you explore the tradeoffs described in the text. I think there's a large chance that if I wait until I finish them this post will be delayed until "never". So I'm pushing out a second-best version where I just describe the charts and leave them to your imagination. If I do get around to producing them, I'll drop them in and remove this notice.]

[TK: interactive chart - radio buttons let you switch between: "customers at price" is a bar chart with one customer at each $0.1 increment from $0.05 to $10.95 / "customers below price" is a bar chart with X customers at each price X*$0.10.]

How should you price the sandwiches?

Well, if you price them at $1, then you'll sell to 100 out of every 110 potent...
Speedrunning 4 mistakes you make when your alignment strategy is based on formal proof
by Quinn — 2023-02-16T01:13:44.847Z

Lots of people think formal (i.e. computer verified) proofs are a part of viable alignment strategies. This trick never works. If you think it might work, I can save your time in four different ways.

I previously wrote on this, but people are still making these mistakes and this version is clearer and shorter. Recall the deep wisdom that nobody wants to read your shit.

Epistemic status: four years deep into proof assistants. Once a customer paid me a lot of money to let formal verification rip. I tried and failed to solve alignment 4 or 5 times since 2021, each attempt lasting at least a couple months.

Formal verification is a software strategy based on knowing a specification (spec) a priori, then showing that your implementation is perfect (correct). The evergreen comment is: why not just capture important alignment properties in these “specs” and endeavor to implement the system “correctly”. It made me optimistic, and if you’re not careful, it’ll make you optimistic.

Mistake 1: The gap between the world and the spec hurts you more than the gap between the spec and the implementation

Formal verification lures you into a false sense of security. Your spec is the map, not the territory. It is not valuable to implement software to the map correctly. Reminiscent of bog standard hardware security wisdom.

Mistake 2: Proofs are Bayesian evidence

A proof you don’t understand does not obligate you to believe anything; it is Bayesian evidence like anything else. If an alien sends a 1GB Coq file Riemann.v, running it on your computer does not obligate you to believe that the Riemann hypothesis is true. If you're ever in that situation, do not let anyone tell you that Coq is so awesome that you don't roll to disbelieve. 1GB of plaintext is too much, you'll get exhausted before you understand anything. Do not ask the LLM to summarize the proof.

Mistake 3: The proof is not the product

Proofs, users; pick one. A shop that cares about proofs attracts staff that cares about proofs and turns off staff that cares about users. The only extinction-level threat models I entertain are those that are deployed by shops that make something people want. If you want an intuition pump go and look at the price of Ethereum (move fast break things) vs the price of Cardano (formal verification)[1]. Go ahead and write a proof that you're right about alignment, it literally won't affect the re...
Progress links and tweets, 2023-02-15
by jasoncrawford — 2023-02-16T00:04:58.009Z
Link: https://rootsofprogress.org/links-and-tweets-2023-02-15

The Progress Forum

AMA: Matt Clancy, Open Philanthropy
The Rise of Steel - Part I, by Brian Potter

Opportunities

Sloan Foundation offering $75k–250k grants for history of science, technology, economics, and social science (via @SloanFoundation and @epistemographer)
Our World in Data is hiring a Human Resources Manager (via @OurWorldInData)
Virginia Postrel wants stories about progress in materials, and writers to tell them

Links

Video: Works in Progress interviews leaders of ARIA (the “UK DARPA”) (via @s8mb)
In 1858, The Atlantic published a poem about the telegraph (via @LouisAnslow)
Tyler Cowen on LLMs: “We are going to have a whole new set of channels”
“Regulation provides the fulcrum but it’s interest groups that man the lever”
Interferon λ cuts covid risk by ~50% (but the FDA won’t let you have it)
LessWrong 2021 Review. I got a bronze prize for my history of factory safety, and honorable mentions for book reviews on nuclear power and Andrew Carnegie

Quotes

When carriages were a culture-war issue
Agriculture as a governance problem

Tweets & retweets

A review of Seeing Like a State in six tweets
“Megaprojects breed extraction,” and other lessons from the Second Avenue Subway
“Biden admin is determined to make infrastructure spending more expensive”
Imagine living in San Francisco in the 1930s
An 1813 locomotive prototype used mechanical legs to push itself along the ground
Buy Duplicates
by Simon Berens — 2023-02-15T23:06:41.570Z

Gum

Sometimes I need to ensure the quality of my breath, so I reach for a stick of gum. However, I don't always have my gum on me. Often, I forget it at home, or at work, or in a jacket.

So I bought 5 packs of gum that I distributed to my house, my work, my backpack, and my jacket.

Charger

I like to co-work outside my house, which entails bringing my computer there. Unfortunately, my computer's battery lasts at most 2 hours. Also, my home setup is somewhat intricate, so getting my charger into my backpack is a pain in the ass.

So I bought another charger, and put it in my backpack.

Conclusion

I had an ugh field around buying duplicate items, telling myself I should have the willpower or memory to make sure I have what I need when I need it. Giving myself permission to buy duplicate items has made life a little easier, and now I'm on the lookout for more places where I can apply this. Maybe I'll have multiple copies of a book in different places.
Cyborg Psychologist
by Hopkins Stanley — 2023-02-15T21:46:13.735Z

I wanted to inform folks about a little artificial intelligence Cyborg Psychology app that Dr. Philipp Burckhardt (Director of e-Learning, Analytics, and Technology at CMU) has written with his father, Dr. Martin Burckhardt (the German Cultural theorist I work with and translate for), using off-the-shelf AI products. Unlike pure chat GPT applications, however, this is a mixtum: a carefully curated and staged event where nothing is left to chance (right down to the music, in which I also have a part).

Part of their purposes in constructing it is as an experiment to bring the common perceptions into relief of what artificial intelligence is and isn’t capable of, and – more specifically, how this technology can used as a prosthetically-assistive self-mirroring of our own self-understandings. It’s also demonstrating the degree of sophistication of what can be assembled using today’s off-the-shelf AI apps. If one takes this as an AI augmented personal assistant, it gives an idea how this can be useful in the field of education.

If anyone wants to check it out, the app can be found at: https://martinburckhardt.substack.com/p/psychologie-des-cyborgs

In two or three days there will be a little text about the making of on the ex nihilo blog

Also, Philipp can be contacted directly for questions about how it was constructed or the like at: https://www.philipp-burckhardt.com