trentmkelly/LessWrong-43k huggingface dataset:
Yudkowsky on AGI risk on the Bankless podcast
by Rob Bensinger — 2023-03-13T00:42:22.694Z

Eliezer gave a very frank overview of his take on AI two weeks ago on the cryptocurrency show Bankless:

I've posted a transcript of the show and a follow-up Q&A below.

Andrea_Miotti, remember, and vonk previously posted transcripts to LessWrong (1,2 — see the comments there for discussion), but they contained many important errors, so I've corrected the transcripts below. (They also weren't available on the EA Forum.)

Intro

Eliezer Yudkowsky: [clip] I think that we are hearing the last winds start to blow, the fabric of reality start to fray. This thing alone cannot end the world, but I think that probably some of the vast quantities of money being blindly and helplessly piled into here are going to end up actually accomplishing something.

Ryan Sean Adams: Welcome to Bankless, where we explore the frontier of internet money and internet finance. This is how to get started, how to get better, how to front run the opportunity. This is Ryan Sean Adams. I'm here with David Hoffman, and we're here to help you become more bankless.

Okay, guys, we wanted to do an episode on AI at Bankless, but I feel like David...

David: Got what we asked for.

Ryan: We accidentally waded into the deep end of the pool here. And I think before we get into this episode, it probably warrants a few comments. I'm going to say a few things I'd like to hear from you too. But one thing I want to tell the listener is, don't listen to this episode if you're not ready for an existential crisis. Okay? I'm kind of serious about this. I'm leaving this episode shaken. And I don't say that lightly. In fact, David, I think you and I will have some things to discuss in the debrief as far as how this impacted you. But this was an impactful one. It sort of hit me during the recording, and I didn't know fully how to react. I honestly am coming out of this episode wanting to refute some of the claims made in this episode by our guest, Eliezer Yudkowsky, who makes the claim that humanity is on the cusp of developing an AI that's going to destroy us, and that there's really not much we can do to stop it.

David: There's no way around it, yeah.

Ryan: I have a lot of respect for this guest. Let me say that. So it's not as if I have some sort of big-brained technical disagreement here. In fact, I don't even know enough to fully disagree with anything he's saying. But the conclusion is so d...
Thoughts on self-inspecting neural networks.
by Deruwyn — 2023-03-12T23:58:38.820Z

While I have been contemplating this subject for quite some time, this is my first attempt at public communication of it. I have been thinking about AI implementation, as well as safety. I've been looking at ideas of various experts in the field and considering how things might be combined and integrated to improve either performance or safety (preferably both). One important aspect of that is legibility of how neural networks work. While I've been thinking about it for some time, a YouTube video I watched last night helped me crystallize the how of implementing it. It's an interview with the authors of this paper on Arxiv. In the paper, they show how they were able to isolate a concept in GPT-3 to a particular set of parameters. Them demonstrating that has made what I've been thinking of appear to be that much more plausible.

So, my concept is for the integration of an ensemble of specialized neural networks that can contribute to a greater whole. Specifically, in this case, I'm considering a network that is trained by watching, and eventually directing, the training of a network such as an LLM. My thought is to export an "image" of a neural network with the pixels representing the parameters in the network. It would be a multidimensional image and I think that standard RGB would probably be insufficiently dimensional to represent everything that I'm considering, or perhaps multiple "views" might be used so that we could maintain compatibility with standard image formats so that we can take advantage of existing tools and also display them for some level of human consumption. It might be best to have a way to transition between multidimensional matrices and images so that each can be looked at when applicable in different contexts. Because of the size of the networks, it will be necessary to not only have a base representation of the network but also be able to represent changes and activations within the network as "diffs" away from the baseline (or at least the previous resultant state of the baselines + previous diffs). I'm thinking something like the way that Git works. We could also use various lossless and lossy image compression techniques to make them more tractable to work with.

So, within these artificial neural images would be contained the various weights and biases of the MLPs and Attention Heads. That would form the base layer that we're working...
An AI risk argument that resonates with NYTimes readers
by Julian Bradshaw — 2023-03-12T23:09:20.458Z

Ezra Klein of NYT put out a surprisingly sympathetic post on AI risk in the Sunday edition. It even quotes Paul Christiano and links back to LessWrong!
But what I'm actually here to talk about is the top reader-recommended comment on the article as of Sunday 11pm UTC:

Dwarf Planet

I wonder how many of these AI researchers have children. What Ezra describes here is what I see every day with my teenager. Of course, no one understands teenagers, but that's not what I mean. I taught my daughter to play chess when she was very young. I consider myself a reasonably good player, and for many years (as I was teaching her), I had to hold myself back to let her win enough to gain confidence. But now that she is thirteen, I suddenly discovered that within a span of weeks, I no longer needed to handicap myself. The playing field was level. And then, gradually and then very suddenly, she leapt past my abilities. As with AI, could understand the broad outlines of what she was doing--moving this knight or that rook to gain an advantage--but I had no clue how to defend against these attacks. And worse (for my game, at least), I would fall into traps where I thought I was pursuing a winning hand but was lead into ambush after ambush. It was very humbling: I had had the upper hand for so long that it became second nature, and then suddenly, I went to losing every game.
As parents, we all want our children to surpass us. But with AI, these "summoners" are creating entities whose motives are not human. We seem to be at the cusp of where I was before my daughter overtook me: confident and complacent, not knowing what lay ahead. But, what we don't realize is that very soon we'll begin to lose every game against these AIs. Then, our turn in the sun will be over.

Generally NYT comments on AI risk are either dismissive, or just laden with general anxiety about tech. (Indeed, the second-most recommended comment is deeply dismissive, and the third is generic anxiety/frustration.) There's hopefully something to learn from commentor "Dwarf Planet" in terms of messaging.
Musicians and Mouths
by jefftk — 2023-03-12T22:50:02.006Z

Most musicians massively underuse their mouths, and the ones
that use their mouths underuse their fingers. This is very
weird and I don't understand it.

The most common instruments I see are probably piano, guitar, drums,
and violin. Add in mandolin, bass, accordion, cello, viola, and
banjo, and they all have one thing in common: your mouth is completely
idle.

Then there are instruments that do use the mouth, but these don't use
anywhere near the full capacity of your fingers. A trumpet, baritone,
or tuba has just three buttons and a mini slide, and the other hand
just sits there. The trombone and harmonica just need to be moved
back and forth. The flute, clarinet, sax, and oboe do at least use
both your hands, but using all ten fingers to select a single note is
terrible. Recorder, tin whistle, and bagpipes are even worse: not
only do they require all your fingers, but your mouth isn't doing very
much.

This does make sense historically: while your mouth is incredibly
versatile, there aren't obvious ways to combine it with a guitar,
drum, or violin. And the easiest ways to build a wind instrument all
require a lot of fingers so you can spread them out along a tube. But
we've had serious electric and electronic instruments for decades: why
is it still uncommon to see people using their mouth's incredible
ability to modulate sound? Some people sing and play at the same
time, which is great, but probably less than half of musicians?

I wrote about this a bit in 2019 in
Where are the new Instruments?:

Breath controllers, which sense breath pressure and make it available
as one input to a synthesizer, have been around for decades, and are
old enough to have been assigned CC-2 in MIDI. Lots of people have
tried them, with some impressive results, and there's no risk of being
tied to a vendor since they're so simple. But even though they would
allow anyone playing keyboard to add a new dimension to their playing,
musicians are sufficiently uninterested that Yamaha discontinued their
BC line in 2011. The electronic instruments that have taken off live
are mostly just keyboard and maybe drums. There are so many
fantastically creative people in the music industry, but in the 80s
innovation here seems to have stopped. Why aren't people coming up
with creative new ways of modulating the sound as you make it?

Since then I've been usin...
Are there cognitive realms?
by TsviBT — 2023-03-12T19:28:52.935Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2022/11/are-there-cognitive-realms.html. First completed November 16, 2022. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text may be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]

Are there unbounded modes of thinking that are systemically, radically distinct from each other in relevant ways?

Note: since I don't know whether "cognitive realms" exist, this essay isn't based on clear examples and is especially speculative.

Realms

Systemically, radically distinct unbounded modes of thinking

The question is, are there different kinds--writ large--of thinking?

To the extent that there are, interpreting the mental content of another mind, especially one with different origins than one's own, may be more fraught than one would assume based on experience with minds that have similar origins to one's own mind.

Are there unbounded modes of thinking that are systemically, radically distinct from each other?

"Unbounded" means that there aren't bounds on how far the thinking can go, how much it can understand, what domains it can become effective in, what goals it can achieve if they are possible.

"Systemically" ("system" = "together-standing-things") means that the question is about all the elements that participate in the thinking, as they covary / coadapt / combine / interoperate / provide context for each other.

"Radical" (Wiktionary) does not mean "extreme". It comes from the same etymon as "radish" and "radix" and means "of the root" or "to the root"; compare "eradicate" = "out-root" = "pull out all the way to the root", and more distantly through PIE *wréh₂ds the Germanic "wort" and "root". Here it means that the question isn't about some mental content in the foreground against a fixed background; the question asks about the background too, the whole system of thinking to its root, to its ongoing source and to what will shape it as it expands into new domains.

Terms

Such a mode of thinking could be called a "realm". A cognitive realm is an overarching, underlying, systemic, total, architectural thoughtform that's worth discussing separately from other thoughtforms. A realm is supposed to be objective, a single metaphorical place where multiple different minds or agents could find themselves....
Creating a Discord server for Mechanistic Interpretability Projects
by Victor Levoso — 2023-03-12T18:00:17.153Z

TL;DR: I think there likely are a lot of people who want to work on mechanistic interpretability projects but couldn’t get into AI safety camp, so I created a

Discord server

for people to organize themselves into such projects. I’m going to lead

one of them

.

Why?

Many people interested in mechanistic interpretability projects were unable to get into AI safety camp. According to Linda Lisenfors, one of the main bottlenecks was a lack of project proposals.

The most popular project at AI safety camp was on understanding search in Transformers, with 46 people selecting it as their first choice. This suggests significant interest in mechanistic interpretability projects.

Moreover, I think that many mechanistic interpretability projects are low-hanging fruit that don’t require lots of experience or specific knowledge to execute. Anyone with basic CS skills can probably learn what they need to contribute along the way.

Mechanistic Interpretability Group

All in all, it would be a shame if people who are interested in working on mechanistic interpretability end up not doing so due to a lack of available projects and project leads.

With that in mind, I have created a Discord server to encourage people to propose and organize their own projects. You can join here:

https://discord.gg/cMr5YqbU4y

This Discord server aims to help interested individuals with forming project teams, whilst being provided with guidance from other researchers in the area. You are welcome to join even if you don’t think you have what it takes to participate in such projects: it is also a place to learn and discover beginner-friendly resources. You can also

check out Neel Nanda's guide to getting started on the field

.

Algorithm distillation project

I’m also organizing a

project

on interpreting models that use the algorithm distillation setup. If you are interested, join the Discord server and go to the #algorithm-distillation-project channel to indicate your willingness to participate.

The project will start once I have a better idea of who wants to join and how I’m going to organize it, in about 1-2 weeks.

Final thoughts

How exactly things will go on from here depends on how popular the Discord server ends up being. If the Discord server barely gets enough people for the algorithm distillation project, then I will focus on that and the server will become the home of that project. But...
Paper Replication Walkthrough: Reverse-Engineering Modular Addition
by Neel Nanda — 2023-03-12T13:25:46.400Z
Link: https://neelnanda.io/modular-addition-walkthrough

I'm excited about trying different formats for mechanistic interpretability education! I've made a video walkthrough where we replicate my paper, Progress Measures for Grokking via Mechanistic Interpretability. With Jess Smith, one of my co-authors, we record ourselves coding a replication and discussed what we did at each step. This is a three part walkthrough and you can see the accompanying code for the walkthrough here:

In part 1, we train a model to perform modular addition, and see that it does grok!

In part 2, we take this model and reverse-engineer the trig-based circuit it's learned to do modular addition. We show that you can both read out intermediate steps of the circuit from the activations, and that you can just read off some of the algorithm's steps from the model weights.

In part 3, we define some progress measures that let us distinguish progress towards the generalising and the memorising algorithm. We then look at the model during training and watch how the circuits develop, and use this to understand why it groks.

This is an experiment with a new format, and I'd love to hear about how useful you find it!
What problems do African-Americans face? An initial investigation using Standpoint Epistemology and Surveys
by tailcalled — 2023-03-12T11:42:32.614Z

This post is also available at my substack.

This post started from a bit of a weird place. I was in a Discord chatroom, and someone started complaining that Standpoint Epistemology had been “taken way past its carrying weight”.

I didn’t know much about Standpoint Epistemology, so I asked for various examples and resources about it. The resources she gave me that were written by Standpoint Epistemologists seemed relatively reasonable, and the resources that criticized it seemed to me to either be misrepresenting what Standpoint Epistemologists were saying, or to be criticizing people for something other than excessive Standpoint Epistemology.

At some point I got to the conclusion that in order to evaluate these things, it would really be useful for me to apply some Standpoint Epistemology myself. Specifically, since a lot of the discussion in the Discord server was about black people’s experiences with racism, I thought I should apply Standpoint Epistemology to this. In this post, I want to detail how I went about this, and what my results were, so that others can learn from it, and maybe usefully apply Standpoint Epistemology themselves.

Disclaimer: As you will see, this is not a thorough investigation into what African-Americans want. Rather, it is a brief initial investigation, which suggests places for further investigation and further learning. This is probably more a practical tutorial into how I would apply Standpoint Epistemology than an article on race issues per se.

What is Standpoint Epistemology?

It may be good to think of Standpoint Epistemology as an erisology, i.e. a theory of disagreement. If you observe a disagreement, Standpoint Epistemology provides one possible answer for what that disagreement means and how to handle it.

According to Standpoint Epistemology, people get their opinions and beliefs about the world through their experiences (also called their standpoint). However, a single experience will only reveal part of the world, and so in order to get a more comprehensive perspective, one must combine multiple experiences. In this way the ontology of Standpoint Epistemology heavily resembles rationalist-empiricist epistemologies such as Bayesian Epistemology, which also assert that people get their opinions by accumulating experiences that contain partial information.

One important difference is that whereas rationalists often focus on individ...
"Liquidity" vs "solvency" in bank runs (and some notes on Silicon Valley Bank)
by rossry — 2023-03-12T09:16:45.630Z

Epistemic status: Reference post, then some evidenced speculation about emerging current events (as of 2023-03-12 morning).

A "liquidity" crisis

There's one kind of "bank run" where the story, in stylized terms, starts like this:

A bank opens up and offers 4%/ann interest on customer deposits.

100 people each deposit $75 to the bank.

The bank uses $7,500 to buy government debt that will pay back $10,000 in five years. Let's call this "$10,000-par of Treasury notes", and call that a 5%/ann interest rate for simplicity. (Normally, government debt pays off a bit every month and then a large amount at the end, but that's just the same thing as having a portfolio of single-payout (or "zero coupon") notes with different sizes and maturity dates, and the single-payout notes are easier to think about, so I'm going to use them here.) We're going to assume for this entire post that government debt never defaults and everyone knows that and assumes it never defaults.

The thing you hope will happen is for every depositor to leave their money for five years, at which point you'll repay them $95 each and keep $500—which is needed to run the bank.

Instead, the next week, one customer withdraws their deposit; the bank sells $100-par of T-notes for $75, and gives them $75 back. No problem.

A second customer withdraws their deposit; oops, the best price the bank can get for $100-par of T-notes, right now after it just sold a bit, is $74. Problem.

But next week, let's say, it would be possible to sell another $100-par for $75 again.

At this point, the simplified bank is stuck. If it sells ~$101-par of T-notes to return the $75 deposit, it won't have enough to pay everyone else back, even if the withdrawals stop here! But if it doesn't give the depositor back $75 right now, then bad things will start to happen.

Equity capital: A liquidity solution

So, we fix this problem by going back in time and starting with an extra step that's now required by law:

Before taking $7,500 of deposits, the bank has to raise 10% of that—so, $750—of what we'll call "equity capital". Equity capital will get used to fill the gap between asset sales and returned deposits

Now, the final step of the original story goes differently:

$1 of equity capital, plus the $74 from the T-notes sale, go to repaying the withdrawn deposit.

Now the bank has 98*$75 of deposits, and $749 of equity capital. If not

...
"You'll Never Persuade People Like That"
by Zack_M_Davis — 2023-03-12T05:38:18.974Z

Sometimes, when someone is arguing for some proposition, their interlocutor will reply that the speaker's choice of arguments or tone wouldn't be effective at persuading some third party.

This would seem to be an odd change of topic. If I was arguing for this-and-such proposition, and my interlocutor isn't, themselves, convinced by my arguments, it makes sense for them to reply about why they, personally, aren't convinced. Why is it relevant whether I would convince some third party that isn't here?

What's going on in this kind of situation? Why would someone think "You'll never persuade people like that" was a relevant reply?

"Because people aren't truthseeking and treat arguments as soldiers" doesn't seem like an adequate explanation by itself. It's true, but it's not specific enough: what particularly makes appeal-to-persuading-third-parties an effective "soldier"?

The bargaining model of war attempts to explain why wars are fought—and not fought; even the bitterest enemies often prefer to grudgingly make peace with each other rather than continue to fight.

That's because war is costly. If I estimate that by continuing to wage war, there's a 60% chance my armies will hold a desirable piece of territory, I can achieve my war objectives equally well in expectation—while saving a lot of money and human lives—by instead signing a peace treaty that divides the territory with the enemy 60/40.

If the enemy will agree to that, of course. The enemy has their own forecast probabilities and their own war objectives. There's usually a range of possible treaties that both combatants will prefer to fighting, but the parties need to negotiate to select a particular treaty, because there's typically no uniquely obvious "fair" treaty—similar to how a buyer and seller need to negotiate a price for a rare and expensive item for which there is no uniquely obvious "fair" price.

If war is bargaining, and arguments are soldiers, then debate is negotiation: the same game-theoretic structure shines through armies fighting over the borders on the world's political map, buyer and seller haggling over contract items, and debaters arguing over the beliefs on Society's shared map. Strong arguments, like a strong battalion, make it less tenable for the adversary to maintain their current position.

Unfortunately, the theory of interdependent decision is ... subtle. Although recent work poin...