trentmkelly/LessWrong-43k huggingface dataset:
Rationalism before the Sequences
by Eric Raymond — 2021-03-30T14:04:15.254Z

I'm here to tell you a story about what it was like to be a rationalist decades before the Sequences and the formation of the modern rationalist community. It is not the only story that could be told, but it is one that runs parallel to and has important connections to Eliezer Yudkowsky's and how his ideas developed.

My goal in writing this essay is to give the LW community a sense of the prehistory of their movement. It is not intended to be "where Eliezer got his ideas"; that would be stupidly reductive. I aim more to exhibit where the drive and spirit of the Yudkowskian reform came from, and the interesting ways in which Eliezer's formative experiences were not unique.

My standing to write this essay begins with the fact that I am roughly 20 years older than Eliezer and read many of his sources before he was old enough to read. I was acquainted with him over an email list before he wrote the Sequences, though I somehow managed to forget those interactions afterwards and only rediscovered them while researching for this essay. In 2005 he had even sent me a book manuscript to review that covered some of the Sequences topics.

My reaction on reading "The Twelve Virtues of Rationality" a few years later was dual. It was a different kind of writing than the book manuscript - stronger, more individual, taking some serious risks. On the one hand, I was deeply impressed by its clarity and courage. On the other hand, much of it seemed very familiar, full of hints and callbacks and allusions to books I knew very well.

Today it is probably more difficult to back-read Eliezer's sources than it was in 2006, because the body of more recent work within his reformation of rationalism tends to get in the way. I'm going to attempt to draw aside that veil by talking about four specific topics: General Semantics, analytic philosophy, science fiction, and Zen Buddhism.

Before I get to those specifics, I want to try to convey that sense of what it was like. I was a bright geeky kid in the 1960s and 1970s, immersed in a lot of obscure topics often with an implicit common theme: intelligence can save us! Learning how to think more clearly can make us better! But at the beginning I was groping as if in a dense fog, unclear about how to turn that belief into actionable advice.

Sometimes I would get a flash of light through the fog, or...
TAI?
by Logan Zoellner — 2021-03-30T12:41:29.790Z

It looks like people around here are now using the acronym TAI with the accompanying definition "transformative AI is AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution."

Is there some kind of consensus that this hasn't already happened?

Because my current belief is that if Moore's laws stopped tomorrow and there was absolutely 0 innovation in AI beyond what GANs and Transformers give us, the social implications are already of that magnitude, they're just not "evenly distributed".

Here's what I think a world where our current level of AI becomes evenly distributed looks like:

AI is built into every product imaginable and used for almost every task.
Most labor (including almost physical labor) has been replaced by robots. The jobs that remain consist of research and application of AI and robotics.

Note: jobs like entertainer, teacher, philosopher, historian, YouTube influencer, etc still exist but these are voluntary in the sense that they do not contribute to providing for the ongoing physical needs of humankind.

Universal Basic Income means the vast majority of people no longer need to work.
"Popular" entertainment is generated using AI and individualized to the taste of the viewer. That is to say human scripted TV, movies and video games still exist, but in the same way that plays exist in our current world.
Space travel becomes routine and humanity is a multi planetary species (I'm not really sure we need AI for this one, but I bet people on Mars will be using robots to clean their solar farms and watching AI generated content instead of waiting for the 30 minute delay to download media from Earth).

So for those who don't think TAI exists, is the claim:

The story you've told requires innovations that do not yet exist
The story you've told doesn't count as FAI
Something else?

Specifically, "If Moore's law stopped tomorrow and there are no more 'breakthroughs' in AI --I'm not counting what an expert in 2021 would consider an obvious or incremental improvement or application-- what would a world where such technology was 'evenly distributed' look like, and how would it fall short of TAI?"

Edit: I thought I should add that I don't think the industrial revolution is "evenly distributed" yet either. Let's posit the industrial age as ending with the introduction of the ...
How do we prepare for final crunch time?
by Eli Tyre — 2021-03-30T05:47:54.654Z

[Crossposted from Musings and Rough Drafts.]

Epistemic status: Brainstorming and first draft thoughts.

Inspired by something that Ruby Bloom wrote and the Paul Christiano episode of the 80,000 hours podcast.]

One claim I sometimes hear about AI alignment [paraphrase]:

It is really hard to know what sorts of AI alignment work are good this far out from transformative AI. As we get closer, we’ll have a clearer sense of what AGI / Transformative AI is likely to actually look like, and we’ll have much better traction on what kind of alignment work to do. In fact, MOST of the work of AI alignment is done in the final few years (or months) before AGI, when we’ve solved most of the hard capabilities problems already so we know what AGI will look like and we can work directly, with good feedback loops, on the sorts of systems that we want to align.

Usually, this is said to argue that to value of the alignment research being done today is primarily that of enabling, future, more critical, alignment work. But “progress in the field” is only one dimension to consider in boosting and unblocking the work of alignment researchers in this last stretch.

In this post I want to take the above posit seriously, and consider the implications. If most of the alignment work that will be done is going to be done in the final few years before the deadline, our job in 2021 is mostly to do everything that we can to enable the people working on the problem in the crucial period (which might be us, or our successors, or both) so that they are as well equipped as we can possibly make them.

What are all the ways that we can think of that we can prepare now, for our eventual final exam? What should we be investing in, to improve our efficacy in those final, crucial, years?

The following are some ideas.

[In this post, I'm going to refer to this last stretch of a few months to a few years, "final crunch time", as distinct from just "crunch time", ie this century.]

Access

For this to matter, our alignment researchers need to be at the cutting edge of AI capabilities, and they need to be positioned such that their work can actually be incorporated into AI systems as they are deployed.

A different kind of work

Most current AI alignment work is pretty abstract and theoretical, for two reasons.

The first reason is a philosophical / methodological claim: There’s a fundamental “nearest unblocked strategy” ...
Logan Strohl on exercise norms
by Rob Bensinger — 2021-03-30T04:28:22.331Z

(cw: exercise, shame)

(also relevant: Shame Processing)

Logan Strohl said some things about exercise (and shame, and aspiring to be better) on Facebook today that I found interesting. Minus some mostly-irrelevant parts:

[...] I definitely wasn't trying to kick people who struggle with exercise while they're down.

But I guess when I make this claim about exercise out loud, I am trying to do something sort of similar to that, which is kind of the opposite of saying, "oh don't worry, exercise doesn't matter anyway". Exercise does matter.

[...]

Being difficult is part of the very nature of exercise; you don't get physically stronger or gain physical endurance unless you work hard enough to be uncomfortable, and you don't improve meaningfully in these ways unless you are uncomfortable over and over again frequently and consistently. So I think sentiments like "it doesn't really matter, only do it if it's easy and you like it" (which rarely pop up in those words but I think are nevertheless subtly pervasive in nerdier subcultures) are especially detrimental to exercise in particular. And that's why I do sometimes bother to promote exercise explicitly.

[...]

I mostly don't think people should be shamed for things, including not exercising. I think shame is a beautiful and powerful psychological process that probably ought to be treated as personal and intimate, much like recountings of first lovemakings. Trying to use it as a public tool to make people act how you want them to seems to break it.

What I think is that there are much, much better reactions to recognizing the goodness of something than shutting down and feeling bad about yourself for not instantiating the good thing. Such as, for example, re-considering whether you are allocating your resources correctly. I suspect that one of the reasons people tend to reject "X is better than Y" style claims is because they aren't quite aware that there is a difference between "someone thinks I'm worse than I could be" and "someone is trying to make me feel bad". Which seems like a great big dumb obstacle to people getting better.

I suspect that part of what's going on here in our differing perceptions is... two things.

1. in general, nerds struggle more with exercise than non-nerds, nerds get bullied by non-nerds as kids (largely for struggling more with exercise), and then as adults nerds form sub-cultures where they're more

...
Julia Galef and Matt Yglesias on bioethics and "ethics expertise"
by Rob Bensinger — 2021-03-30T03:06:07.323Z

I recently collected excerpts from 33 random bioethics papers from 2014–2020, plus 10 papers from 2000, in the hope of getting a better picture of what the field is like, how healthy it is, and what its social effects may be.

One of the conversations that prompted me to look into this was Julia Galef's Feb. 3, 2021 interview of Matt Yglesias. The exchange is interesting in its own right, so I've copied it below. The relevant discussion pivots from the Iraq War to COVID-19:

Julia: Well, not to bring it back to metacognitive updates, but I was wondering whether there's an update in here about trusting expertise.

Because with the mistakes that public health communicators have made -- in communicating whether masks work, and how big of a deal COVID was going to be -- there've been a series of debates this year about whether people should just “trust experts,” on a topic that they aren't an expert in. And a lot of people have pointed out that the failure of the public health establishment kind of undercuts that “trust the experts” policy.

So do you think there's a similar thing that should be said about the Iraq War? Where the, not politicians, but the “experts” in the intelligence community really fucked up, so should that update us away from “Trust experts in general”?

Or is it kind of an exception, for some reason?

Matt: It's hard for me to know what the alternative to trusting experts is exactly --

Julia: I mean, it's like having a higher prior on… If you're pretty confident that war is really bad, and should be avoided unless absolutely necessary, then if you have lower trust in the intelligence community, it should mean you should need a really, really compelling case, with strong evidence, before you go with their recommendation.

Matt: Yeah, I think in both foreign policy and public health that… particularly thinking after the pandemic, which has shook me in a variety of ways, is: I feel like I need to be raising the bar for what is the actual subject matter, that the subject matter expert is an expert in?

A lot of what we get is adjacent expertise. So somebody who studies viruses, and maybe knows a lot about the protein structure of viruses, will opine about masks, right? And you’ve got to ask yourself, "Do they have subject matter expertise in this mask thing? Whose expertise do we need here?"

Because one thing that I think clearly came out of the whole mask

...
Progressive Highlighting: Picking What To Make Into Flashcards
by DirectedEvolution — 2021-03-29T22:57:27.649Z

Using a flashcard system like Anki can help maintain long-term knowledge. To use flashcards effectively, it's important to analyze how they fit into the larger task of reading and review.

Understanding and Prioritization: A Chicken and Egg Problem

We can't make flashcards of every bit of knowledge conveyed in our textbook. How do we know what's important and what's not, unless we are already familiar with the subject? After all, the sentences in the textbook can be:

Core knowledge that we want to make into flashcards
Underlying principles or equations that are extremely important, but will come up so many times that we don't need flashcards to remember them.
Details that we can re-derive or infer if we remember the "core knowledge"
Tangents and examples that help explain or give context to the core knowledge, but aren't critical to remember
Filler material

Many learners attempt to make notes and flashcards as they go, but lack the understanding that would let them prioritize better. Others just skim or read, then return later, but will have forgotten much of the material along the way.

Mini-Towers

As I wrote in The Multi-Tower Study Strategy, some topics are split into discrete, minimally-intersecting "boxed topics." This makes them easier to learn, since on your first encounter, forgetting about one won't damage your ability to understand the next.

Other topics are "tower topics," which build on a set of fundamentals, layer by layer. You are constantly at risk of forgetting the fundamentals, even while new information is piled on. This makes them harder than "boxed topics."

Of course, almost all topics are "tower topics" on some level. A single page of dense biochemistry might introduce a multi-step reaction, requiring you to absorb and understand a number of chemical names, reaction types, products, and the way the overall reaction fits into larger pathways.

If you read closely, sentence by sentence, looking up terms you don't know, you can individually remember and understand via your short-term memory. But as you plow through more and more information, you can easily get overloaded, forgetting the beginning by the time you reach the end, even though you understood each step.

Simply going back and re-reading immediately is a form of "massed practice," and may be an inefficient use of your time.

Most learning comes in the form of "mini-towers," and learning how to ...
Some Complaint-Action Gaps
by mike_hawke — 2021-03-29T21:15:50.012Z

[This was previously a shortform post]

Sometimes I scroll social media (because I am yet weak) and I see rationalists raising Concerns about various news topics and current events. I am bothered by what appears to be a high ratio of (raising awareness) : (taking action).

To better articulate this, here is a list of some of the concerns I hear a lot lately. Each issue has a sublist of potential actions, starting with the least useful. I’m definitely interested in hearing better ideas too.

Our institutions are losing trustworthiness and competence

Shout about the CDC on social media

#GhostTheNYT

Try to understand what leads to institutional success/failure

Try to advance incentive-compatible systems and social tech that can scale[1]

Freedom of speech is under attack! Gatekeepers! Reality czar! The Narrative!

Take traditional political action

Less twitter, more substack

Research and post about the economics and logistics of running your own servers.

Experiment with ring signatures?

Social media is harming mental health, undermining individual & group epistemics, and enabling horrible actions in meatspace

Complain about it on social media in between dunking on your outgroup.

Await humanity’s inexorable slide down the fitness gradients toward memetic collapse.

Promote more group activities in your group house. Establish device-free zones.

Coordinate to stigmatize doomscrolling, hate-reading, and contempt addiction. While we’re at it, let’s stigmatize using Twitter at all.

Large, free, liquid prediction markets cannot exist, supposedly due to the global influence of burdensome US trading regulations.

Traditional political action

Pay a competent YouTuber to make a really good explanatory video (series) about prediction markets.

Write a postmortem on Augur to help similar endeavors in the future avoid the same pitfalls

Research and post about why prediction markets haven’t gotten big even where regulation is looser

Start up a competitor to Kalshi, and launch before they do

Ongoing Uighur atrocity

Share an article about it once in a while

Take traditional political action, push for substantial international response

Make fewer purchases from companies that profit from the abuse of Uighurs

See Vitalik Buterin: Credible Neutrality, Legitimacy, Change The Incentives, Change The World ↩︎
Systematizing Epistemics: Principles for Resolving Forecasts
by Davidmanheim — 2021-03-29T20:46:06.923Z

In a

previous post

, I discussed many methods for resolving predictions. I want to argue that there is a systematic distinction between rules and principles which I think is valuable.

In short, when making rules, one can front-load intentions by writing details upfront, or back-load work by stating high-level principles and having procedures to decide on details on an as-needed basis*. American accounting systems rely on the former, and international accounting systems (and most law systems) focus more on the latter. I think that the question shouldn’t be implicitly decided by front loading assumptions, which is often the current default. More than that, I think the balance should be better and more explicitly be addressed.

Reframing the Problem

Ozzie Gooen's new organization, QURI (pronounced the same as "query") is interested in what he's started to call "systematizing epistemics," and he offered an analogy that I found very insightful - accounting. Just like keeping track of money is possible without accounting, keeping track of reality is possible without any systematic approach to epistemics - but it's harder to communicate or agree about money without standardized accounting systems that talk about the same things the same way.

In the aforementioned

post

, I discussed a variety of ways to resolve predictions. Here, I want to present a more systematic argument about how to think about prediction systems and resolutions. To make this point, I plan to take a detour into accounting - but don’t worry, the post really is about predictions. I want to lay out the analogy between systematizing epistemics and systematizing accounting (even) more in a different post, but for now I'll jump to the key point for writing prediction questions and resolving predictions.

Accounting Principles versus Accounting Rules

In financial accounting, which is only half as boring as it sounds, there is a conceptual disagreement between

Rule-based or Principle-based methods

.

A rule based accounting system has (millions of) rules that get updated and adapted to deal with all of the new ways that clever accountants devise to lie with accounting. That is, a rule based system tries to cover every eventuality. This creates complexity, but still makes sure everything is legible to those who have the necessary expertise to decipher accounting statements. On the other hand, every time someone...
Bureaucracy is a world of magic
by dominicq — 2021-03-29T12:41:24.405Z

I previously wrote about some practical game-theoretical (game-practical?) realizations I had while buying a house. Today I want to talk about how bureaucracy is a ritualistic, magical place.

In our home-buying process, every step of the way, there were papers to be signed. Paperwork is how the magic of bureaucracy comes in view. I'm not saying "magic" to mean good or beautiful. I'm referring to the ritualistic nature of bureaucracy.

Everything in our journey was a ritual. When you debate the point of something, people participating in the ritual are confused. On the one hand, they understand that your request makes sense, because you're asking for the same function. On the other hand, you shall not ignore the Ritual!

Let me explain with several examples what I mean by ritual.

The Summoning (of the PDF)

To buy a house and get state subsidies, you have to present an official document to the bank, confirming that the building may indeed be used as a dwelling, i.e. a use permit. It is not necessary that this document is an original, a copy will suffice.

Well, I got to the bank with printouts of photos of this permit. I don't have the original, and the agent simply took photos of it with his phone, and sent these photos to me. I printed them out on paper, and presented them to the bank. Problem: they have to be scans, not photos. "Photos aren't scans", the bank lady said, "They won't be accepted as official". My first impulse was to protest: "But since you don't need originals, what does it matter what form the copy has? Obviously the informational content is what's necessary - what's written in the document, not what device was used to transfer this information. And anyway, scans and photos are literally the exact same thing. Scans are just photos taken in a particular way. How is it important that-", but I stopped myself before saying any of this. There's a particular art to navigating bureaucracy, and arguing about the nature of information and how it represented is Not It, Chief ®. Instead, the Art is to constantly weigh where you can insist on being reasonable, and where you have to suck it up and comply with a dumb request.

What the bank lady actually wanted is a semblance of officiality. Photos simply don't look official, and that's it. To complete the ritual, a conventional way is required, and the most modern of the conventional ways is the offering of a scan....