trentmkelly/LessWrong-43k huggingface dataset:
How to be skeptical about meditation/Buddhism
by Viliam — 2022-05-01T10:30:13.976Z

Here is how I think we should approach the topic of meditation/Buddhism in the rationalist community. The short version is that a meaningful "yes" requires a credible possibility of "no", and the long version is that:

If we post scientific studies showing that "meditation works", then we should either also post scientific studies showing that "meditation doesn't work" or explicitly mention their absence. Otherwise there is a possibility that simply by doing a lot of studies about any topic, 5% of them will confirm the hypothesis at "p<0.05". In other words, is there a meta-review on meditation research? (Then we should ask Scott Alexander to review it.)
There are many different claims made about the effects of meditation. I find it quite plausible that some of them may be true (e.g. "meditation helps you relax") and some others may be false (e.g. "meditation helps you remember your previous reincarnations"). So instead of talking about proving "meditation" we should talk about proving specific claims about meditation.
Actually, we should first make the list of "claims usually made about meditation" and then evaluate each of them individually. Otherwise, if we mention the claims that are supported by evidence, but keep silent about those that are not, it creates a biased overall picture, and contributes to a halo effect. (It is easier to assume that X is supported by evidence if all you know is that A, B, C are supported; compared to a situation where you know that A, B, C are supported, but D, E, F are not.)
The problem with anecdotal evidence about meditation is that we would get it even in a universe where meditation helps 1/3 of the population, does nothing for another 1/3, and actively harms the remaining 1/3. The people who get no or harmful results would simply stop doing it, the people who get useful results would continue... and one or two of them would happen to be high-status in the rationalist community.
Generally, how do you distinguish between "meditation only works for some people, or only in some situations" and "you are doing meditation wrong / not enough"?
What about the anecdotal evidence in the opposite direction, such as sex scandals of famous experts on meditation? (In context of Buddhism, "sex scandals" is not just a bad behavior, but specifically the kind of behavior that meditation is supposed to prevent. So I am not mentioning it here a

...
My Approach to Non-Literal Communication
by Isaac King — 2022-05-01T02:47:46.821Z

This is a linkpost for this post on my blog. It's primarily intended for my non-rationalist acquaintances, so regular LessWrong readers will likely be familiar with most of the concepts I mention. I'm cross-posting it here because there's enough overlap in content that I think some LessWrongers might find it useful, and because I'd be interested in getting feedback on my approach to daily communication.

Abstraction levels

Imagine a society much like ours, except that people always speak literally. No implicature, no hyperbole.

When someone asks "does this dress make me look fat", they get back a "yes" or "no", and can use that information to decide on what to wear.

People in this society are still subject to normal human emotions, so hearing that they look bad in a dress makes many of the askers sad. Some of the answerers feel sympathetic and don't want to be the cause of the asker's distress. So they start saying "no, it doesn't make you look fat", even if that's not true.

This behavior spreads and becomes normalized. Askers start asking the question not because they want to know the answer, but because they want to know how much the answerer cares about their feelings. Only a rude, uncaring person would make them feel bad by saying "you look fat in that dress". So answering "no" is a signal that they care about the asker enough to figure out what they're actually asking. The question has turned from one about the facts of the dress to one about the relationship between the speakers. It's better than asking "do you care about my feelings?" directly, because that question has far too obvious an answer; no one would ever answer it wrong, so it's not a credible signal of care.[1]

One day, someone wants to know if a dress makes them look fat. They're going to an important gala and want to make their best impression, so they need to know the real answer. They know that they can't just ask "does this dress make me look fat?", since those words actually mean "do you care about my feelings?". So they include a request that the answerer answer honestly. The answerer does so, the asker figures out what to wear, and then has a great time at the gala.

Other people start doing this too whenever they want to know the real answer. Now there are two types of people: The ones who want validation and the ones who want the answer. But now the difference is evident; if on...
Narrative Syncing
by AnnaSalamon — 2022-05-01T01:48:45.889Z

Partial attempt to make sense of: On Doing the Improbable

Epistemic status: On my inside view, this seems like a clear and useful distinction that could use a short, shared term. Hasn’t been checked much by others. Some edge cases I’m still confused about. Also I may have a suboptimal term; help generating the right term greatly appreciated.

I’d like a short term for such sentences as:

“The sand is lava; if you touch it you die” (when introducing the rules to a game, not making a false prediction)
“Colloquium is at 3pm on Wednesdays” (when creating a new colloquium, or adding weight to a common-knowledge agreement / Schelling point)
“We don’t shout around here” (stated normatively)

I’m going to suggest the term “narrative syncing.” I’d like to distinguish “narrative syncing” from “sharing information” of a sort whose accuracy is not intended to be created/affected by the act of thinking/stating/believing the given sentence, i.e. to distinguish the sentences above from such sentences as:

“It will probably rain tomorrow.”
“3 + 5 = 8.”
“The people over there in that organization seem to avoid shouting, and to somewhat shun those who shout – though I have no stance on whether that’s a good norm, I’m just describing/predicting it.”

That is, I’d like to use the term “narrative syncing” (or some better such term that y’all suggest; suggestions appreciated) to distinguish sentences whose primary purpose is to sync up with some set of other people as to how we all think or talk about a given thing around here (as opposed to sentences whose primary purpose is to describe a piece of outside reality).

A situation where the concept of “narrative syncing” seems to me to add clarity

One reason I’d like a short term for this, is IMO having such a term would’ve helped me understand some past social situations I was in.

Example: Narrative syncing about “helpful” careers

Sometimes participants at CFAR’s/MIRI’s “AI risks for computer scientists” program would ask me something like “what careers should I go into, if I want to help with AI risk?” And I would sort-of interpret this as a request for information, and would nevertheless feel a predictable frustration when I heard it as I braced for the subterranean/confusing-to-me conflict that I anticipated would follow my words, without really understanding why (I would anticipate this based partly on the particular person’s nonverbals as ...
Why hasn't deep learning generated significant economic value yet?
by Alex_Altair — 2022-04-30T20:27:54.554Z

Or has it, and it's just not highly publicized?

Five years ago, I was under the impression that most "machine learning" jobs were mostly just data cleaning, linear regression, working with regular data stores, and debugging stuff. Or, that was at least the meme that I heard from a lot of people. That didn't surprise me at the time. It was easy to imagine that all the fancy research results were fragile, or hard to apply to products, or would at the very least take a long time to adapt.

But at this point it's been quite a few years since there have existed machine learning systems that immensely impressed me. The first such system was probably AlphaGo -- all the way back in 2016! AlphaGo then spun off in to multiple better faster cheaper systems that I didn't even keep track of them. And since then I've lost track of the number of unrelated systems that immensely impressed me. And their capabilities are so general that I feel sure that they must be convertible into enormous economic value. I still believe that it takes a long time to boot up a company around novel research results, but I'm not actually well calibrated on how long that takes, and it's been long enough that it's starting to feel awkward, like my models are missing something. Here are examples of AI products that I wouldn't have been surprised if they existed by now, but which I don't think do. (I can imagine that many of these examples technically exist, but not at the level that I mean).

Spotify playlists that are actually just procedurally generated music of various genres
A tool that helps researchers/legislators/et cetera by summarizing papers, books, laws on demand
Tools that help people (like writers) brainstorm, flesh out ideas by generating further details, asking questions, etc
A version of photoshop but with tons of AI tools
Widely available self-driving cars
Physics simulators that are way faster
Paradigmatically different and better web search

So what's the deal? Here's a list of possible explanations. I've love to hear if anyone has evidence for any of them, or if you know of reasons not on the list.

The research results are actually not all that applicable to products; more research is needed to refine them
They're way too expensive to run to be profitable
Yeah, no, it just takes a really long time to convert innovation into profitable, popular product
Something something regulation?
The AI comp

...
Nuclear Energy - Good but not the silver bullet we were hoping for
by Marius Hobbhahn — 2022-04-30T15:41:25.983Z

This article was written by Samuel Scheuer and Marius Hobbhahn (equal contribution). It was crossposted from Marius' personal blog.

Nuclear power is a very polarized issue. Proponents think it is a clean solution to most of our energy problems and opponents think it is too dangerous or too expensive. In this post, we want to take a deeper look at the case for and against nuclear while trying to be as unbiased as possible.

Executive summary:

Disclaimer: Samuel does an MSc in Environmental Change and Management at Oxford but has no special focus on nuclear energy. Marius has no expertise in nuclear energy and just likes to dig deep into topics. There is a chance we might have misunderstood important details. You should think of the following as our best guess after spending a collective ~50 hours on it. Both of us were proponents of nuclear before doing this research but we are now considerably less convinced of its value.

Answering questions about nuclear power is more complicated than we assumed. However, there are a couple of things we can say with high confidence.

Nuclear power is a carbon-neutral, stable source of energy that gives you high independence from other countries and conflicts.
Nuclear power is likely very safe and creates way less harm than e.g. a coal power plant of similar size.
The nuclear waste problem is likely neither very big nor very expensive.
On average nuclear energy is more expensive than renewables (~2x) and we can expect this gap to widen in the future.

On the other hand, there are a couple of other questions that carry more uncertainty.

Some people argue that nuclear energy regulation is unreasonably stringent and thus prevents innovation and inflates prices. The majority of reports (pro- and anti- nuclear) don’t mention overregulation as a major problem. We think the narrative is plausible because of misaligned regulatory incentive structures but we aren’t very confident.
The “true cost” of nuclear energy is hard to determine because it depends on specific timelines (e.g. waste management) and discount factors. Therefore, all arguments around “nuclear is cheap/expensive” carry considerable uncertainty. Our best guess is that nuclear is and will stay on the more expensive side of energy production.
There are many people who claim that “nuclear just needs more innovation/research”. However, governments have invested large amounts of mo

...
Quick Thoughts on A.I. Governance
by Nicholas / Heather Kross — 2022-04-30T14:49:26.694Z
Link: https://www.thinkingmuchbetter.com/main/quick-thoughts-on-ai-governance/

My friend Devin Kalish recently convinced me, at least to some extent, to focus on A.I. governance in the short-to-medium term, more than technical A.I. safety.

The argument that persuaded me was this:

A key point of stress among AI doomsayers like Yudkowsky is that we, first of all, have too little time, and second of all, have no way to implement any of the progress alignment workers do make. Both of these are governance problems, not alignment problems. They are also, arguably, far easier to picture possible promising interventions for than alignment research.

To lay out the logic more explicitly...

The case for governance now (Skip if you're already convinced)

AI safety is urgent insofar as different capabilities groups are working towards it.

Different capabilities groups are propelled at least partly by "if we don't do this, another group will anyway", or the subtly different "we must do this, or another group will first".

(2) is a coordination problem, potentially solveable with community governance.

AI safety is less tractable insofar as capabilities groups don't have ways to implement alignment research into their work.

Solutions to (4) will, at some point, require groups/resources made available to capabilities groups.

(5) looks kinda like a governance problem in practice.

The AI alignment problem is quite hard, on the technical level. Governance work, as noted in (3) and (6), is both more tractable and more neglected than technical work. At least, it is right now.

The rest of this essay is less organized, but contains my thoughts for how and why this could work.

Specific stories of how this would really look in the real world, for real

An OpenAI team is getting ready to train a new model, but they're worried about it's self improvement capabilities getting out of hand. Luckily, they can consult MIRI's 2025 Reflexivity Standards when reviewing their codebase, and get 3rd-party auditing done by The Actually Pretty Good Auditing Group (founded 2023).

A DeepMind employee has an idea for speeding up agent-training, but is worried about its potential to get out of hand. Worse, she's afraid she'll look like a fearmonger if she brings up her concerns at work. Luckily, she can bring up her concerns with The Pretty Decent Independent Tip Line, where it can then go to her boss anonymously.

OpenAI, DeepMind, and Facebook AI Research are al

...
Discussion on Thomas Philippon's paper on TFP growth being linear
by Arjun Yadav — 2022-04-30T14:25:00.148Z
Link: https://forum.effectivealtruism.org/posts/hbPvAjuLviwkphDJj/discussion-on-thomas-philippon-s-paper-on-tfp-growth-being

Note: This post was written quite quickly and I'm not well versed in this subject matter.
Thomas' paper here and Dylan Matthews' excellent write-up on it here.
I would love to spark some discussion on this: total factor productivity growth being linear in many developed countries, not exponential, could potentially be very scary.
Of course, as Dylan mentioned, TFP has issues. I believe the main critique is that, due to its simplicity, it can sometimes remain the same even after changes in technology and productivity.
Note-Taking without Hidden Messages
by Hoagy — 2022-04-30T11:15:00.234Z

Summary

We can create two ANNs which use language 'notes' to explain their internal state. They will use language to communicate a mixture of language-as-understood-by-humans and additional, hidden information.

The form of the hidden information will vary between the two, and so by training one network to have notes which are similar both to those made by the other network and by the past state of that same network, we can strongly bias these notes towards containing human-comprehensible information.

Background

This post develops my previous ELK-based work, developing a 'note-taking' architecture, as explained in this post.

A basic summary is that we create a MuZero-like agent, and during its rollouts we force the agent to convert its latent space into human-like language (or other media) and back into latent space. We train it with hints in the human space to incentivize it to use language in the same way the humans do, and create a model called a Synonymizer to freely convert any human-comprehensible media into alternate media that preserves the meaning, and shuts down any detected attempt to hide additional information.

The core problem with this setup is that there is going to be a strong incentive to hide additional information in the ostensibly human-comprehensible media. These mappings will contain a mixture of using human concepts in the way that they are intended, and steganographic (hidden) usage of human language. I will denote human-comprehensible information as ¯H and steganographic information as ¯S. The challenge is to maximize the ratio of ¯H to ¯S when learning.

In my last post I suggested that training multiple models on different data and with different initialization could provide a means to remove this hidden information but this strategy was underdeveloped. This is what I'll expand upon in this post.

A Puzzle - Extracting only shared information

We have two similarly trained networks which map from observations to a latent space. We again have two mappings from these to a human space.

Crucially, through the pre-training we can expect these models to use language at least partly as humans understand it, and so this aspect of their communication will be similar in both models. For steganographic information, however, there's no Schelling point for how it should be communicated, and so we can expect thi...
How good is spending?
by tryactions — 2022-04-30T07:27:27.455Z

In classic microeconomics, both parties benefit from transactions:

Buyers receive a "consumer surplus" measured by the maximum amount they would've been willing to pay versus the actual price.
Sellers receive a "producer surplus" measured by the minimum amount at which they would've been willing to sell versus the actual price.

Let's assume I make a purchase that I will selfishly enjoy but which won't help anyone else, like a PS5.

On one hand: when I make a purchase that I'm happy with, I capture consumer surplus and provide the other party with producer surplus. I also put money in someone else's hands which they can use to make further beneficial trades. If I skipped the trade and set my money on fire instead, both of us would be worse off.

On the other hand: when I make a purchase, I'm consuming labor and resources that could've been used for other things. I and my trade partner may be better off, but there's a negative externality of using up some of civilization's finite productive capacity. I'm not sure how best to think of this, but I assume this negative externality shows up as higher labor and resource costs.

On average the first consideration clearly wins out -- otherwise, the world would be better off halting all trade. But what about on the margins? Has there been any work on quantifying the size of consumer + producer surplus versus negative externalities of production? Do we know anything about how this changes as average spending changes?
[Linkpost] New multi-modal Deepmind model fusing Chinchilla with images and videos
by p.b. — 2022-04-30T03:47:13.258Z

Seems to be flying under the radar so far. Maybe because it looks more like incremental progress at first glance, similar to what, for example, Aleph Alpha has done continuing the Frozen approach.

However, with the (possibly cherry-picked) examples, it looks to me a lot like the image/video/text-GPT-4 many are expecting.

Blogpost here. Paper here.
Salvage Epistemology
by jimrandomh — 2022-04-30T02:10:41.996Z

A funny thing happens with woo sometimes, in the rationality community. There's a frame that says: this is a mix of figurative stuff and dumb stuff, let's try to figure out what the figurative stuff is pointing at and salvage it. Let's call this "salvage epistemology". Unambiguous examples include the rationality community's engagement with religions, cold-reading professions like psychics, bodywork, and chaos magic. Ambiguous examples include intensive meditation, Circling, and many uses of psychedelics.

The salvage epistemology frame got locally popular in parts of the rationality community for awhile. And this is a basically fine thing to do, in a context where you have hyper-analytical programmers who are not at risk of buying into the crazy, but who do need a lens that will weaken their perceptual filters around social dynamics, body language, and muscle tension.

But there's a bad thing happens when you have a group that are culturally adjacent to the hyper-analytical programmers, but who aren't that sort of person themselves. They can't, or shouldn't, take for granted that they're not at risk of falling into the crazy. For them, salvage epistemology disarms an important piece of their immune system.

I think salvage epistemology is infohazardous to a subset of people, and we should use it less, disclaim it more, and be careful to notice when it's leading people in over their heads.
Learning the smooth prior
by Geoffrey Irving — 2022-04-29T21:10:18.064Z

Most of this document is composed of thoughts from Geoffrey Irving (safety researcher at DeepMind) written on January 15th, 2021 on Learning the Prior/Imitative Generalization, cross-examination, and AI safety via debate, plus some discussion between Geoffrey and Rohin and some extra commentary from me at the end. – Evan

Geoffrey on learning the smooth prior

Vague claims

This doc is about a potential obstacle to Paul’s learning the prior scheme (LTP). Before reading this doc, please read either Beth’s simplified exposition or Paul’s original.

The intention of this doc was to argue for two claims, but weakly since I don’t have much clarity:

LTP has an obstacle in assigning joint probabilities to similar statements.

The best version of LTP may collapse into a version of debate + cross-examination

However, I don’t quite believe (2) after writing the doc (see the section on “How LTP differs from cross-examination”).

Wall of text → generative model

As originally sketched, the prior z in learning the prior is a huge wall of text, containing useful statements like “A husky is a large, fluffy dog that looks quite like a wolf”, and not containing wrong facts like “if there are a lot of white pixels in the bottom half of the image, then it’s a husky” (statements taken from Beth’s post).

A wall of text is of course unreasonable. Let’s try to make it more reasonable:

No one believes that a wall of text is the right type for z; instead, we’d like z to be some sort of generative network that spits out statements. (The original proposal wasn’t really a wall of text either; the wall was just a thought experiment.)

We likely want probabilities attached to these statements so that the prior can include uncertain statements.

Whether we attach probabilities or not, the validity of a statement like “A husky is a large, fluffy dog that looks quite like a wolf” depends on the definition of the terms. At a high level, we presumably want to solve this with something like cross-examination, so that our generative z model can be independently asked what a husky is, what fluffy is, etc.

The high level LTP loss includes a log p(z) term: we need to be able to compute log probabilities for z as a whole. It’s at least plausible to me that humans can be asked to assign probabilities to individual statements like our husky statements, but stitching this together seems rough.

The inter

...