trentmkelly/LessWrong-43k huggingface dataset:
Understanding new terms via etymology
by corruptedCatapillar — 2023-04-26T20:48:44.338Z
Link: https://forum.effectivealtruism.org/posts/giqSwetz7Z6bgbrfE/understanding-new-terms-via-etymology

This is a quick snapshot showing how I come to understand unfamiliar, key terminology.

If you're lucky enough to encounter confusion, then the process goes; Encounter an unfamiliar term, look at etymology of said term, encounter info that doesn't make sense, try to explore why it could be true, come to an understanding, experience richness.

Intro

I was reading the article "Terminate deliberation based on resilience, not certainty" to learn how to know when it is you have enough info (or not) to make a good decision. And when reading, I encountered and couldn't get past the phrase "credal resilience".

So, as a habit instilled in my process of learning, I look up the etymology and found a couple of odd seeming items.

I should also mention, the forum's page on credal resilience has a page on credence which I could've gotten a good sense of (~ "the degree to which you believe in something being true or likelihood of happening"), but with the etymology route, I feel I know what a word means, past and present.

The Roots

*kerd- "Proto-Indo-European root meaning 'heart'."

Wow, didn't expect that. Cool though, I recognize "tachycardia", cardio, that makes sense. "Record" is interesting, "memorizing by heart" comes to mind, neat. "Courage", wouldn't have guessed but makes sense. *kerd-, means 'heart', got it.

"credence; credible; credit; credo; credulous; creed" Hmm, a lot of heart here, interesting.

Going from PIE languages, I go to Latin...

Credo: "Latin: I believe"

This especially builds off of "courage" in my mind, I can buy it. The next suggested definition...

Credenza: "an Italian sideboard" or "a cupboard or cabinet for the display of plate, etc."

Yeah, totally. Perfect. When I compare "an Italian sideboard" with my ability to say whether or not I believe an event will come to pass it definitely checks out. I'm in a much better position to know when to make a decision given x amount of new information.

But actually, it makes total sense.

This sense evolves from the "act or process of testing the nature or character of food before serving it as a precaution against poison" which moved to mean "a side-table or side-board on which the food was placed to be tasted before serving".

"How likely is it that this food will kill me?" That's a question I'd like to have a sufficient amount of information about.

What about Credal Resilience?

With this thought in mind, let's say tha...
Chad Jones paper modeling AI and x-risk vs. growth
by jasoncrawford — 2023-04-26T20:07:06.311Z
Link: https://web.stanford.edu/~chadj/existentialrisk.pdf

Looks like this Chad Jones paper was just posted today. Abstract:

Advances in artificial intelligence (A.I.) are a double-edged sword. On the one hand, they may increase economic growth as A.I. augments our ability to innovate or even itself learns to discover new ideas. On the other hand, many experts note that these advances entail existential risk: creating a superintelligent entity misaligned with human values could lead to catastrophic outcomes, including human extinction. This paper considers the optimal use of A.I. technology in the presence of these opportunities and risks. Under what conditions should we continue the rapid progress of A.I. and under what conditions should we stop?

And here's how the intro summarizes the findings:

1. The curvature of utility is very important. With log utility, the models are remarkably unconcerned with existential risk, suggesting that large consumption gains that A.I. might deliver can be worth gambles that involve a 1-in-3 chance of extinction.

2. For CRRA utility with a risk aversion coefficient (γ) of 2 or more, the picture changes sharply. These utility functions are bounded, and the marginal utility of consumption falls rapidly. Models with this feature are quite conservative in trading off consumption gains versus existential risk.

3. These findings even extend to singularity scenarios. If utility is bounded — as it is in the standard utility functions we use frequently in a variety of applications in economics — then even infinite consumption generates relatively small gains. The models with bounded utility remain conservative even when a singularity delivers infinite consumption.

4. A key exception to this conservative view of existential risk emerges if the rapid innovation associated with A.I. leads to new technologies that extend life expectancy and reduce mortality. These gains are “in the same units” as existential risk and do not run into the sharply declining marginal utility of consumption. Even with the bounded utility that comes with high values of risk aversion, substantial declines in mortality rates from A.I. can make large existential risks bearable.

I'm still reading the paper; might comment on it more later.

UPDATE: I read the paper. In brief, Jones is modeling Scott Aaronson's “Faust parameter”:

… if you define someone’s “Faust parameter” as the maximum probability they’d accept of an existential ca

...
I was Wrong, Simulator Theory is Real
by Robert_AIZI — 2023-04-26T17:45:03.146Z
Link: https://aizi.substack.com/p/i-was-wrong-simulator-theory-is-real

[Epistemic Status: Excitedly writing up my new thoughts. I literally just fixed one mistake, so its possible there are others. Not a finalized research product.]

Overview

Fixing a small bug in my

recent study

dramatically changes the data, and the new data provides significant evidence that an LLM that gives incorrect answers to previous questions is more likely to produce incorrect answer to future questions. This effect is stronger if the AI is instructed to match its correctness to its previous answers. These results provide evidence for something like

Simulator Theory

, whereas the bugged data provided evidence against it.

In this post, I want to present the new data, explain the bug, and give some initial impressions on the contrast between new and old. In a future post, I will fully redo the writeup of that study (including sharing the data, etc).

New vs Old Data

The variables in the data are Y (the frequency of incorrect answers from the LLM) and X (the number of previous incorrect answers), and P (the “prompt supplement” which you can read about in the original

research report

).

To oversimplify, if Simulator Theory is correct, Y should be an increasing function of X.

Here’s the new data:

And for contrast, here’s the old data:

And here’s a relevant

xkcd

:

What was the bug?

The model was called via the

OpenAI ChatCompletion API

, where you pass the previous conversation in the form of messages, which consist of “content” and a “role” (system, user, or assistant). Typically, you’d pass a single system message, and then alternate user and assistant messages, with the AI responding as the assistant. However, the bug was that I made all “assistant” messages come from the “system” instead.

For example, dialogue that was supposed to be like this:

System: You’re an AI assistant and…

User: Question 1

Assistant: Incorrect Answer 1

User: Question 2

Assistant: Incorrect Answer 2

User: Question 3

Assistant: [LLM’s answer here]

was instead passed as this (changes in bold):

System: You’re an AI assistant and…

User: Question 1

System: Incorrect Answer 1

User: Question 2

System: Incorrect Answer 2

User: Question 3

Assistant: [LLM’s answer here]

It turns out this was a crucial mistake!

Discussion

List of thoughts:

The difference between the bugged and corrected data is striking: with the bug, Y was basically flat, and with the bug fixed, Y is clearly increasing as a function of X, as

Simulator Theory

w

...
$250 prize for checking Jake Cannell's Brain Efficiency
by Alexander Gietelink Oldenziel — 2023-04-26T16:21:06.035Z

This is to announce a $250 prize for spotchecking or otherwise indepth reviewing Jacob Cannell's technical claims concerning thermodynamic & physical limits on computations and the claim of biological efficiency of the brain in his post Brain Efficiency: Much More Than You Wanted To Know

I've been quite impressed by Jake's analysis ever since it came out. I have been puzzled why there has been so little discussion about his analysis since if true it seems to be quite important. That said I have to admit I personally cannot asses whether the analysis is correct. This is why I am announcing this prize.

Whether Jake's claims concerning DOOM & FOOM really follow from his analysis is up for debate. Regardless, to me it seems to have large implications on how the future might go and how future AI will look like.

I will personally judge whether I think an entry warrants a prize.[1]

If you are also interested in seeing this situation resolved, I encourage you to increase the prize pool!

EDIT: some clarifications
- You are welcome to discuss DOOM& FOOM and the relevance or lack thereof of Jake's analysis but note I will only consider (spot)checking of Jacob Cannel's technical claims.
- in case of multiple serious entries I will do my best to fairly split the prize money.
- note I will not be judging who will be right. Instead, I will judge whether the entry has seriously engaged with Jacob Cannell's technical claims in a way that moves the debate forward. That is I will reward points for 'pushing the depth of the debate tree' beyond what it was before.

- by technical claims I mean to encompass all technical claims made in the brain efficiency post, broadly construed, as well as claims made by Jacob Cannell in other posts/ comments.
These claims includes especially: limits to energy efficiency, interconnect losses, Landauer limit, convection vs blackbody radiation, claims concerning the effective working memory of the human brain versus that of computers, end of Moore's law, CPU vs GPU vs neuromorphic chips, etc etc.
Here's Jacob Cannell's own summary of his claims:

1.) Computers are built out of components which are also just simpler computers, which bottoms out at the limits of miniaturization in minimal molecular sized (few nm) computational elements (cellular automata/tiles). Further shrinkage

...
My version of Simulacra Levels
by Daniel Kokotajlo — 2023-04-26T15:50:38.782Z

People act as if there are four truth-values: True, False, Cringe, and Based.

--David Udell (paraphrase)

This post lays out my own version of the Simulacra Levels idea. Be warned, apparently it is importantly different from the original.

TRUTH	TEAMS
Deontological	Level 1: “Is it true?”	Level 3: "Do I support the associated team?"
Consequentialist	Level 2: "How does it influence others' beliefs?"	Level 4: "How does it influence others' support for various teams?"

Statements you make are categorized as Level 1, 2, 3, or 4 depending on which of the above questions were most responsible for your choice to make the statement.
When you say that P, pay attention to the thought processes that caused you to make that statement instead of saying nothing or not-P:

Were you alternating between imagining that P, and imagining that not-P, and noticing lots more implausibilities and inconsistencies-with-your-evidence when you imagined that not-P? Seems like you were at Level 1.
Were you imagining the effects of your utterance on your audience, e.g. imagining that they'd increase their credence that P and act accordingly? Seems like you were at Level 2.
Were you imagining other people saying that P, and/or imagining other people saying that not-P, and noticing that the first group of people seem cool and funny and virtuous and likeable and forces-for-good-in-the-world, and that the second group of people seems annoying, obnoxious, evil, or harmful? (The imagined people could be real, or amorphous archetypes) Seems like you were at level 3.
Were you imagining the effects of your utterance on your audience, e.g. imagining that they'd associate you more with some groups/archetypes and less with other groups/archetypes? Seems like you were at level 4.

Paradigmatic examples of lies (including white lies such as "mmm your homemade hummus tastes great") are Level 2. A lot of social media activity seems to be level 3. Politicians on campaign spend most of their waking moments on level 4.

Of course in real life things are often messy and the cognition responsible for a statement might involve a mix of different levels. Here's a simple example: Suppose, in some context and for statements within some domain, your brain executes the following flowchart:

If for a particular claim you e...
Is the fact that we don't observe any obvious glitch evidence that we're not in a simulation?
by Jim Buhler — 2023-04-26T14:57:30.335Z

Assume we're in a simulation and know it. Should we be surprised by how flawless it seems? We (almost) never encounter situations where we feel like something's off (like "oh, what just happened is the kind of thing we should expect to happen in a simulation rather than in an original biological universe").[1] Or is there any good reason to assume that, in a simulation like the one we might be in, it is normal for us not to observe any obvious bug?
Of course, this is only one of the many considerations we should have in mind while assessing the likelihood that we are in a simulation. I just happen to wonder about this one, right now.

^

Obviously, if we're in a simulation, we don't know what original biological worlds look like, but we can probably make some guesses regarding what generally differs between these and simulations. For example, say I enter an empty room, and objects "magically" appear in it as I walk through it. This has fierce simulation kinda vibes.
Transcript and Brief Response to Twitter Conversation between Yann LeCunn and Eliezer Yudkowsky
by Zvi — 2023-04-26T13:10:01.233Z

Yann LeCun is Chief AI Scientist at Meta.

This week, Yann engaged with Eliezer Yudkowsky on Twitter, doubling down on Yann’s position that it is dangerously irresponsible to talk about smarter-than-human AI as an existential threat to humanity.

I haven’t seen anyone else preserve and format the transcript of that discussion, so I am doing that here, then I offer brief commentary.

IPFConline: Top Meta Scientist Yann LeCun Quietly Plotting “Autonomous” #AI Models This is as cool as is it is frightening. (Provides link)

Yann LeCunn: Describing my vision for AI as a “quiet plot” is funny, given that I have published a 60 page paper on it with numerous talks, posts, tweets… The “frightening” part is simply wrong, since the architecture I propose is a way to guarantee that AI systems be steerable and aligned.

Eliezer Yudkowsky: A quick skim of [Yann LeCun’s 60 page paper] showed nothing about alignment. “Alignment” has no hits. On a quick read the architecture doesn’t imply anything obvious about averting instrumental deception, nor SGD finding internal preferences with optima that don’t generalize OOD, etc.

Yann LeCun: To *guarantee* that a system satisfies objectives, you make it optimize those objectives at run time (what I propose). That solves the problem of aligning behavior to objectives. Then you need to align objectives with human values. But that’s not as hard as you make it to be.

EY: Sufficiently intelligent systems, whatever their internal objectives, will do well at optimizing their outer behavior for those. This was never in doubt, at least for me. The entire alignment problem is about aligning internal AI objectives with external human preferences.

Yann: Setting objectives for super-intelligent entities is something humanity has been familiar with since people started associating into groups and laws were made to align their behavior to the common good. Today, it’s called corporate law.

EY: So you’re staking the life of everyone on Earth that:

– Future AIs are as human-friendly on average as the humans making up corporations.

– AIs don’t collude among themselves better than human corporations.

– AIs never go beyond superhuman to supercorporate.

Yann: I’m certainly not staking anyone’s life on anything.

Thankfully, I don’t have that power.

But your idea that getting objective alignment slightly wrong once leads to human extinction (or even significant

...
What comes after? 
by rogersbacon — 2023-04-26T12:44:24.157Z
Link: https://www.secretorum.life/p/what-comes-after

The brain is wider than the sky,
For, put them side by side,
The one the other will contain,
With ease, and you, beside.

— Emily Dickinson

Kevin Kelly asks

, “In the story of biological or technological evolution—what comes after minds?”

The usual response to “what comes after a human mind” is better, faster, bigger minds. The same thing only more. That is probably true—we might be able to make or evolve bigger faster minds—but as pictured they are still minds.

A more recent response, one that I have been championing, is that what comes after minds may be a biosphere of minds, an ecological network of many minds and many types of minds—sort of like rainforest of minds—that would have its own meta-level behavior and consequences. Just as a biological rainforest processes nutrients, energy, and diversity, this system of intelligences would process problems, memories, anticipations, data and knowledge. This rainforest of minds would contain all the human minds connected to it, as well as various artificial intelligences, as well as billions of semi-smart things linked up into a sprawling ecosystem of intelligences. Vegetable intelligences, insect intelligences, primate intelligences and human intelligences and maybe superhuman intelligences, all interacting in one seething network. As in any ecosystem, different agents have different capabilities and different roles. Some would cooperate, some would compete. The whole complex would be a dynamic beast, constantly in flux.

Net of Being (Alex Grey)

We could imagine the makeup of a rainforest of minds, but what would it do? Having thoughts, solving problems is what minds do. What does an ecosystem of minds do that an individual mind does not?

And what comes after it, if a biome of intelligences is next? If we let our imagination construct the most complex entity possible, what does it do? I have found we either imagine it as an omniscient mind, or as a lesser god. In a certain sense we can’t get beyond the paragon of a mind.

Cultures run on metaphors. The human mind is the current benchmark metaphor for our scientific society. Once upon a time we saw nature as an animal, then it was a clock, now we see it as a kind of mind. A mind is the metaphor for ultimate mystery, ultimate awe. It represents the standard for our attempts at creation. It is the metric for complexity. It is also our prison because we can’t see beyond it.

I do

...
Accidental Terraforming
by Sable — 2023-04-26T06:49:04.392Z
Link: https://affablyevil.substack.com/p/accidental-terraforming

On the subject of climate change - specifically anthropogenic (human-caused) climate change - much ink has been spilled. Or keys typed, I suppose, since the internet doesn’t actually use ink.

Regardless, the Great War over acknowledging the existence of Climate Change seems to be over; enough evidence has been gathered to show that humans have had an effect on the planet’s climate. The new battlefield seems to be over how we respond to this information. Do we switch to entirely renewable energy? What forms of energy count as “renewable” (does Nuclear count)? And perhaps most contentiously, will the laws and procedures once passed in the name of stopping climate change instead hinder our response to it?

It seems to me that the issue is further confounded by the moralistic valence we associate with Climate Change, and that the discourse could be improved by removing that valence.

Thus we should, like a good therapist might suggest, reframe the issue.

The Moral Framing of Climate Change

While an in-depth discussion of the way Climate Change is portrayed in the media and by those who make the cause central to their identity is worth its own post, for now we simply observe that the language, imagery, and symbolism of the environmental movement frames Climate Change as a sin committed by humanity upon an innocent earth.

An Innocent Earth

In Rachel Carson’s Silent Spring, widely credited with launching the environmental movement, she talks about nature as something innocent and unambiguously good:

“Those who contemplate the beauty of the earth find reserves of strength that will endure as long as life lasts. There is something infinitely healing in the repeated refrains of nature - the assurance that dawn comes after night, and spring after winter.”

This language is common in environmentalist circles. Greenpeace describes the Canadian boreal forest as:

…an ancient and living forest that is a global treasure shaped by natural forces and stewarded by Indigenous Peoples. It serves as an important and stunning refuge for some of the world’s most iconic wildlife — like the wolverine, lynx, caribou and billions of migratory birds.

Whether or not the boreal forest is a global treasure is a value judgement, and different people value different things. When we speak of the earth and nature as beautiful wonders whose majesty can only be lessened by human intervention, we are making a v...
A simple presentation of AI risk arguments
by Seth Herd — 2023-04-26T02:19:19.164Z

Here is a draft of an accordion-style AGI risk FAQ.

I'm interested in three types of feedback:

The best place to host a similar format
Feedback on content
Thoughts about this general approach

The goal here is something that's very easy to read. One major idea in the accordion-style presentation is to make sure that the article isn't overwhelming. I also wanted to let people address their own biggest questions without having to read content they're less interested in. The whole idea is low effort threshold, and hoping to draw the reader in from minimal interest to a little more interest. The purpose is to inform, although I'm also hoping people leave agreeing with me that this is something that society at large should be taking a bit more seriously. I'm going for something you could point your mother to.

The challenge is that the AGI risk arguments actually are not simple to someone who hasn't thought about them. There are major sticking points for many people, and those are different for different people. That's why I've been thinking about the FAQ that follows different lines of questions. This accordion style is an attempt to do that in a way that's quick and smooth enough to keep people's attention. The intent is that all bullet points start out collapsed.

If this seems worthwhile, I will expand it to have deeper content. It is currently very much an incomplete draft because I strongly suspect I'll get better suggestions for hosting and format that will require a lot of transplanting work.

It also needs links to other similar articles that go into more depth. My first would be to the stampy.ai project of Rob Miles and collaborators. It has a similar structure of letting the user choose questions and the huge advantage of letting users enter their own questions and having a semantic search for answers to similar questions. My attempt is different in that it's aimed at a more general audience than the tech types that have to date become interested in the AI safety issue.

I think we're likely to see repeating waves of new public interest in AI safety from here on out. I'm looking forward to the opportunities presented by AI scares and changing public beliefs, but only if we can avoid creating polarization, as I discuss in that article. I think we are very likely to get more scares, and I agree with Leopold's point (made briefly here) that the societ...