trentmkelly/LessWrong-43k huggingface dataset:
NYT: Lab Leak Most Likely Caused Pandemic, Energy Dept. Says
by trevor — 2023-02-26T21:21:54.675Z
Link: https://www.nytimes.com/2023/02/26/us/politics/china-lab-leak-coronavirus-pandemic.html

I don't have much to say about this publicly, other than the fact that NYT is probably the largest news website in the US and it's reporting is very influential and high-stakes as a result. Also, that if you ever noticed a news website routinely being horribly wrong about things before, you probably shouldn't forget about that.

Right now, anything critical of China seems to be popular, but that by itself doesn't actually tell us that much, since it might just be because lots of people in lots of places noticed that China-bashing gets clicks.

Webarchive link is here: https://web.archive.org/web/20230226211102/https://www.nytimes.com/2023/02/26/us/politics/china-lab-leak-coronavirus-pandemic.html

On my viewing of nytimes.com at ~1pm EST, it was the fourth article from the top. It was very easy to miss due to being sandwiched between eye-catching articles, including one with a large moving image.

(it's important to note that I check the site every morning, which is not very generalizeable, since most english-speakers currently find news articles through social media and not any news org's homepage).

_________________________________________________________

The conclusion, which was made with “low confidence,” came as America’s intelligence agencies remained divided over the origins of the coronavirus.

New intelligence has prompted the Energy Department to conclude that an accidental laboratory leak in China most likely caused the coronavirus pandemic, though U.S. spy agencies remain divided over the origins of the virus, American officials said on Sunday.

The conclusion was a change from the department’s earlier position that it was undecided on how the virus emerged.

Some officials briefed on the intelligence said that it was relatively weak and that the Energy Department’s conclusion was made with “low confidence,” suggesting its level of certainty was not high. While the department shared the information with other agencies, none of them changed their conclusions, officials said.

Officials would not disclose what the intelligence was. But many of the Energy Department’s insights come from the network of national laboratories it oversees, rather than more traditional forms of intelligence like spy networks or communications intercepts.

Intelligence officials believe the scrutiny of the pandemic’s beginnings could be important to improving global response to future...
[Link Post] Cyber Digital Authoritarianism (National Intelligence Council Report)
by Phosphorous — 2023-02-26T20:51:49.579Z
Link: https://www.dni.gov/files/ODNI/documents/assessments/NICM-Declassified-Cyber-Operations-Enabling-Expansive-Digital-Authoritarianism-20200407--2022.pdf

Disclaimer: The U.S. government selectively declassified this report for unknown reasons, and anti-Chinese-Communist-Party sentiment is high in Washington D.C. While I doubt this means the contents are factually inaccurate, it is likely framed to highlight dangerous or unethical aspects of CCP activity, and likely suffers from whatever bias one would expect from the National Security community.

I don't think that anything in this report is very surprising (50% of it is redacted, and it's only 6 pages anyways), but I think it is useful to get a sense of what some members of the national security community are thinking about re: China and digital authoritarianism.

Highlights

Concern around China and Russia using large scale data analysis to influence or coerce actors in the United States and other countries.
"Growing concern in Europe and other democracies about Chinese and Russian cyber actions and personal privacy creates an opportunity to propose alternatives to blunt digital authoritarianism"

My Takeaway

Maybe folks with an interest in national security topics could do work on understanding the risk of and preventing long-term, stable, AI powered totalitarianism. I'd be excited for more work in that area.
Maybe there could be interest among many stakeholders for AI governance work to help prevent or regulate capabilities that could contribute to digital authoritarianism. I might imagine this could overlap with AI governance efforts to reduce existential or catastrophic risks.
Reflections on Zen and the Art of Motorcycle Maintenance
by LoganStrohl — 2023-02-26T20:46:29.553Z

I read Zen and the Art of Motorcycle Maintenance by Robert Pirsig twice in 2019. I wrote reflections on many of the chapters as I went through the second time. I posted them to a Goodreads book group at the time, but I've decided to collect them here as well (unedited) because I keep referring to them and I'm tired of saying "but you may need a Goodreads account". They're chapter-by-chapter (though I did not have any notes for some chapters), so I don't know that they'll make any sense if you've never read the book.

There's nothing like a thesis here, and it's not a distillation. These are literally my journal entries. I don't know what you're likely to get out of this if you read it, but I think Zen and Motorcycles is important, and there are a lot of people who seem to like it when I think things and then say which things I thought. That's what this is.

Chapter 1

I like how vividly Pirsig evokes the anti-technology feels in me. I'm very sympathetic with John and Sylvia at the end of this chapter.
I'm strongly reminded of my recent visit to the San Francisco Museum of Modern Art. I did not have type-1 fun while I was there. In fact I became quite upset, and when I left I had the thought "burn that place to the ground".
I think it's because of what so many artists were doing in the second half of the 20th century. If the SF MOMA collection is representative, I think modern artists have been grappling with how to reconcile technology with human brains, mainly by demonstrating that doing so isn't automatic. I tried to pick out the least unpleasant pieces there to spend time with, but even given that selection criterion, two out of three of the pieces I really sat with made me feel isolated, disoriented, jarred, betrayed, and in contact with something horribly alien.
I've wondered about why I responded that way to so much of the art there. The heart of it seems to be that it's all "unnatural" and "inhuman". Popular elements include

sharp angles

,

straight lines

,

sudden color transitions

(lack of gradients),

literal randomness

, and

deliberately thwarted expectations

. All of these elements are rare in nature, and indeed in the structure of our own minds. At one point I went out to a court yard where there's a wall completely covered in plants, and I've seldom felt such tremendous relief.

Chapter 2

I'm most interested in two questions at the end of this chapter. 1) What do...
Taboo "human-level intelligence"
by Sherrinford — 2023-02-26T20:42:25.880Z

Maybe calls for tabooing terms can become a bit inflationary. However, talking about if and when AI will reach or surpass “human-level intelligence” can misleadingly frame the discussion, so I recommend avoiding that term.
What is "human-level intelligence" and when will AI surpass it?
My understanding is that there is a wide spectrum of human intelligence, and when people are discussing AI capabilities and threats, it is important to know where in that spectrum the AI in question sits.
Moreover, intelligence is not a single-dimensional concept. Sure, there is g etc. - in human brains. But even an ordinary calculator can, well, calculate much faster than humans. For being dangerous, an AI does not need to be better than all people in every discipline. Does the AI have to be a great composer, or art historian? No.
Yet the vagueness of the concept of "human-level intelligence" may lead to further misleading ways of assessing AI capabilities. Firstly, dismissing domain-specific abilities; "sure, this AI is great at explaining jokes but it cannot invent new salsa recipes, whereas this is something that humans can do." True, but the human brain is quite modular, subsystems do different things, so why should an AI not also use different AIs? Secondly, comparing with the best human experts in some domain; "look, this AI may win a game of Diplomacy, but not against the best players." Fine, but if human-level intelligence means that you have to be best at everything, then no human has human-level intelligence. This would only be relevant if humanity coordinated itself perfectly.
Additionally, to be dangerous, an AI does not have to be conscious. But this is something else that the term "human-level intelligence" may suggest. Why, if the term does not contain the concept of consciousness? The reason for this is that the lack of concreteness about what human intelligence is could invite people to substitute what they believe exists in everyone's brain, regardless of their intelligence, and that is consciousness. Saying that AI transcending human-level intelligence is a relevant threshold may sound to people as if the real danger is AIs developing consciousness.
So avoid the term "human-level intelligence" if you can.

Thanks to Justis Mills for feedback on a draft of this post.
[Link] Petition on brain preservation: Allow global access to high-quality brain preservation as an option rapidly after death
by Mati_Roy — 2023-02-26T15:56:55.656Z
Link: https://www.change.org/p/allow-global-access-to-high-quality-brain-preservation-as-an-option-rapidly-after-death

I've put a few extracts I found notable below, but check out the URL for the full text and to sign the petition. Note that signing the petition doesn't mean you necessarily want to preserve your own brain.

Extracts I found notable:

In 2016, the Brain Preservation Foundation Prize was awarded for a scalable technique that – for the first time – demonstrably preserved the key features of brain structure that neuroscientists know are involved in learning and memory.

[...]

Our best current models suggest that carefully preserving the pattern of connections within the human brain, i.e. our "connectome," alongside essential biomolecules that regulate those connections, will allow the retention and later automated reconstruction of unique and valuable information about a person’s identity, including their long-term memories.

[...]

Here, we are performing advocacy for brain preservationists. We call upon the global scientific, medical, and legal communities, as well as local and national governments, to immediately make changes in norms, policy, and laws in end-of-life medicine and care, to allow brain preservation to be accessible as an elective procedure at death, and for this procedure to be regulated so that it is done in a competent, sustainable, and humane way.

[...]

Currently, with our existing medico-legal frameworks, elective brain preservation is often delayed after death, in ways that people cannot opt out of, even after legal and medical preparations. This delay leads to unnecessary decomposition of the brain, and may irreparably damage the structures that contain our memories. These restrictive laws must be changed with haste.

[...]

We also understand that brain preservation can be expensive and that many of the existing methods are not currently attainable for all people at the end of life. We call upon scientists, technologists, and physicians to develop low-cost options, and for governments and philanthropists to make the necessary funds available so that all individuals who wish to pursue brain preservation at the end of life will be able to have access to it.

PS: I provide free cryonics sign up support (and a 50% discount for Alcor's 1st year of membership); you can email me at mathieu.roy.37@gmail.com.
Some thoughts on the cults LW had 
by Noosphere89 — 2023-02-26T15:46:58.535Z

There have been several cults in LW history, whether it's the Zizians, or Leverage cults.

Today I want to talk about these cults and the tradeoffs LW makes.

Zizians

The Zizians were a cult that focused on relatively extreme animal welfare, even by EA standards, and used a Timeless/Updateless decision theory, where being aggressive and escalatory was helpful as long as it helped other world branches/acausally traded with other worlds to solve the animal welfare crisis.

They apparently made a new personality called Maia in Pasek, and this resulted in Pasek's suicide.

They also used violence or the threat of violence a lot to achieve their goal.

This caused many problems for Ziz, and she now is in police custody.

Edit: Removed the Vassarites due to updated information that they didn't use psychedelics to jailbreak people's minds.

Now I'll talk about what in my view are the tradeoffs of the level of cults.

And I think one tradeoff is that the less cults you have, the more you miss weird opportunities that generate most of the value.

EDIT: I have removed the takeaways section due to the fact that we don't have base rates for people entering cults..
A library for safety research in conditioning on RLHF tasks
by James Chua — 2023-02-26T14:50:56.762Z

There is a couple of discussions regarding conditioning / decision transformer training. "Conditioning" as in, placing your reward model's reward in the prompt. We then train our language model to create a completion that follows the specified reward.

See Safety considerations for online generative modeling, Soft optimization makes the value target bigger, RLHF bad, conditioning good.

The tldr is that training models this way could have safety benefits.

I've created a library so that we can extend a pre-trained LLM (gpt2, gpt-j) to work with conditioning by scalar rewards. This allows researchers to save time by avoiding the need to modify attention masks, positions, and labels themselves. For example, researchers can retrain GPT-2 to replicate OpenAI's summarization RLHF paper, but by relying purely on conditioning. I created it because I couldn't find an existing library that did so.

Note: an easier way of conditioning would be to use discrete tokens. Pretraining models With Human Preferences implements conditioning via discrete 
 and 
 tokens.

However using scalar rather than discrete rewards could have the following benefits

The ability to specify a higher reward than the maximum reward available during training. For example, during training the maximum reward obtained was 1. However during inference you can set it to 2. Would we get a better result, as demonstrated in the decision transformer paper? (Thanks @Tomek Korbak for pointing this out)
More efficient training since we don't lose information during discretization (yet to be empirically shown?)

While the library is still in early development, it can already be used for offline training of GPT-2.

I'm writing this post earlier rather than later to:

Get feedback on whether this is useful. Or that it isn't useful and I should spent time on something else :)
Get feedback on how to improve the library if the above statement is true.
Save someone hours trying to do the same thing (just in case that happens)
The Preference Fulfillment Hypothesis
by Kaj_Sotala — 2023-02-26T10:55:12.647Z

Short version

Humans have an innate motivation ("preference fulfillment", PF) to fulfill the preferences of those they care about. It corresponds to at least some of the senses of the word "love", as well as related words such as "kindness" and "compassion".

I hypothesize that it works by simulating the other person and predicting what they would want or how they would like to be treated. PF is when you take your simulation of what other people would want and add an extra component that makes you intrinsically value outcomes that your simulation predicts the other people would prefer.

I also hypothesize that this is the same kind of simulation that forms our ability to work as a social species in the first place. A mental simulation process is active in virtually every situation where we interact with other people, such as in a grocery store. People use masks/roles/simulations to determine the right behavior in any social situation, running simulations of how others would react to various behaviors. These simulations involve both the actual people present in the situation as well as various other people whose opinions we've internalized and care about. The simulations generally allow people to engage in interactions by acting the way a normal person would in a given situation.

Once you have this kind of a simulation, constantly running in basically any social situation, it’s likely already exhibiting the PF drive to a weak degree. Doing things that we expect to fulfill other people’s preferences often feels intrinsically nice, even if the person in question was a total stranger. So does wordless coordination in general, as evidenced by the popularity of things like dance.

If this is true, capabilities progress may then be closely linked to alignment progress. Getting AIs to be better at following instructions requires them to simulate humans better. Once you have an AI that can simulate human preferences, you already have most of the machinery required for having PF as an intrinsic drive. This is contrary to the position that

niceness is unnatural

. The preference fulfillment hypothesis is that niceness/PF is a natural kind that will be relatively easy to get out of any AI smart enough to understand what humans want it to do. This implies that constructing aligned AIs might be reasonably easy, in the sense that most of the work necessary for it will...
"Rationalist Discourse" Is Like "Physicist Motors"
by Zack_M_Davis — 2023-02-26T05:58:29.249Z

Imagine being a student of physics, and coming across a blog post proposing a list of guidelines for "physicist motors"—motor designs informed by the knowledge of physicists, unlike ordinary motors.

Even if most of the things on the list seemed like sensible advice to keep in mind when designing a motor, the framing would seem very odd. The laws of physics describe how energy can be converted into work. To the extent that any motor accomplishes anything, it happens within the laws of physics. There are theoretical ideals describing how motors need to work in principle, like the Carnot engine, but you can't actually build an ideal Carnot engine; real-world electric motors or diesel motors or jet engines all have their own idiosyncratic lore depending on the application and the materials at hand; an engineer who worked on one, might not the be best person to work on another. You might appeal to principles of physics to explain why some particular motor is inefficient or poorly-designed, but you would not speak of physicist motors as if that were a distinct category of thing—and if someone did, you might quietly begin to doubt how much they really knew about physics.

As a student of rationality, I feel the same way about guidelines for "rationalist discourse." The laws of probability and decision theory describe how information can be converted into optimization power. To the extent that any discourse accomplishes anything, it happens within the laws of rationality.

Rob Bensinger proposes "Elements of Rationalist Discourse" as a companion to Duncan Sabien's earlier "Basics of Rationalist Discourse". Most of the things on both lists are, indeed, sensible advice that one might do well to keep in mind when arguing with people, but as Bensinger notes, "Probably this new version also won't match 'the basics' as other people perceive them."

But there's a reason for that: a list of guidelines has the wrong type signature for being "the basics". The actual basics are the principles of rationality one would appeal to explain which guidelines are a good idea: principles like how evidence is the systematic correlation between possible states of your observations and possible states of reality, how you need evidence to locate the correct hypothesis in the space of possibilities, how the quality of your conclusion can only be improved by arguments that have the power to change ...
Ways to prepare to a vastly new world? 
by Annapurna — 2023-02-26T04:56:12.505Z

I've come to the realization that humanity is at a place similar to where the Aztecs were prior to the arrival of Cortez.

The big difference is that several of us are very confident that Cortez (AGI) will arrive within our lifetimes. I don't know exactly when, and there is a lot of debate as to whether it will kill us all, but I find that there is a general agreement that it's on its way.

Unlike some of you I am not having a deep existential crisis, but I am having a lot of thoughts on how different the world is going to be. There might be very drastic changes in things such as law, government, religion, family systems, economics, etc. I am having trouble coming up with ways to prepare for these drastic changes.

Is anyone thinking of ways to modify their lifestyle in order to reduce the shock of the AGI arrival?
Incentives and Selection: A Missing Frame From AI Threat Discussions? 
by DragonGod — 2023-02-26T01:18:13.487Z

Epistemic Status

Written quickly, originally as a Twitter thread.

Thesis

I think a missing frame from AI threat discussion is incentives (especially economic) and selection (pressures exerted on a system during its development).

I hear a lot of AI threat arguments of the form: "AI can do X/be Y" with IMO insufficient justification that:

It would be (economically) profitable for AI to do X

The default environments/training setups select for systems that are Y

That is such arguments establish that somethings can happen, but do not convincingly argue that it is likely to happen (or that the chances of it happening are sufficiently high). I think it's an undesirable epistemic status quo.

Examples

1: Discrete Extinction Events

Many speculations of AI systems precipitating extinction in a discrete event[1].

I do not understand under what scenarios triggering a nuclear holocaust, massive genocide via robot armies or similar would be something profitable for the AI to do.

It sounds to me like just setting fire to a fuckton of utility.

In general, triggering civilisational collapse seems like something that would just be robustly unprofitable for an AI system to pursue[2].

As such, I don't expect misaligned systems to pursue such goals (as long as they don't terminally value human suffering/harm to humans/are otherwise malevolent).

2. Deceptive Alignment

Consider also deceptive alignment.

I understand what deceptive alignment is, how deception can manifest and why sufficiently sophisticated misaligned systems are incentivised to be deceptive.

I do not understand that training actually selects for deception though[3].

Deceptive alignment seems to require a peculiar combination of situational awareness/cognitive sophistication that complicates my intuitions around it.

Unlike with many other mechanisms/concepts we don't have a clear proof of concept, not even with humans and evolution.

Humans did not develop the prerequisite situational awareness/cognitive sophistication to even grasp evolution's goals until long after they had moved off the training distribution (ancestral environment) and undergone considerable capability amplification.

Insomuch as humans are misaligned with evolution's training objective, our failure is one of goal misgeneralisation not of deceptive alignment.

I don't understand well how values ("contextual influences on decision making") form in intel...