trentmkelly/LessWrong-43k huggingface dataset:
COVID China Personal Advice (No mRNA vax, possible hospital overload, bug-chasing edition)
by Lao Mein — 2022-12-14T10:31:22.902Z

So, uh, things in China are getting a bit interesting.

There's a general consensus that the public COVID numbers no longer reflect reality. Baidu started hiding their case tracking display. Their last updates had weird inconsistencies between asymptomatic and symptomatic cases. Testing stations are going offline. While previously a single case of close contact would have been enough to shut down an entire college, now facilities are running even with multiple symptomatic cases. Every time things have loosened to this degree before, cases have spiked. Many people I know mentioned family members being infected, some of which are unreported, while official case numbers are on the decline. The security people who checked health app codes aren't even pretending to care anymore. I was explicitly told not to show them my health codes when I come to work today. Lines for testing booths at peak times are basically non-existent.

I think this is it. COVID will go full endemic in China. Exponential.

I'm in my late 20s and in great physical condition. I exercise multiple times a week (weightlifting only). Unfortunately, I only have 2 shots of Sinovac. I'm planning to get some dental work taken care of this weekend and will get a third shot if it's available.

I think I have 2 options, really. First, I fly to Hong Kong and get the mRNA vax there, which an executive at my company mentioned she planned to do. Not sure if the EV is going to be positive, and I'll have to take a week off work. Second, I go to a hospital/visit a poz family friend and intentionally poz myself ASAP before the hospitals get overloaded.

Overall, I'm not too concerned, but am interested in what you guys think. I would also love to hear any investment advice from those familiar with Chinese markets.
Beyond a better world
by Davidmanheim — 2022-12-14T10:18:26.810Z
Link: https://progressforum.org/posts/pgad37kxSC9yND8Ba/beyond-a-better-world

As I take man’s last step from the surface, back home for some time to come — but we believe not too long into the future — I’d like to just say what I believe history will record: that America’s challenge of today has forged man’s destiny of tomorrow. And, as we leave the Moon at Taurus–Littrow, we leave as we came and, God willing, as we shall return, with peace and hope for all mankind. - Eugene Cernan, 14 December, 1972

The past 50 years of the great stagnation have been marked by a marked decline in humanity’s ambitions. I am referring to, among other things, humanity’s unfortunate retreat from space exploration. Today, marks 50 years since a human being left Low Earth Orbit. Of course, there has long been the argument that we need to focus on making things right here on earth, and keep our heads, and astronauts, firmly planted on the ground.

And here on earth, there has been progress. Most notably, we have seen amazing advances in more broadly shared prosperity. At an interpersonal level, violence and discrimination against minorities and women is an ongoing problem, but one that is thankfully (if too-slowly) being addressed. Child abuse is now rare and widely condemned, instead of a fact of life for most children. And obviously, life expectancies have been greatly increased. Humanity has eliminated smallpox, and is poised to do the same for polio. Global poverty has declined precipitously, and while poverty is far from eliminated, the worst-off fraction of the population in most of the world today has access to foods, entertainment, and material comforts undreamt of by kings centuries ago.

Of course, there is the concern that with prosperity and newer technology comes capacity for violence, and through World War Two it seemed humanity was on a trajectory to destroy itself. But instead of destruction, we have seen a continuation and expansion of the post-WWII long peace. While this is at present threatened, the Western world has taken steps to curtail future territorial incentives to violence, reaffirming post-WWII norms against territorial conquest. Our international structures have been wildly successful.

Even newer threats like climate change and engineered pandemics are being addressed - slowly, but with every expectation of success. These new and more global problems could not have been managed by a world at war with itself, but by-and-larg...
Proof as mere strong evidence
by adamShimi — 2022-12-14T08:56:26.790Z
Link: https://epistemologicalvigilance.substack.com/p/proof-as-mere-strong-evidence

Proofs at first exude a strong aroma of certainty.

On closer inspection though, more subtle notes emerge. Despite the dreams of formalists, the overwhelming majority of proofs in use and circulation are not minute low-level logical derivations, but rather intuitive high-level explanations. Some of which turn out to be false even after peer-review. Similarly, the level of details varies enormously across accepted proofs.

And yet it works, most of the time. Enough to build massive and intricate systems, and enough for millions of daily practical applications in the physical world.

What gives?

I think this mystery dissolves if we refuse to give proofs a special status. Instead of being the one and only form of certainty accessible to man, they make far more sense as “just” a strong form of evidence in the traditional scientific sense.

If proof is only evidence, then obviously different contexts call for different levels of details. We accept many geometric or intuitive proofs of the Pythagorean theorem, but ask for detailed checks on tens-of-pages-long advanced algebraic geometry proofs.

The other positive externality of desacralizing proofs comes in the form of methodological flourishing. Indeed, if proofs are merely sources of evidence, then any other way to generate evidence about formal statements is as valid, as long as the total evidence ends up approximately the same.

Or to put it more provocatively: proofs are not necessary if we find alternative ways of gathering the same amount of evidence.

Nowhere has this rich ground been better exploited than in computational complexity theory, the field studying the resources needed (time, memory, coin flips) to solve computational problems. For you see, conjectures in complexity theory are atrociously hard to prove. They’re some of the hardest across all of mathematics. But complexity theorists still want to advance their field, and so they have evolved an opportunistic methodology that accumulates evidence through alternative ways (based on proofs of other, simpler statements mostly).

Let’s look at two example methods to get an idea:

(Approximation algorithms breaking) Imagine that we have a class of problem C that we conjecture cannot be solved with a certain amount of resources (say polynomial time). But we have approximation algorithms that can approximate the solutions to problems in C within the resource bounds. If w

...
Trying to disambiguate different questions about whether RLHF is “good”
by Buck — 2022-12-14T04:03:27.081Z

(A few of the words in this post were written by Ryan Greenblatt and Ajeya Cotra. Thanks to Oliver Habryka and Max Nadeau for particularly helpful comments.)

Sometimes people want to talk about whether RLHF is “a promising alignment strategy”, or whether it “won’t work” or “is just capabilities research”. I think that conversations on these topics are pretty muddled and equivocate between a bunch of different questions. In this doc, I’ll attempt to distinguish some of these questions, and as a bonus, I’ll give my opinions on them.

I wrote this post kind of quickly, and I didn’t have time to justify all the claims I make; I hope that this post is net helpful anyway. I’m sympathetic to

claims

that alignment researchers should err more on the side of writing fewer but better posts; maybe I’ll regret making this one now instead of waiting.

Is “make a powerful AGI by using RLHF, where the feedback is coming from unaided humans”, a promising strategy for building aligned AGI?

IMO, this seems like the baseline, “we didn’t really try much at all” alignment scheme. Ajeya calls this a “naive safety effort” in her

training game post

, which lays out the basic case for pessimism about this strategy.
Here’s how I’d quickly summarize my problems with this scheme:

Oversight problems:

Overseer doesn’t know: In cases where your unaided humans don’t know whether the AI action is good or bad, they won’t be able to produce feedback which selects for AIs that do good things. This is unfortunate, because we wanted to be able to make AIs that do complicated things that have good outcomes.
Overseer is wrong: In cases where your unaided humans are actively wrong about whether the AI action is good or bad, their feedback will actively select for the AI to deceive the humans.

Catastrophe problems:

Even if the overseer’s feedback was perfect, a model whose strategy is to lie in wait until it has an opportunity to grab power will probably be able to successfully grab power.

I don’t think we’re 100% doomed if we follow this plan, but it does seem pretty likely to go badly.
RLHF with unaided humans is not literally the most doomed alignment scheme I’ve ever heard seriously proposed. For example, “train models with automated rewards (e.g. simulating evolution, or training models on a curriculum of math problems) and hope that the resulting models are aligned” might be a worse plan. (Though it

...
How can one literally buy time (from x-risk) with money?
by Alex_Altair — 2022-12-13T19:24:06.225Z

This post talks about types of projects that can buy us more time to work on AI alignment. But like, I am just definitely not going to pivot to working on any of those projects right now. What pre-existing projects could someone donate money to, right now? (This seems especially relevant in the shadow of lost FTX funding.)
Best introductory overviews of AGI safety?
by JakubK — 2022-12-13T19:01:37.887Z
Link: https://forum.effectivealtruism.org/posts/aa6wwy3zmLxn7wLNb/best-introductory-overviews-of-agi-safety

I'm interested what people think are the best overviews of AI risk for various types of people. Below I've listed as many good overviews as I could find (excluding some drafts), splitting based on "good for a popular audience" and "good for AI researchers." I'd also like to hear if people think some of these intros are better than others (prioritizing between intros). I'd be interested to hear about podcasts and videos as well.

I am maintaining a list at this Google doc to incorporate people's suggestions.
Popular audience:

Vox

(Kelsey Piper)
AI alignment (Wikipedia -- perhaps the most important to get right!)

Why alignment could be hard with modern DL

(Ajeya Cotra)

Stampy wiki

and aisafety.info (Stampy team)

The most important century blog post series summary

and AI could defeat all of us combined and

Why would AI "aim" to defeat humanity

(Holden Karnofsky)

Current work in AI alignment

(Paul Christiano)

Future of Life Institute

(Ariel Conn)

Why worry about future AI?

(Gavin Leech)

80k full profile

(Benjamin Hilton)

AGI Ruin: A list of lethalities

(Eliezer Yudkowsky)
Extinction Risk from Artificial Intelligence (Michael Cohen)

Set Sail For Fail? On AI risk

(Nintil)

A shift in arguments for AI risk

(Tom Adamczewski)

Is power-seeking AI an x-risk

(Joseph Carlsmith)

More is different for AI

(Jacob Steinhardt)

The Basic AI Drives

(Steve Omohundro)
AI Risk Intro 1: Advanced AI Might Be Very Bad (TheMcDouglas and LRudL)
Four Background Claims (Nate Soares)

AI researchers:

CHAI bibliography

Unsolved Problems in ML Safety

(Dan Hendrycks)

AI safety from first principles

(Richard Ngo)

The alignment problem from a DL perspective

(Richard Ngo)

X-risk analysis for AI research

(Dan Hendrycks and Mantas Mazeika)

Why I think more NLP researchers should engage with AIS concerns

(Sam Bowman)
More thoughts on outreach to researchers:

Marius Hobbhahn

,

AISFB

and

Resources I sent to AI researchers about AIS

(Vael Gates)

Alignment landscape:

My Overview of the AI Alignment Landscape: A Bird's Eye View

(Neel Nanda)

My understanding of what everyone in technical alignment is doing and why

(Thomas Larsen and Eli Lifland) and

Alignment org cheat sheet

(Thomas Larsen and Akash Wasil)

Podcasts and videos:

Intro to AI Safety, Remastered

(Rob Miles)
Researcher Perceptions of Current and Future AI (Vael Gates)
AI Fire Alarm (Connor Leahy)
Eliezer Yudkowksy interview with Sam Harris -- full audio is still

...
Applications open for AGI Safety Fundamentals: Alignment Course
by Richard_Ngo — 2022-12-13T18:31:55.068Z

The AGI Safety Fundamentals (AGISF): Alignment Course is designed to introduce the key ideas in AGI safety and alignment, and provide a space and support for participants to engage, evaluate and debate these arguments. Participants will meet others who are excited to help mitigate risks from future AI systems, and explore opportunities for their next steps in the field.

The course is being run by the same team as for previous rounds, now under a new project called BlueDot Impact.

Apply here, by 5th January 2023.

Time commitment

The course will run from February-April 2023. It comprises 8 weeks of reading and virtual small-group discussions, followed by a 4-week capstone project.

The time commitment is around 4 hours per week, so participants can engage with the course alongside full-time work or study.

Course structure

Participants are provided with structured content to work through, alongside weekly, facilitated discussion groups. Participants will be grouped depending on their ML experience and background knowledge about AI safety. In these sessions, participants will engage in activities and discussions with other participants, guided by the facilitator. The facilitator will be knowledgeable about AI safety, and can help to answer participants’ questions.

The course is followed by a capstone project, which is an opportunity for participants to synthesise their views on the field and start thinking through how to put these ideas into practice, or start getting relevant skills and experience that will help them with the next step in their career.

The course content is designed by Richard Ngo (Governance team at OpenAI, previously a research engineer on the AGI safety team at DeepMind). You can read the curriculum content here.

Target audience

We are most excited about applicants who would be in strong position to pursue technical alignment research in their career, such as professional software engineers and students studying technical subjects (e.g. CS/maths/physics/engineering).

That said, we consider all applicants and expect 25-50% of the course to consist of people with a variety of other backgrounds, so we encourage you to apply regardless. This includes community builders who would benefit from a deeper understanding of the concepts in AI alignment.

We will be running another course on AI Governance in early 2023 and expect a different distribution of target par...
What Does It Mean to Align AI With Human Values?
by Algon — 2022-12-13T16:56:37.018Z
Link: https://www.quantamagazine.org/what-does-it-mean-to-align-ai-with-human-values-20221213/

The author has some weird misunderstandings about what AI-will-kill-everyone-ism advocates belive, but seems to have a weirdly[1] decent grasp of the problem, given their aforementioned misunderstandings. They argue IRL won't be enough[2]. Here's the interesting quote IMO:

It should be clear that an essential first step toward teaching machines ethical concepts is to enable machines to grasp humanlike concepts in the first place, which I have argued is still AI’s most important open problem.

An example of a weird misunderstanding:

Moreover, I see an even more fundamental problem with the science underlying notions of AI alignment. Most discussions imagine a superintelligent AI as a machine that, while surpassing humans in all cognitive tasks, still lacks humanlike common sense and remains oddly mechanical in nature. And importantly, in keeping with Bostrom’s orthogonality thesis, the machine has achieved superintelligence without having any of its own goals or values, instead waiting for goals to be inserted by humans.Moreover, I see an even more fundamental problem with the science underlying notions of AI alignment. Most discussions imagine a superintelligent AI as a machine that, while surpassing humans in all cognitive tasks, still lacks humanlike common sense and remains oddly mechanical in nature. And importantly, in keeping with Bostrom’s orthogonality thesis, the machine has achieved superintelligence without having any of its own goals or values, instead waiting for goals to be inserted by humans.

^

By "weird" I mean "odd for the class of people writing pop sci articles on AI alignment whilst not being within the field".

^

For some reason they think "Many in the alignment community think the most promising path forward is a machine learning technique known as inverse reinforcement learning." Perhaps they're making a bucket error and lumping in CIRL with IRL?
It Takes Two Paracetamol?
by Eli_ — 2022-12-13T16:29:35.012Z

Note: Depending on where you live paracetamol is also known as acetaminophen, APAP and Tylenol.

I have had a version of the following conversation a surprising number of times:

"I have a headache."

"Aah, that sucks. Did you take some paracetamol?"

"Yes, but it still hurts."

"How many did you take?"

The fact that I even ask this question is because I have been noticing a trend.

"One."

"You should take two, it makes a big difference."

"Nah, I'm fine with one."

"But you still have a headache."

"Yeah, but it doesn't make much of a difference. Paracetamol just doesn't work that well for me."

Everyone is free to choose how many paracetamol they take, but I've always had this hunch that there is a big difference between taking one or two paracetamol. Where taking one has almost no effect on my headaches, but taking two makes a big difference. Effectively I'm making two claims:

1000mg paracetamol works better than 500mg paracetamol.
500mg paracetamol doesn't work much better than placebo.

Today I chose to finally resolve this mystery and laugh in the face of my friends! Or I will admit in shame that I was wrong...

Claim 2: 500mg vs placebo

There is a Cochrane Review from 2016 that tries to answer exactly my question.[1] The review looks at pain relief with paracetamol for tension-type headaches, the most common form of headaches. They reference two studies that have compared 500mg/650mg with placebo that also fit their inclusion criteria.

Steiner 2003[2], which comes to the following conclusion:

They didn't find a significant result when comparing 500mg paracetamol and placebo! But they did find a significant result when comparing 1000mg paracetamol and placebo.

Dahlöf 1996 is less conclusive. This study only contains 29 participants, where only 14 of them took paracetamol at a dose of 500mg.

In this study both 500mg and 1000mg performed equally well, but with the low participant numbers it is hard to draw any conclusions.

The Cochraine Review itself comes to the conclusion that they couldn't find a significant difference between 500mg paracetamol and placebo, but that the quality of evidence is low.

Claim 1: 1000mg vs 500mg

The same Cochrane Review concludes that a 1000mg paracetamol outperforms placebo. There is high quality evidence for this being the case when looking at

Pain-free at two hours: the proportion of attacks/participants wh

...
[Interim research report] Taking features out of superposition with sparse autoencoders
by Lee Sharkey — 2022-12-13T15:41:48.685Z

We're thankful for helpful comments from Trenton Bricken, Eric Winsor, Noa Nabeshima, and Sid Black.

This post is part of the work done at

Conjecture

.

TL;DR: Recent results from Anthropic suggest that neural networks represent features in superposition. This motivates the search for a method that can identify those features. Here, we construct a toy dataset of neural activations and see if we can recover the known ground truth features using sparse coding. We show that, contrary to some initial expectations, it turns out that an extremely simple method – training a single layer autoencoder to reconstruct neural activations with an L1 penalty on hidden activations – doesn’t just identify features that minimize the loss, but actually recovers the ground truth features that generated the data. We’re sharing these observations quickly so that others can begin to extract the features used by neural networks as early as possible. We also share some incomplete observations of what happens when we apply this method to a small language model and our reflections on further research directions.

This post is not intended to be a polished research product. It’s a set of preliminary and incomplete results that we’d like to share early in case it benefits the broader mechanistic interpretability research community.

Introduction

Features in a neural network are represented in high dimensional neural activations. Recent

work

from Anthropic has demonstrated that neural networks probably use a phenomenon called superposition to represent more features than available dimensions. Superposition is a challenge for mechanistic interpretability because it leads to polysemanticity, which prevents us from telling simple, human-understandable stories about how individual representations in neural networks relate to each other. Superposition also introduces another, more practical problem: Many of the best known feature extraction methods are dimensionality reduction techniques that find at most N directions in an N dimensional space. But we need a method that finds more than N features in superposition (i.e. an ‘overcomplete’ set of features). If we could develop a method that decomposes a neural network’s activations into its component features, we would essentially be taking features out of superposition, thus resolving...
Is the ChatGPT-simulated Linux virtual machine real?
by Kenoubi — 2022-12-13T15:41:39.368Z

Context: Building A Virtual Machine inside ChatGPT

It very much triggers my "roll to disbelieve" reflex, but people seem to be talking about it as if it's real and not that surprising. I'd try it myself, but OpenAI is saying ChatGPT is overloaded (and I don't have an OpenAI account, although I assume that part would be solvable if not for ChatGPT being overloaded). Can anyone confirm from personal experience that this really works? If so, did you probe its limits, and where did it fail? (It obviously wouldn't be that hard for it to simulate a login sequence, but the entire interaction in the blog post seems ~impossible to me without either using an actual Linux VM or massive cherry-picking of the ChatGPT results.)