trentmkelly/LessWrong-43k huggingface dataset:
What do ML researchers think about AI in 2022?
by KatjaGrace — 2022-08-04T15:40:05.024Z

Katja Grace, Aug 4 2022

AI Impacts just finished collecting data from a new survey of ML researchers, as similar to the 2016 one as practical, aside from a couple of new questions that seemed too interesting not to add.

This page reports on it preliminarily, and we’ll be adding more details there. But so far, some things that might interest you:

37 years until a 50% chance of HLMI according to a complicated aggregate forecast (and biasedly not including data from questions about the conceptually similar Full Automation of Labor, which in 2016 prompted strikingly later estimates). This 2059 aggregate HLMI timeline has become about eight years shorter in the six years since 2016, when the aggregate prediction was 2061, or 45 years out. Note that all of these estimates are conditional on “human scientific activity continu[ing] without major negative disruption.”
P(extremely bad outcome)=5% The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. This is the same as it was in 2016 (though Zhang et al 2022 found 2% in a similar but non-identical question). Many respondents put the chance substantially higher: 48% of respondents gave at least 10% chance of an extremely bad outcome. Though another 25% put it at 0%.
Explicit P(doom)=5-10% The levels of badness involved in that last question seemed ambiguous in retrospect, so I added two new questions about human extinction explicitly. The median respondent’s probability of x-risk from humans failing to control AI1 was 10%, weirdly more than median chance of human extinction from AI in general2, at 5%. This might just be because different people got these questions and the median is quite near the divide between 5% and 10%. The most interesting thing here is probably that these are both very high—it seems the ‘extremely bad outcome’ numbers in the old question were not just catastrophizing merely disastrous AI outcomes.
Support for AI safety research is up: 69% of respondents believe society should prioritize AI safety research “more” or “much more” than it is currently prioritized, up from 49% in 2016.
The median respondent thinks there is an “about even chance” that an argument given for an intelligence explosion is broadly correct. The median respondent also believes machine intelligence will probably (60%) be “vastly

...
Interpretability isn’t Free
by Joel Burget — 2022-08-04T15:02:54.842Z

Epistemic status: This post uses a simple and intuitive model[1], which seems obvious in retrospect, but it hasn’t been formalized and I’m not confident in its conclusions.

Current consensus is that large language models will be a core component of transformative AI systems. If that’s the case, interpretability for language models is probably an important piece of solving alignment. Anthropic’s recent

Transformer Circuits Thread

is what I consider the state of the art in the field. They’ve done well to make quick progress, but I think their work also hints at an insoluble core to the problem.

In the most recent article,

Softmax Linear Units

, the authors replace the GeLU activation function in their transformer model with a new Softmax Linear Unit (SoLU). This is designed to discourage

polysemanticity / superposition

, in which a single neuron takes on multiple unrelated meanings. However,

SoLU is a double-edged sword for interpretability. On the one hand, it makes it much easier to study a subset of MLP layer features which end up nicely aligned with neurons. On the other hand, we suspect that there are many other non-neuron-aligned features which are essential to the loss and arguably harder to study than in a regular model. Perhaps more concerningly, if one only looked at the SoLU activation, it would be easy for these features to be invisible and create a false sense that one understands all the features.

Gwern

responded

on LessWrong:

Extremely concerning for safety. The only thing more dangerous than an uninterpretable model is an 'interpretable' model. Is there an interpretability tax such that all interpretability methods wind up incentivizing covert algorithms, similar to how CycleGAN is incentivized to learn steganography, and interpretability methods risk simply creating mesa-optimizers which optimize for a superficially-simple seeming 'surface' network while it gets the real work done elsewhere out of sight?

The theory is that because polysemanticity hinders interpretability, it makes sense to attempt to prevent it. However, requiring the same performance (loss) from the modified architecture suggests that we’re expecting to get interpretability for free (without losing any capability). Is this a reasonable expectation? Possibly not, since the network has been trained under the optimization pressure of an entire TPU pod to maximiz...
Covid 8/4/22: Rebound
by Zvi — 2022-08-04T11:20:01.482Z

I’ll be at Future Forum over the weekend. If you’re here, come and say hello. I don’t plan on scheduling a lot of meetings because I hate scheduling, but talking to people is the plan. I have a few people that I know I want to see, but mostly I’m going to be improvising, and whatever happens happens.

Because of the trip I wrote most of this earlier, and all I did Thursday was fill in the numbers section and check for anything big from overnight – some stuff likely got shifted to next week as a result.

It was probably a good week for a trip, as Covid news continues to quiet, to the point where the majority of this week’s post is about non-Covid things. So far I am mostly fine with this transforming gradually into a weekly roundup. Indeed do many things come to pass.

Executive Summary

Covid continues on its expected trajectory.
Monkeypox continues on its expected trajectory.
Nuclear reactor design approved, hope springs eternal.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 850k cases (+5%) and 2,850 deaths (-3%).

Results: 730k cases (-10%) and 3,054 deaths (+3%).

Prediction for next week: 750k cases (+3%) and 3,000 deaths (-2%).

Adjustments were the reverse of what I expected, but everything involved was relatively small. Everything points to a stable situation, with reporting errors and gaps likely to be most or all of the story.

Deaths

Cases

Included 10k presumed cases in South Carolina, since they did not report, along with standard Florida adjustment.

The drop in the West isn’t mostly due to one particular state although some (~20k) of it is Nevada being obviously too high last week. I am continuing not to make adjustments that aren’t super obvious, especially retroactively. Better not to put one’s finger on the scale even when it’s probably in the correct direction.

Physical World Modeling

President Biden took Paxlovid early, and got a rebound case of Covid for his trouble. He then continues to report feeling completely fine during the rebound. If you test positive for a Covid rebound, but wouldn’t have noticed without the test, should you care? In terms of preventing further spread you care a bit, but if nothing changes for you then nothing changes for you. The extent to which Paxlovid is being postponed to minimize rebounds reflects the degree to which what people actu...
High Reliability Orgs, and AI Companies
by Raemon — 2022-08-04T05:45:34.928Z

Epistemic Effort: Rough notes from a shallow dive. Looked into this for a few hours. I didn't find a strong takeaway but think this is probably a useful jumping-off-point for further work.

Most likely, whether I like it or not, there will someday be AGI research companies working on models that are dangerous to run. Maybe they'll risk an accidental unfriendly hard takeoff. Maybe they'll cross a threshold accelerate us into a hard-to-control multipolar smooth takeoff.[1]

There's some literature on organizations that operate in extremely complex domains, where failure is catastrophic. They're called High Reliability Organizations (HROs). The original work focused on three case studies: A nuclear power plant, an air traffic control company, and a nuclear aircraft carrier (aka nuclear power plant and air traffic control at the same time while people sometimes shoot at you and you can't use radar because you don't want to give away your position and also it's mostly crewed by 20 year olds without much training)

These were notable for a) being extremely complex systems where it would be really easy to screw up catastrophically, and b) somehow, they manage to persistently not screw up.

How do they do that? And does this offer any useful insights to AGI companies?

I started writing this post before Eliezer posted Six Dimensions of Operational Adequacy in AGI Projects. It's not pointed at the exact same problem, but I had a similar generator of "what needs to be true of AI companies, for them to safely work on dangerous tech?". (I think Eliezer had a higher bar in mind with Six Dimensions, which is like, "what is an AI company a researcher could feel actively good about joining")

I was initially pointed in the HRO direction by some conversations with Andrew Critch. Some of his thoughts are written up in the ARCHES report. (There's been some discussion on LessWrong)

My TL;DR after ~10 hours of looking into it:

The literature has at least some useful takeaways/principles.
Some hospitals (maybe?) successfully implemented HRO principles and drove some classes of accidents down to 0% (!? roll to disbelieve)
The literature (even the hospital "replication") isn't obviously super relevant because it's focused on domains that have much tighter feedback loops than AI research.
The most interesting part of the Air Traffic Control and Aircraft Carrier stuff is not t

...
Surprised by ELK report's counterexample to Debate, IDA
by Evan R. Murphy — 2022-08-04T02:12:15.139Z

Summary

I had assumed the original ELK report had fundamental objections to Debate and IDA in terms of their robustness. Re-reading the report, I was surprised to find that that the only counterexample it provides for these proposals is that they don't seem computationally competitive compared to unaligned AI.

ELK's iconic SmartVault AI toy scenario seems like a pretty difficult world to imagine this kind of competitiveness mattering. I provide humorous fictional dialogue at a yacht party to illustrate this point.

My best guess is that the ELK report authors just didn't get around to doing in-depth plot continuity editing on their SmartVault scenario, but that they care about competitiveness because if its importance in the real world. Still it would be good to clarify. And also, is this really the main objection to Debate and IDA for ELK?

The ELK report on Debate and IDA

I was re-reading the original Eliciting Latent Knowledge (ELK) report from ARC. At first it seemed to me like there was an unbacked claim that Debate and Iterated Distillation and Amplification (IDA) can't solve ELK:

Unfortunately, none of these strategies seem sufficient for solving ELK in the worst case. In particular, after considering strategies like

Debate

,

Iterated Distillation and Amplification

and

Imitative Generalization

— and even assuming that every other counterexample to those strategies could be overcome — we believe they still don’t address ELK.

As I dug into it more though, I realized I had read too quickly over the section where Paul, Ajeya and Mark explain this. It's called Strategy: have AI help humans improve our understanding.

In this section, the authors explain some of the potential of using Debate or IDA, before introducing New counterexample: gradient descent is more efficient than science, which they apparently consider to invalidate these strategies for ELK:

This means it’s plausible that an AI assistant who can automate the process of doing science well enough to impart us with all the SmartVault AI’s knowledge can only be trained using much more total computation than the original SmartVault AI itself.

As an intuition, imagine the cost of training a more-powerful future version of

AlphaFold

vs a model that can break down and explain why a particular protein folds in a particular way; it’s very plausible that the latter would be sig

...
Clapping Lower
by jefftk — 2022-08-04T02:10:04.509Z

Clapping is normally relatively high pitch:

(mp3)

I find clapping annoyingly high, so I usually clap a bit differently:

(mp3)

If you also want to clap lower, the main idea is to maximize the
volume of air you trap between your clapping hands. Arch the palms,
and try to have the full circumference of your palms collide. It
takes some practice, but after a bit it should come pretty naturally.

This is also useful musically, if you're clapping along and want to
clap low for some beats (ex: 1 and 3) and high for others (ex: 2 and
4).
How do I know if my first post should be a post, or a question?
by Nathan1123 — 2022-08-04T01:46:15.312Z

Hello!

My name is Nathan, which you could divine from my username but I figured it would start off on the right foot to greet using conventional social norms.

I consider myself a novice in decision theory philosophy and ethical AI, so I am intimidated by the high standard of quality writing and research in these posts. It feels more like people are submitting to a scientific periodical than an online forum. While I may never thoroughly read through Eliezer's books, I have been trying to browse the top posts of the site in order to understand the basic fundamentals of the community.

As for me personally, I have been working a few years as a Software Engineer, with a masters degree in Computer Science. In the last year I have also been studying a lot in AI algorithms, and recently obtained a certification in Deep Reinforcement Learning. I also have a certain love of philosophy, although most books I have read are pre-modern philosophy of the Neo-Platonic or Scholastic variety.

I stumbled across this site just in the last few days, after following a rabbit hole of sensationalist news articles surrounding the hysteria on UAI and the Basilisk. But after seeing how things work around here, I am more fascinated by the concepts of ethical AI and the need to model human moral values mathematically. (From what I could tell, it seems that the Basilisk has run its course a while ago and is no longer talked about much, anyway). Personally, I hold a lot of doubt that a sentient, superintelligent AI will ever be physically possible, but regardless I still understand how the discussions of ethical AI are extremely relevant.

For a while, I have had a lot of thought about this topic, which if I were to post them would fill out at least a moderately-sized article of a few pages. That being said, it would be mainly ideas of my own personal knowledge and not a rigorous, academic research. Would that be appropriate as a post? Or would it be best to keep such personal musings out of the posting space, and instead post it here, phrased in the form of a question?

Thanks,

Nathan
Three pillars for avoiding AGI catastrophe: Technical alignment, deployment decisions, and coordination
by LintzA — 2022-08-03T23:15:22.652Z

Generated by Midjourney

Cross-posted to the EA Forum: Three pillars for avoiding AGI catastrophe

Epistemic status: This model is loosely inspired by a conversation with Nate Soares but has since warped into something perhaps significantly different from his original intent. I think this model is a useful thinking tool when it comes to examining potential interventions to mitigate AI risk and getting a grasp of the problem we face. That said, I’m sure that the model has its flaws - I just haven’t spent enough time to find them.

I’ve tried to write this post so it’s very skimmable (except the hypothetical scenarios section) - reading the first sentence or two of each heading and skipping the bullet points should get you the gist of the model.

Summary

The three pillars model attempts to describe the conditions needed to successfully avoid the deployment of unaligned AGI. It proposes that, to succeed, we need to achieve some sufficient combination of success on all three of the following:

Technical alignment research
Safety-conscious deployment decisions
Coordination between potential AI deployers.

While how difficult success is depends on the difficulty of solving any given pillar, this model points toward why we may well fail to avoid AGI catastrophe: we need to simultaneously succeed at three difficult problems.

More generally, the model aims to help longtermists flesh out our mental pictures of what success on AGI risk looks like. In particular, it suggests that a strategy aimed solely at a single pillar is unlikely to be sufficient, and our community might need to take ambitious actions in several directions at once.

This model is intended as an imperfect but hopefully useful thinking tool. Further work in this area could add nuance to the model, formalize it, and try to use it to describe different viewpoints (e.g. Christiano’s stories of AI failure) or strategies (e.g. raising awareness of AI risk, which might affect several pillars).

The three pillars

Background assumption: Deploying unaligned AGI means doom. If humanity builds and deploys unaligned AGI, it will almost certainly kill us all. We won’t be saved by being able to stop the unaligned AGI, or by it happening to converge on values that make it want to let us live, or by anything else.

We need to attain sufficient success on some combination of three pillars in order to atta...
Precursor checking for deceptive alignment
by evhub — 2022-08-03T22:56:44.626Z

This post is primarily an excerpt from “Acceptability Verification: a Research Agenda” that I think is useful enough on its own such that I’ve spun it off into its own post.

The central idea of this section in the original agenda document is to understand the necessary desiderata for doing precursor checking for deceptive alignment. The basic idea of precursor checking here is that, if you want to prevent deceptive alignment from ever arising in the first place—e.g. because you think it’ll be too difficult to detect after the fact—you need to find some condition to look for instead that rules out the possibility of deceptive alignment.

In the language of this post, I’ll refer to the precursor we’re looking for as an acceptability predicate, with the idea being that it’s some predicate that determines whether a model is “acceptable” in the sense that it excludes the possibility of deceptive alignment. Thus, the goal here is to understand what desiderata such an acceptability predicate would have to satisfy.

Acceptability desiderata

What follows is a near-verbatim excerpt from “Acceptability Verification: a Research Agenda.” Thus, the below writing is primarily from around 2020.

Given that significant scaling of transparency and interpretability is possible, if we want to do acceptability verification we still need to figure out the very important question of what we need those interpretability tools to be able to understand about our models—that is, what acceptability predicate should we be checking? To answer this question, the first thing we need to do is define what makes an acceptability predicate good—that is, what are the desiderata that we want our acceptability predicate to satisfy?

We’ll start by making some definitions. Let M be the full model space and we’ll define the following three predicates on that model space.

Let S:M→B represent whether the model actually avoids whatever problematic thing we’re trying to avoid (e.g. deception). The S here stands for “safe,” though note that it’s a very weak form of safety. The idea here is that we should get actual safety—that is, the model is not just non-catastrophic but also trying to do what we want—by adding in good (behavioral) performance on the original loss function. In other words, acceptability verification is just trying to solve the problem of worst-case guarantees—but you still need standard outer

...
Transformer language models are doing something more general
by Numendil — 2022-08-03T21:13:42.472Z

Epistemic status: speculative.

First, a few paper titles:

Pretrained Transformers as Universal Computation Engines
Can Wikipedia Help Offline Reinforcement Learning? (Answer: yes.)
Pretrained Transformers Improve Out-of-Distribution Robustness
Data Distributional Properties Drive Emergent In-Context Learning in Transformers

The gist of the first three studies is that transformers (specifically) trained on natural language (specifically) generalize better than expected, with little or no fine-tuning, not only to unseen tasks but even to unseen and apparently unrelated modalities like offline reinforcement learning. The last study takes this a step further—it doesn't actually pretrain on language at all, but instead tries to mimic the specific statistical properties of natural language that lead to this behavior with various sampling procedures from an image classification dataset.

The difference between these results and the plethora of text-to-text transformer multitask/transfer learning results that have come out since GPT-1 is that transfer learning to new modalities requires learning priors general enough to apply to both text and the other modality—implying, first of all, that such priors exist, which has updated me in the following directions:

Well-trained transformers, regardless of task, occupy the same relatively small subspace of parameter space
Most of the gradient-descent steps of a training run from scratch are spent just getting to this subspace; relatively few are spent learning the specific task

Taken together, these hypotheses seem to imply that within today's gigantic and notoriously data-hungry language models is a sparser, far more efficient architecture trying to get out. I don't have any idea what this architecture looks like. If I did, I wouldn't post about it here. I am quite confident that it exists, because human children manage to acquire language without ingesting the equivalent of terabytes of text. I'm even reasonably confident that it's simple, because the human genome doesn't have enough space to code for complex mental priors (also, the evidence seems to point to the neocortex being fairly uniform), and because whatever “universal grammar” pretrained transformers are learning, it has to be fundamental enough to apply to domains as unlike language as offline reinforcement learning.

Only the last paper of the four I linked above, from Deep...
Some doubts about Non Superintelligent AIs
by aditya malik — 2022-08-03T19:55:45.454Z

Hello. I'm new to less wrong and would appreciate some help. I've been trying to understand basilisk since the more you understand, the less worried you are. While acausal trade requires clear understanding of the other agent, thus ruling out trading with a Superintelligent AI, I've been trying to find a answer to non Superintelligent AI's with which you 'might' be able to acausally trade with. I've arrived at 2:

There are simply too many potential non Superintelligent AIs to care about a particular one. Also following one could just result is another being pissed at you. (Many Gods Refutation)

If I can imagine the AI's decision making, then it's not smart enough to create a Utopia or make my simulations to torture anyway. It won't even try an acausal trade then.

Are my refutations valid?
Any replies would be greatly appreciated.

Edit: is there a particular reason for the downvotes? I really do need help.
Edit 2: spelling