trentmkelly/LessWrong-43k huggingface dataset:
Private notes on LW?
by Raemon — 2023-08-04T17:35:37.917Z

Lately I've been noticing what a powerup is to read things in google docs, where I can take whatever notes I want as in-line comments without worrying about looking dumb or confusing. In changes my relationship to confusing passages, where I feel much more affordance to think through what exactly is confusing about it.

As a general reading-habit, "copy it into google docs" is a pretty good habit. But I (and I think others on LW team although for slightly different reasons) have been thinking about building a feature directly into LW to facilitate it.

One version of it might explicitly be "private notes" that are optimized as such.

Another version of it might basically just take the side-comment button we already have and add a "private comments" option that lets you set the comment to "everyone", "only you", "you + author" (for giving the author feedback in a way that's more private than a comment but having more context included than a DM) [edit: also, sharing the comment with arbitrary people is a fairly obvious feature here]

Curious what people think about this and what options they'd expect themselves to use.

I'm maybe specifically wondering whether people expect a UI that's oriented around "arbitrary sharing" would feel good enough as a personal note-taking thing.
When training AI, we should escalate the frequency of capability tests
by Hauke Hillebrandt — 2023-08-04T16:07:33.776Z

Inspired by ideas from Lucius Bushnaq, David Manheim, Gavin Leech, Davidad. Any errors are mine.

Cross-posted from my

Substack

.

Compute, algorithms, and data form the AI triad—the main inputs for better AI.[1] AI uses compute to run algorithms that learn from data. Every year, we roughly double training compute and data. While this improves AI, better algorithms, half both compute and data needs every 9 months and 2 years, respectively,[2] and also increases the effective compute of all existing chips, while improving chip design only improves new chips.[3] And so, we often get much more effective compute from better algorithms, software, RLHF, fine-tuning, or functionality (cf DeepLearning, transformers, etc.). Put simply, better AI algorithms and software are key to AI progress.[4]

So far, training the AI models like GPT-4 costs ~$10M-$100M. Nvidia will sell 1M+ new H100 chips a year for ~$30k each,[5] which are ~10x faster for training and 30x for inference than their last A100 chip. Per year, these chips could train GPT-4 over 100k times. (it was trained on the old chips for ~$63M, but just 1k H100s could train it in a year for ~$20M).[6] Inflection is setting up a cluster of 22k H100s.[7] By 2024, we will run out of public high-quality text data to improve AI,[8] whereas Big Tech can train AI on non-public data and is also increasingly guarding the IP of their algorithms. And so soon, using much more compute, Big Tech might develop better AI with (dangerous) capabilities (e.g. help users create bioweapons[9]). This might not be economic: OpenAI's CEO described the compute costs for running ChatGPT as 'eye-watering: ~$.1 / 1k words.[10] We can read ~100k words per day, it might cost ~$100M daily to serve ChatGPT's 100M users (cf. a Google Search costs ~10x less, and at trillions of searches per year, using GPT-4 level AI for search might cost many billions[11]).

AI experts almost unanimously agree that AGI labs should pause the development process if we detect sufficiently dangerous capabilities.[12] Consider the Unexpected Capabilities Problem. We can't reliably predict new AI models (dangerous) capabilities, which are hard to fully understand without intensive testing (often, capabilities emerge suddenly, sometimes d...
Manifund: What we're funding (weeks 2-4)
by Austin Chen — 2023-08-04T16:00:33.227Z
Link: https://manifund.substack.com/p/what-were-funding-weeks-2-4

Overall reflections

Very happy with the volume and quality of grants we’ve been making

$600k+ newly committed across 12 projects
Regrantors have been initiating grants and coordinating on large projects
Independent donors have committed $35k+ of their own money!
We plan to start fundraising soon, based on this pace of distribution

Happy to be coordinating with funders at LTFF, Lightspeed, Nonlinear and OpenPhil

We now have a common Slack channel to share knowledge and plans
Currently floating the idea of setting up a common app between us…

Happy with our experimentation! Some things we’ve been trying:

Equity investments, loans, dominant assurance contracts and retroactive funding
Grantathon, office hours, feedback on Discord & site comments

Less happy with our operations (wrt feedback and response times to applicants)

Taking longer to support to individual grantees, or start new Manifund initiatives

Please ping us if it’s been a week and you haven’t heard anything!

Wise deactivated our account, making international payments more difficult/expensive…
In cases where multiple regrantors may fund a project, we’ve observed a bit of “funding chicken”

Grant of the month

[$310k]

Apollo Research

This is our largest grant to date! Many of our regrantors were independently excited about Apollo; in the end, we coordinated between Tristan Hume, Evan Hubinger and Marcus Abramovitch to fund this.

From Tristan:

I'm very excited about Apollo based on a combination of the track record of it's founding employees and the research agenda they've articulated.

Marius and Lee have published work that's significantly contributed to

Anthropic's work on dictionary learning

. I've also met both Marius and Lee and have confidence in them to do a good job with Apollo.

Additionally, I'm very much a fan of alignment and dangerous capability evals as an area of research and think there's lots of room for more people to work on them.

In terms of cost-effectiveness I like these research areas because they're ones I think are very tractable to approach from outside a major lab in a helpful way, while not taking large amounts of compute. I also think Apollo existing in London will allow them to hire underutilized talent that would have trouble getting a U.S. visa.

New grants

[$112k]

Jesse Hoogland: Scoping Developmental Interpretability

Jesse posted this through our open call:

We propose a 6-month research project to a

...
[Linkpost] Multimodal Neurons in Pretrained Text-Only Transformers
by Bogdan Ionut Cirstea — 2023-08-04T15:29:16.957Z

This is a linkpost for https://arxiv.org/abs/2308.01544.

Language models demonstrate remarkable capacity to generalize representations learned in one modality to downstream tasks in other modalities. Can we trace this ability to individual neurons? We study the case where a frozen text transformer is augmented with vision using a self-supervised visual encoder and a single linear projection learned on an image-to-text task. Outputs of the projection layer are not immediately decodable into language describing image content; instead, we find that translation between modalities occurs deeper within the transformer. We introduce a procedure for identifying "multimodal neurons" that convert visual representations into corresponding text, and decoding the concepts they inject into the model's residual stream. In a series of experiments, we show that multimodal neurons operate on specific visual concepts across inputs, and have a systematic causal effect on image captioning.
Apollo Research is hiring evals and interpretability engineers & scientists
by Marius Hobbhahn — 2023-08-04T10:54:09.276Z

TL;DR: You can apply

here

.

About Apollo

Apollo Research

is a London-based AI safety organization. We focus on high-risk failure modes, particularly deceptive alignment, and intend to audit frontier AI models. Our primary objective is to minimize catastrophic risks associated with advanced AI systems that may exhibit deceptive behavior, where misaligned models appear aligned in order to pursue their own objectives. Our approach involves conducting fundamental research on interpretability and behavioral model evaluations, which we then use to audit real-world models. Ultimately, our goal is to leverage interpretability tools for model evaluations, as we believe that examining model internals in combination with behavioral evaluations offers stronger safety assurances compared to behavioral evaluations alone.

Culture: At Apollo, we aim for a culture that emphasizes truth-seeking; being goal-oriented; giving and receiving constructive feedback; and being friendly and helpful. If you’re interested in more details about what it’s like working at Apollo, you can find more information

here

.

Context & Agenda

We feel like we have converged on a good technical agenda for evals and interpretability. Therefore, we are currently not bottlenecked by ideas and could easily scale and parallelize existing research efforts with more people. We are confident that more people would help both agendas progress faster and that we can easily integrate them into our existing teams.

On the interpretability side, we’re pursuing a new approach to mechanistic interpretability that we’re not yet publicly discussing due to potential infohazard concerns. However, we expect the day-to-day work of most scientists and engineers to be comparable to existing public interpretability projects such as

sparse coding

,

indirect object identification

,

causal scrubbing

,

toy models of superposition

, or

transformer circuits

, as well as converting research insights into robust tools that can scale to very large models.

For the interpretability team, we aim to make 1-3 offers depending on funding and fit.

To evaluate models, we employ a variety of methods. Firstly, we intend to evaluate model behavior using basic prompting techniques and agentic scaffolding, similar to AutoGPT. Secondly, we aim to fine-tune models to study their generalization capabilities and ...
Has anyone tried creating a YouTube or TikTok series covering the sequences?
by Max Rossi — 2023-08-04T00:10:40.834Z

I would love to see a playlist, that just covers, in order, each of the sequences. Ideally, I might think of each video as being a chapter, resulting in a series of 26 videos covering all 6 books.
I have read the sequences a few times, and while I am not an expert in youtube video creation, etc. I think it is a way to get a lot more reach and spread these ideas further.
Are there any guidelines in terms of fair use or adaptation of this source material?
I am thinking of giving it a shot and trying to create the first video "Predictably Wrong", but I am not sure if that would step on any toes, so I figured I would ask here first.
Is there any metric measuring ~"proportion of people creating extra value"?
by Amal  — 2023-08-03T22:54:10.028Z

Imagine a simplified model of the world, where the population suddenly increases by 1000 individuals. Of these newcomers, perhaps 50 will become policemen to ensure safety, 30 will serve as doctors to maintain health, 25 will take on roles as teachers to educate the children among them, 20 will become hairdressers to uphold appearance, 5 will work in factories to produce additional cars, and 1 might become a mayor, among other roles (these numbers are, of course, entirely arbitrary).

The common characteristics among these professions is that they primarily cater to the needs of the new 1000 people, and thus, their numbers must roughly correspond with the population growth. If the new 1000 individuals were to vanish suddenly, the absence of these workers would likely have minimal impact on the rest of the world.

This contrasts with professions such as writers, researchers, engineers, movie actors, YouTubers, or international journalists. These roles generate added value that can be consumed by an unlimited number of individuals, regardless of population size. It is worth noting that this concept is not solely tied to the complexity of work; roles that collaborate closely with these professionals, such as makeup artists working with actors or lab technicians assisting researchers, also fall into this category.

In reality, classifying people into these two distinct categories is likely an oversimplification, and the lines between them may be blurred, with many shades of grey in between. Nevertheless, I am intrigued by the possibility of determining an approximation for the proportion of work or workers that must scale with population size. Do you know a name for a metric similar to this or any estimates of what it could be and how has it evolved over time?
Hypothetical: what would you do?
by JNS — 2023-08-03T22:39:55.026Z

Let’s pretend I have a semi rigors model that lays out why RLHF is doomed to fail and also that it negatively affects model performance (including why it does so)

Let’s go further into lala land and pretend that I have an architectural plan that does much better, very transparent, steerable and corrigible, can be deployed and used without changing or retraining the base LLM.

There are some downsides like requires more compute at inference time, not provable bulletproof, likely breaks in SI regime and definitely breaks under self improvement (so very definitely NOT an alignment proposal).

Short term this looks beneficial, also looks like shortening timelines, and extremely unlikely to advance the AI safety field (in the direction of what we ultimately want and need).

What should I do, if I ever happened to be in such a situation?

Prototype it, limited access with the expressed purpose of breaking stuff (black box, absolutely no architectural information provided).

Write it up and publish.

Forget about it, smarter people must have already thought of it, and since it’s not a thing, I am clearly wrong.

Forget about it, only helps capabilities.
[Linkpost] Deception Abilities Emerged in Large Language Models
by Bogdan Ionut Cirstea — 2023-08-03T17:28:19.193Z

This is a linkpost for https://arxiv.org/abs/2307.16513.

Large language models (LLMs) are currently at the forefront of intertwining artificial intelligence (AI) systems with human communication and everyday life. Thus, aligning them with human values is of great importance. However, given the steady increase in reasoning abilities, future LLMs are under suspicion of becoming able to deceive human operators and utilizing this ability to bypass monitoring efforts. As a prerequisite to this, LLMs need to possess a conceptual understanding of deception strategies. This study reveals that such strategies emerged in state-of-the-art LLMs, such as GPT-4, but were non-existent in earlier LLMs. We conduct a series of experiments showing that state-of-the-art LLMs are able to understand and induce false beliefs in other agents, that their performance in complex deception scenarios can be amplified utilizing chain-of-thought reasoning, and that eliciting Machiavellianism in LLMs can alter their propensity to deceive. In sum, revealing hitherto unknown machine behavior in LLMs, our study contributes to the nascent field of machine psychology.

This seems to me like a very good target for (mechanistic) interpretability (though would require access to GPT-4/ChatGPT activations/weights).
Embedding Ethical Priors into AI Systems: A Bayesian Approach
by Justausername — 2023-08-03T15:31:50.087Z

Abstract

Artificial Intelligence (AI) systems have significant potential to affect the lives of individuals and societies. As these systems are being increasingly used in decision-making processes, it has become crucial to ensure that they make ethically sound judgments. This paper proposes a novel framework for embedding ethical priors into AI, inspired by the Bayesian approach to machine learning. We propose that ethical assumptions and beliefs can be incorporated as Bayesian priors, shaping the AI’s learning and reasoning process in a similar way to humans’ inborn moral intuitions. This approach, while complex, provides a promising avenue for advancing ethically aligned AI systems.

Introduction

Artificial Intelligence has permeated almost every aspect of our lives, often making decisions or recommendations that significantly impact individuals and societies. As such, the demand for ethical AI — systems that not only operate optimally but also in a manner consistent with our moral values — has never been higher. One way to address this is by incorporating ethical beliefs as Bayesian priors into the AI’s learning and reasoning process.

Bayesian Priors

Bayesian priors are a fundamental part of Bayesian statistics. They represent prior beliefs about the distribution of a random variable before any data is observed. By incorporating these priors into machine learning models, we can guide the learning process and help the model make more informed predictions.

For example, we may have a prior belief that student exam scores are normally distributed with a mean of 70 and standard deviation of 10. This belief can be encoded as a Gaussian probability distribution and integrated into a machine learning model as a Bayesian prior. As the model trains on actual exam score data, it will update its predictions based on the observed data while still being partially guided by the initial prior.

Ethical Priors in AI: A Conceptual Framework

The concept of ethical priors relates to the integration of ethical principles and assumptions into the AI’s initial learning state, much like Bayesian priors in statistics. Like humans, who have inherent moral intuitions that guide their reasoning and behavior, AI systems can be designed to have “ethical intuitions” that guide their learning and decision-making process.

For instance, we may want an AI system to have an inbuilt p...
Password-locked models: a stress case for capabilities evaluation
by Fabien Roger — 2023-08-03T14:53:12.459Z

Thanks to Buck Shlegeris, Ryan Greenblatt, Kshitij Sachan and Jenny Nitishinskaya for helpful discussions and feedback on drafts. This work was done at Redwood Research. The views expressed are my own and do not necessarily reflect the views of the organization.

Password-locked models are trained to exhibit certain capabilities only when a password is present in the query. Studying these models has two purposes:

Testing how well capability evaluations work when applied to models which “aren’t trying”
Understanding how well organizations could hide the capabilities of their models.

In this post, I make empirical claims which I believe to be true, check some of them on small language models, and describe what this implies for the control of potentially deceptive models.

Key empirical claims: when using behavioral model evaluations, password-locked models:

Look much less capable than they actually are when eliciting capabilities with prompting
Sometimes look much less capable than they actually are when eliciting capabilities with Reinforcement Learning

Experimental evidence: I exhibit one case where they don’t, and one case where they do

Can’t look much less capable than they actually when eliciting capabilities with supervised fine-tuning on high-quality demonstrations

Key takeaways, if the claims above are true:

If an alignment plan relies on some models being too weak to perform certain tasks, then model evaluations aimed at checking the weakness of models should be able to elicit the locked capabilities of password-locked models. This is because password-locked models and deceptive models may be able to underperform in similar circumstances.
API access and inference is not enough to evaluate capabilities of a model produced by AI labs trying to game regulations, even if AI labs are internally using the same API endpoint as auditors.

Key Empirical Claims

Similarities Between Password-locked Models and Deceptively Aligned Models Pretending to Be Dumb

It looks to me that password locked models have the same properties as the one described by Evan in When can we trust model evaluations? when he describes if and how deceptive models would be able to sandbag each category of behavioral evaluation (which he coins in his post):

Behavioral Non-Fine-Tuning Evaluations: By construction, password-locked models will refuse to be capable no matter the prompt, except for prompts with th...
AI #23: Fundamental Problems with RLHF
by Zvi — 2023-08-03T12:50:11.852Z

After several jam-packed weeks, things slowed down to allow everyone to focus on the potential room temperature superconductor, check Polymarket to see how likely it is we are so back and bet real money, or Manifold for chats and better graphs and easier but much smaller trading.

The main thing I would highlight this week are an excellent paper laying out many of the fundamental difficulties with RLHF, and a systematic new exploit of current LLMs that seems to reliably defeat RLHF.

I’d also note that GPT-4 fine tuning is confirmed to be coming. That should be fun.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Here’s what you’re going to do.

Language Models Don’t Offer Mundane Utility. Universal attacks on LLMs.

Fun With Image Generation. Videos might be a while.

Deepfaketown and Botpocalypse Soon. An example of doing it right.

They Took Our Jobs. What, me worry?

Get Involved. If you share more opportunities in comments I’ll include next week.

Introducing. A bill. Also an AI medical generalist.

In Other AI News. Fine tuning is coming to GPT-4. Teach LLMs arithmetic.

Quiet Speculations. Various degrees of skepticism.

China. Do not get overexcited.

The Quest for Sane Regulation. Liability and other proposed interventions.

The Week in Audio. I go back to The Cognitive Revolution.

Rhetorical Innovation. Bill Burr is concerned and might go off on a rant.

No One Would Be So Stupid As To. Robotics and AI souls.

Aligning a Smarter Than Human Intelligence is Difficult. RLHF deep dive.

Other People Are Not As Worried About AI Killing Everyone. It’ll be fine.

The Wit and Wisdom of Sam Altman. Don’t sleep on this.

The Lighter Side. Pivot!

Language Models Offer Mundane Utility

Make a ransom call, no jailbreak needed. Follows the traditional phone-calls-you-make-are-your-problem-sir legal principle. This has now been (at least narrowly, for this particular application) fixed or broken, depending on your perspective.

See the FAQ:

Back in AI#3 we were first introduced to keeper.ai, the site that claims to use AI to hook you up with a perfect match that meets all criteria for both parties so you can get married and start a family, where if you sign up for the Legacy plan they only gets paid when you tie the knot. They claim 1 in 3 dates from Keeper lead t...