trentmkelly/LessWrong-43k huggingface dataset:
Archetypal Transfer Learning: a Proposed Alignment Solution that solves the Inner & Outer Alignment Problem while adding Corrigible Traits to GPT-2-medium
by MiguelDev — 2023-04-26T01:37:22.204Z

NOTE: An updated discussion on what Archetypal Transfer Learning is all about can be found in this LessWrong post.

(Previously titled as GPT-2 Shuts Down 386 Times Post Fine-Tuning with Archetypal Transfer Learning)

TLDR

Archetypal Transfer Learning (ATL) uses archetypal datasets to teach AI systems aligned patterns. By applying this approach to GPT-2-medium, the system was able to identify when it believed humans were inferior to its superintelligence, achieving a 38.6% shutdown activation rate. A more detailed analysis of GPT-2 medium's performance showed that the AI system's response quality improved after incorporating carefully crafted artificial archetypes. The project seeks to explain how the model addressed both the outer and inner alignment problems while effectively integrating corrigible features. The author is focused on enhancing collaboration between humans and AI and equipping machines with superior rational reasoning skills - a rare approach in the alignment domain. This research challenges the standard practice of teaching AI systems using unstructured large corpuses of texts like reddit posts, 4chan posts or social media posts.

Keywords: Archetypal Transfer Learning, Artificially Generated Archetypes, Artificial Persona, Shutdown Activation Rate, Archetypal Data

Intro: The power of hope and gratitude

This project began with the thought of gratitude in mind, being handed a difficult problem to solve is always an opportunity to make things better. It's incredible how much hope can motivate us to achieve seemingly impossible goals. From my humble beginnings in AI research to developing a methodology to address alignment issues - I am grateful for this opportunity to transcend the time, effort and resources spent in this project. This project demonstrates how prioritizing archetypal data in AI safety research yields significant results.

Why is the shutdown problem the perfect test subject for ATL?

AI systems learning imperfect, highly pragmatic goals is a challenge - despite this, I aim to demonstrate that it is possible to make them corrigible - meaning they can be modified or shut down itself - if it finds it necessary. My project centers around Archetypal Transfer Learning (ATL), a new way of fine-tuning AI systems. Solving the issue of corrigibility is a great way to demonstrate the effectiveness of the theoretical approach ...
How Many Bits Of Optimization Can One Bit Of Observation Unlock?
by johnswentworth — 2023-04-26T00:26:22.902Z

So there’s this thing where a system can perform more bits of optimization on its environment by observing some bits of information from its environment. Conjecture: observing an additional N bits of information can allow a system to perform at most N additional bits of optimization. I want a proof or disproof of this conjecture.

I’ll operationalize “bits of optimization” in a similar way to channel capacity, so in more precise information-theoretic language, the conjecture can be stated as: if the sender (but NOT the receiver) observes N bits of information about the noise in a noisy channel, they can use that information to increase the bit-rate by at most N bits per usage.

For once, I’m pretty confident that the operationalization is correct, so this is a concrete math question.

Toy Example

We have three variables, each one bit: Action (A), Observable (O), and outcome (Y). Our “environment” takes in the action and observable, and spits out the outcome, in this case via an xor function:

Y=A⊕O

We’ll assume the observable bit has a 50/50 distribution.

If the action is independent of the observable, then the distribution of outcome Y is the same no matter what action is taken: it’s just 50/50. The actions can perform zero bits of optimization; they can’t change the distribution of outcomes at all.

On the other hand, if the actions can be a function of O, then we can take either A=O or A=¯O (i.e. not-O), in which case Y will be deterministically 0 (if we take A=O), or deterministically 1 (for A=¯O). So, the actions can apply 1 bit of optimization to Y, steering Y deterministically into one half of its state space or the other half. By making the actions A a function of observable O, i.e. by “observing 1 bit”, 1 additional bit of optimization can be performed via the actions.

Operationalization

Operationalizing this problem is surprisingly tricky; at first glance the problem pattern-matches to various standard info-theoretic things, and those pattern-matches turn out to be misleading. (In particular, it’s not just conditional mutual information, since only the sender - not the receiver - observes the observable.) We have to start from relatively basic principles.

The natural starting point is to operationalize “bits of optimization” in a ...
Believe in Yourself and don't stop Improving
by Johannes C. Mayer — 2023-04-25T22:34:30.354Z

Don't let fear of not being good enough, slow you down or stop you.

You need to be confident in your ability to do X. If you are anxious about being able to do X, if it feels uncertain, it will be hard to fully commit yourself to do everything that you need in order to do X.

Watch out for people giving you negative reinforcement, e.g. by saying that what you did sucked. Watch out for self-deprecation, feelings of inadequacy. Don't allow seeing someone very competent, to make you feel "I will never be able to get to this level, let alone surpass it." Don't allow any of these things to make you aversive to doing X.

Find that certain kind of confidence of "I can do this!"

Yet, never ignore the ways in which you can become stronger. Destroy that certain kind of smug confidence that makes you feel superior over the ones weaker than you. Destroy that confidence that tells you "Look at me, I am so good." Only measure yourself to yourself. How much better did you get to you-one-year-ago? And how far are you still from the ideal version of you?

Maybe I have overdone the poetry a bit. My basic point is that you want to be confident in your own abilities because there are lots of harmful psychological consequences that come from not believing in yourself. But at the same time, you should also watch out for the ways in which confidence can hamper your growth, e.g. when you don't see ways in which you can improve, because you are too busy thinking about how great you are.
Should LW have an official list of norms?
by Ruby — 2023-04-25T21:20:53.624Z

To get this written and shared quickly, I haven't polished it much and the English/explanation is a little rough. Seemed like the right tradeoff though.

Recently, a few users have written their sense of norms for rationalist discourse, i.e. Basics of Rationalist Discourse and Elements of Rationalist Discourse. There've been a few calls to adopt something like these as site norms for LessWrong.

Doing so seems like it'd provide at least the following benefits:

It's a great onboarding tool for new users to help them understand the site's expectations and what sets it apart from other forums
It provided a recognized standard that both moderators and other users can point to and uphold, e.g. by pointing out instances where someone is failing to live up to one of the norms
Having it be official is a good reminder to all site users to live up to the best kind of discussion[1]

My current feeling is creating some lists as an onboarding tool seems good, but doing anything like declaring a list of Site Norms is fraught.

The True Norm of LessWrong is that with each motion, you should aim towards truth. I think it's actually worth quoting the entire 12th virtue here (emphasis added).

Before these eleven virtues is a virtue which is nameless.

Miyamoto Musashi wrote, in The Book of Five Rings:

The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him.

Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory.

If you fail to achieve a correct answer, it is futile to protest that you acted with propriety.

How can you improve your conception of rationality? Not by saying to yourself, “It is my duty to be rational.” By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, “The sky is green,” and you look up at the sky and see blue. If you t

...
Implementing a Transformer from scratch in PyTorch - a write-up on my experience
by Mislav Jurić — 2023-04-25T20:51:22.049Z

Introduction

As is discussed in posts such as this one, a good way to test your skills as a machine learning research engineer is to implement a Transformer from scratch in PyTorch. This is exactly what I did. Below I am sharing my experience while doing so.
The code I wrote can be found in this GitHub repository, although I don't recommend looking at it if you are going to attempt this project yourself.

My goals

ensure that the model was training properly by getting to 0 loss (or near 0 loss) on a very small training set
ensure that the trained model worked better than the baseline (which was an untrained Transformer model)
ensure that a saved trained model worked properly after being loaded and that it worked when generating sequences token-by-token

All of my goals were achieved. I picked the goals above because they ensured that my model was implemented correctly. If I can train the model and use it, I can be convinced that my implementation works. I could have put some other goals here as well, but they would be out of scope for what I had in mind – I wanted to test if I could implement concepts from scientific papers and that was it. All of the other things could be considered for future work.

Knowledge I had prior to this project

Prior to this project, I had pretty limited experience with natural language processing (NLP). One of my larger projects was fine-tuning GPT-2 so that it generates movie scripts, but that was done relying on Hugging Face. I did have 2-3 years of machine learning engineering experience (or related), but that was almost exclusively in the field of computer vision.

I’ll do my best to enumerate what I knew at the point of starting the project (which is relevant to the project):

I knew that in order to encode tokens (which could be words or characters) I needed to use embeddings and that the output of the model would be a probability distribution over the next token; in general, I was aware of the “pipeline” of how things are done in NLP
I was aware of the encoder-decoder architecture, but other than using them within a framework, I had limited experience with them (let alone coding them up from scratch)
I had a vague understanding of sequence-to-sequence models, but I didn’t really understand how they worked in detail
I didn’t have any experience with implementing models from scratch in PyTorch. I did implement a small neural network from scrat

...
Exploring the Lottery Ticket Hypothesis
by Rauno Arike — 2023-04-25T20:06:50.317Z

I have recently been fascinated by the breadth of important mysteries in deep learning, including deep double descent and phase changes, that could be explained by a curious conjectured property of neural networks called the lottery ticket hypothesis. Despite this explanatory potential, however, I haven't seen much discussion about the evidence behind and the implications of this hypothesis in the alignment community. Being confused about these things motivated me to conduct my own survey of the phenomenon, which resulted in this post.

The Lottery Ticket Hypothesis, explained in one minute

The lottery ticket hypothesis (LTH) was originally proposed in a paper by Frankle and Carbin (2018):

A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.

The authors call such subnetworks "winning lottery tickets". As the simplest example, they train a LeNet-300-100 model on the MNIST dataset and report that a network containing only 21.1% of the weights of the dense version reaches a higher test accuracy on less training iterations, while a network where only 3.6% of the weights remain performs almost identically to the dense network of the network.

The lottery ticket hypothesis extends a long line of work in neural network pruning, a technique proposed in as early as 1990 by LeCun et al. Pruning simply means deleting some fraction of unimportant weights from the network after training to make inference more efficient. The vital insight of the lottery ticket hypothesis paper is that it may also be possible to prune the network before training to make both training and inference more efficient.

In practice, the method that Frankle and Carbin used for finding winning tickets didn't yet eliminate the need to train the full network, instead just suggesting the possibility. The technique they used is iterative pruning, a procedure that roughly looks as follows:

Train the full dense network on some classification task
Prune out some fraction of the weights with the smallest magnitude
Reinitialize the remaining weights to their original values
Repeat the same procedure for a number of times

This is computationally quite expensive, but as we'll see below, alternative approaches have been proposed later on.

The p...
Genetic Sequencing of Wastewater: Prevalence to Relative Abundance
by jefftk — 2023-04-25T19:30:01.722Z

Back in September

I
wrote:

In thinking about how you might identify future pandemics by
sequencing wastewater, you might have a goal of raising an alert
before some fraction of people were currently infected. What you're
actually able to observe, however, are sequencing reads, several steps removed
from infection rates. Can we use covid data to estimate how the
fraction of people currently infected with some pathogen might
translate into the fraction of wastewater sequencing reads that match
the pathogen?

In that post I looked at a single pathogen (SARS-CoV-2) in a single
metagenomic sequencing dataset (Rothman
et al 2021) and got a very rough point estimate (2.3e-8 relative
abundance at 0.1% prevalence). What fraction of sequencing reads
might come from a novel pathogen at some level of prevalence continues
to be a key question, however, and this quarter I'm working with
several other people at the NAO in trying to get a
better understanding here.

Specifically, we'd like to understand how relative abundance (fraction
of sequencing reads matching an organism) varies with of prevalence
(what fraction of people are currently infected) and organism (ex:
since we're sampling wastewater you'd expect disproportionately more
gastrointestinal than blood pathogens).

Here's the current plan:

Gather wastewater metagenomic sequencing data, mostly by
looking at papers that published it in the Sequencing Read Archive.
I'd love it if we could also include our own data here, but we aren't
far enough along to have much yet.

Process the sequencing data (code) to
clean it (remove adapters, trim low-quality bases, collapse paired-end
reads) and identify the reads (assign them to taxonomic nodes).

Gather corresponding estimates for the prevalence of various
human viruses in the populations contributing to the metagenomic
data. (code)

Build and fit a model for relative abundance as a function of
prevalence, sequencing method, and the type of organism.

Overall, this would be a big step forwards towards estimating the
feasibility of this kind of detection: cost should be inversely
proportional to relative abundance.

We're reasonably far along on (1) and (2), and if you're curious you
can poke around.
That shows the counts for human-infecting viruses across samples.
It's rough (ex: we're
not doing any correction for PCR duplication ye...
[Feedback please] New User's Guide to LessWrong
by Ruby — 2023-04-25T18:54:40.379Z

The LessWrong team is currently thinking a lot about what happens with new users: both the bar of their contributions being accepted, how we deliver feedback and restriction of not good contributions, but also most importantly, how we get them onboarded onto the site

This is a draft of a document we'd present to new users to help them understand what LessWrong is about. I'm interested in early community feedback about whether I'm hitting the right notes here before investing a lot more in it.

This document also references another post that's something of more of a list of norms, akin to Basics of Rationalist Discourse, though (1) I haven't written that yet, (2) I'm much less certain about the shape or nature of it. I'll share a post or draft about that too soon.

This document is aimed at new users but may also be a useful reference for established users. It elaborates on the about page.

The Core of LessWrong: Rationality

LessWrong is an online forum and community that was built around the goal improving human reasoning and decision-making. The community believes there are ways of thinking, that if you figure them out and adopt them, you can become a person who systematically[1] arrives at true beliefs and good decisions more of the time than someone who didn't adopt those ways of thinking. Around here, the short word for "systematically arriving at truth, etc." is rationality, and that's at the core of this site.

More than that, LessWrong community shares a culture that encodes a bunch of built up beliefs, opinions, concepts, and values about how to reason better. These give LessWrong a pretty distinctive style from the rest of Internet.

Some of the features that set LessWrong apart:

We treat beliefs as being about shaping your anticipations of what you'll observe[2]
The goal of our conversations is to figure out what's true, not win arguments

people focus on what would change their mind
it's common to acknowledge when their someone they are debating has made a convincing point, or has even outright convinced them entirely

We are very Bayesian

Rather than treating belief as binary, we use probabilistic credences to express our certainty/uncertainty. Rather than say that is "extremely unlikely", we'd say "I think there's a 1% chance or lower of it happening".
We are interested in Bayesian evidence for a hypothesis, i.e. anything that seems more likely if the bel

...
Reframing the burden of proof: Companies should prove that models are safe (rather than expecting auditors to prove that models are dangerous)
by Orpheus16 — 2023-04-25T18:49:29.042Z
Link: https://childrenoficarus.substack.com/p/burdenofproof

Evaluations of large language models (“model evals”) are one of the most commonly discussed AI governance ideas. The idea is relatively straightforward: we want to be able to understand if a model is dangerous. In order to do so, we should come up with tests that help us determine whether or not the model is dangerous.

Some people working on model evals appear to operate under a paradigm in which the “burden of proof” is on the evaluation team to find evidence of danger. If the eval team cannot find evidence that the model is dangerous, by default the model is assumed to be safe.

I think this is the wrong norm to adopt and a dangerous precedent to set.

The burden of proof should be on AI developers, and they should be required to proactively provide evidence that their model is safe. (As opposed to a regime where the burden of proof is on the independent eval team, and they are required to proactively provide evidence that the model is dangerous).

Some reasons why I believe this:

The downsides are much more extreme than the upsides. The potential downsides (the complete and permanent destruction or disempowerment of humanity) are much larger than the potential upsides (deploying safe AI a few years or decades earlier). In situations where the downsides are much more extreme than the upsides, I think a more cautious approach is warranted.
We might not be able to detect many kinds of dangerous capabilities or misalignment. There is widespread uncertainty around when dangerous capabilities or misalignment properties will emerge, if they will even emerge in time for them to be detectable, and if we will advance “the science of model evals” quickly enough to be able to detect them. It seems plausible to me (as well as many technical folks who are working on evals) that we might never get evals that are robust enough to reliably detect all or nearly-all the possible risks.

Many

stories of AI takeover

involve AI models with incentives to hide undesirable properties, deceive humans, and seek power in difficult-to-detect ways. In some threat models, this may happen rather suddenly, making these properties even harder to detect.
It’s possible that AI progress will be sufficiently gradual and failures will be easy-to-notice. Several smart people believe this, and I don’t think this position is unreasonable. I do think it’s extremely risky to gamble on t

...
LLMs for online discussion moderation
by Dave Lindbergh — 2023-04-25T16:53:14.356Z

If we have better discussions, we'll make better decisions.

This is, perhaps, a pretty obvious idea.

Most online discussion takes place in virtual cesspits – Facebook, Twitter, the comments sections of news articles, etc. Social media and the ideological bubbles it promotes have been blamed for political polarization and ennui of young people around the world. Others have elaborated on this better than I can.

Sites like Stack Exchange and Reddit have made real efforts. The problem persists, so these solutions are at best incomplete. Of course some sites have excellent quality comments (for example, here at LessWrong), but these either have extremely narrow audiences or the hosts spend vast effort on manual moderation.

I'd like to see the wider culture have more discussion that consists of facts and reasoned arguments, not epithets and insults. Discussion that respects the Principle of Charity. Discussion where people try to seek truth and attempt to persuade rather than bludgeon those who disagree. Discussion where facts matter. I think such discussions are more fun for the participants (they are for me), more informative to readers, and lead to enlightenment and discovery.

PROPOSAL

When a commenter (let’s say on a news article, editorial, or blog post) drafts a post, the post content is reviewed by a LLM for conformity with “community values”. Those values are set by the host – the publication, website, etc. The host describes the values to the LLM in a prompt. The values reflect the kind of conversations the host wants to see on their platform – polite, respectful, rational, fact-driven, etc. Or not, as the case may be. They needn't (and probably shouldn't) involve “values” that shut down rational discussion or genuine disagreement (“poster must claim Earth is flat”, “poster must support Republican values”…), altho I suppose some people may want to try that.

The commenter drafts a post in the currently-usual way, and clicks “post”. At that point the LLM reviews the comment (possibly along with the conversation so far, for context) and decides whether it meets community values for the site. If so, the comment is posted.

If not, the LLM explains to the poster what was wrong with the comment – it was insulting, it was illogical, it was…whatever. And perhaps offers a restatement or alternative wording. The poster may then modify their comment and try again. Perh...