trentmkelly/LessWrong-43k huggingface dataset:
A push towards interactive transformer decoding
by R0bk — 2023-05-31T17:56:58.680Z
Link: https://github.com/R0bk/Transpector

In Brief:

I've been developing an interactive tool that I believe is helpful in accelerating transformer mechanistic analysis and that has the potential to reduce the barrier of entry.

Motivations

For a while now, my focus has been shifting towards alignment research, but getting involved and building intuition in this field has been challenging. I believe this is somewhat of a common view, and perhaps gives reasoning towards the number of posts discussing differing intuitions and getting started.

To this end, I'd like to share a transformer mechanistic analysis tool I've developed that has been helpful in my own personal intuition construction and has enabled me to build up others intuition quickly as well. It is currently focused on activation layer visualisation, ablation and freezing but there is work regarding patching, gradient and weight based interactions I plan on expanding to as-well.

Furthermore, there has also been a secondary goal of the project. To "industrialise" the process of finding, analysing and explaining circuits and algorithms within transformers. Several groups have shown their ability to decode these out of models, but all with significant effort (from my understanding). It is my belief that focusing on the toolset used to achieve these goals; standardising it, and making it quick and easy to use is key in expanding our ability to do this at a broader scale and potentially even automating it.

Design Philosophy

While translating these goals into what the tool is today I've had to make some key considerations. Primarily, how can we give a more human intelligible view of models that are large in terms of layer count and dimensionality? And how can we enable interacting, adjusting and observing how the model is affected live through these views?

To this end, the visual components of the tool can be broken down into two categories, the high level architecture (layers, residual stream...) and the individual components (attn heads, logits...). Displaying model architecture within the tool has taken a similar approach to Anthropic's figures for transformer circuits, providing an interactive map with attention and MLP layers adding to the residual backbone. I believe this is one of the most intuitive visualisations to represent transformer architecture, and is somewhat interpretable even to those who aren't in the field.

For in...
Neuroevolution, Social Intelligence, and Logic
by vinnik.dmitry07 — 2023-05-31T17:54:24.741Z

Do Transformers Reason about Electric Sheep?

Do not underestimate evolution

Evolution for 3.5–4.5 billion years has developed unconditioned reflexes and instincts in terrestrial carbon life sufficient for survival and distribution. Cumulatively, these and other types of behavior are called genetic memory. Homo sapiens, in this respect, is no different from other forms of life. However, something differentiates them favorably in the ability to distinguish, evaluate, and understand what is true and false, valuable and useless, moral and immoral. Instead of two extremes, there may be an interval, but invariably one thing, this ability to distinguish is called intelligence (a noun from the Latin verb interlego = inter — between + lego — (se)lect = to distinguish, sort out — later acquired the meaning of to understand — understanding).

At what the moment did the critical leap from the slowly adapting collective unconscious occur? When did the rapid growth of intelligence, not associated with genetic memory, begin?

The Dual Inheritance Theory and the ideas of Memetics suggest that the growth of intelligence was accompanied by an increase in the amount of knowledge accumulated in the population, what would not have been possible without effective mechanisms for assimilation, transfer, and, what is more important, the derivation of new knowledge. Our species overtook evolution with the help of collective knowledge — culture. And it all began when one of the organisms projected its knowledge to another.

Another type of the dual development

Assimilation

Assimilation of knowledge comes from the physical world, a complex system by its nature. The primary knowledge about the environment of its existence, the protohuman, like other animals, assimilated in the form of conditioned (learned) reflexes. The variety of situations gives rise to a variety of knowledge, for example, about falling objects under the influence of gravity, about pain when in contact with fire, about the fact that the discovered fruit is poisonous as a result of the reaction of the body. Such knowledge is absorbed through the joint work of the sensory and central nervous systems, and forms a world model. The activity that forms this model is motivated by universal reinforcement. A form of reinforcement is the intrinsic reward, which is regulated by the neurotransmission of dopamine in dopaminergic pathw...
Contrast Pairs Drive the Empirical Performance of Contrast Consistent Search (CCS)
by Scott Emmons — 2023-05-31T17:09:02.288Z

tl;dr

Contrast consistent search (CCS)[1] is a method by Burns et al. that consists of two parts:

Generate contrast pairs by adding pseudolabels to an unlabelled dataset.
Use the contrast pairs to search for a direction in representation space that satisfies logical consistency properties.

In discussions with other researchers, I've repeatedly heard (2) as the explanation for how CCS works; I've heard almost no mention of (1).

In this post, I want to emphasize that the contrast pairs drive almost all of the empirical performance in Burns et al. Once we have the contrast pairs, standard unsupervised learning methods attain comparable performance to the new CCS loss function.

In the paper, Burns et al. do a nice job comparing the CCS loss function to different alternatives. The simplest such alternative runs principal component analysis (PCA) on contrast pair differences, and then it uses the top principal component as a classifier. Another alternative runs linear discriminant analysis (LDA) on contrast pair differences. These alternatives attain 97% and 98% of CCS's accuracy!

"[R]epresentations of truth tend to be salient in models: ... they can often be found by taking the top principal component of a slightly modified representation space," Burns et al. write in the introduction. If I understand this statement correctly, it's saying the same thing I want to emphasize in this post: the contrast pairs are what allow Burns et al. to find representations of truth. Empirically, once we have the representations of contrast pair differences, their variance points in the direction of truth. The new logical consistency loss in CCS isn't needed for good empirical performance.

Notation

We'll follow the notation of the CCS paper.

Assume we are given a data set {x1,x2,…,xn} and a feature extractor ϕ(⋅), such as the hidden state of a pretrained language model.

First, we will construct a contrast pair for each datapoint xi. We add “label: positive” and “label: negative” to each xi. This gives contrast pairs of the form (x+i,x−i).

Now, we consider the set {x+1,x+2,…,x+n} of positive pseudo-labels and {x−1,x−2,…,x−n} of negative pseudo-labels. Because all of the x+i have "label: positive" and all of the x−i have "label: negative", we normalize the positive pseudo-labels and the negative pseudo-labels separately:...
Cosmopolitan values don't come free
by So8res — 2023-05-31T15:58:16.974Z

Short version: if the future is filled with weird artificial and/or alien minds having their own sort of fun in weird ways that I might struggle to understand with my puny meat-brain, then I'd consider that a win. When I say that I expect AI to destroy everything we value, I'm not saying that the future is only bright if humans-in-particular are doing human-specific things. I'm saying that I expect AIs to make the future bleak and desolate, and lacking in fun or wonder of any sort[1].

Here's a parable for you:

Earth-originating life makes it to the stars, and is having a lot of fun, when they meet the Ant Queen's Horde. For some reason it's mere humans (rather than transhumans, who already know my argument) that participate in the first contact.

"Hello", the earthlings say, "we're so happy to have brethren in the universe."

"We would like few things more than to murder you all, and take your resources, and lay our eggs in your corpse; but alas you are too powerful for that; shall we trade?" reply the drones in the Ant Queen's Horde.

"Ah, are you not sentient?"

"The ant queen happens to be sentient", the drone replies, and the translation machine suggests that the drones are confused at the non-sequitur.

"Then why should she want us dead?", ask the humans, who were raised on books like (rot13 of a sci fi story where it turns out that the seemingly-vicious aliens actually value sentient life) Raqre'f Tnzr, jurer gur Sbezvpf jrer abg njner gung gurl jrer xvyyvat fragvrag perngherf jura gurl xvyyrq vaqvivqhny uhznaf, naq jrer ubeevsvrq naq ertergshy jura gurl yrnearq guvf snpg.

"So that she may use your resources", the drones reply, before sending us a bill for the answer.

"But isn't it the nature of sentient life to respect all other sentient life? Won't everything sentient see that the cares and wants and desires of other sentients matter too?"

"No", the drones reply, "that's a you thing".

Here's another parable for you:

"I just don't think the AI will be monomaniacal", says one AI engineer, as they crank up the compute knob on their next-token-predictor.

"Well, aren't we monomaniacal from the perspective of a squiggle maximizer?" says another. "After all, we'll just keep turning galaxy after galaxy after galaxy into flourishing happy civilizations full of strange futuristic people having strange futuristic fun times, never saturating and deciding to spend a spare

...
Arguments Against Fossil Future?
by Sable — 2023-05-31T13:41:03.860Z

I'm currently most of the way through Alex Epstein's Fossil Future, and I find the arguments within very convincing.

In (extreme) brevity, they are:

Fossil Fuel use and the cheap energy it enables is responsible for unprecedented human flourishing for billions. Billions more need access to ever-increasing amounts of cheap energy to flourish.
Any argument against fossil fuel use must argue that its side effects (CO2 warming the planet) overwhelm the good they do by providing cheap energy. These side effects must be so bad that it's worth compromising the safety and flourishing of billions of humans to curtail their use. Such an argument must also prove that those negative side effects are beyond what humanity is capable of adapting to or overcoming, given cheap energy provided by fossil fuels.
No such argument is justifiable, given the current state of climate science.

Does anyone (preferably those who've read the book, although I don't want to restrict answers to just those people) have an opposing view/opinion, and if so why?

I'd like to do my intellectual homework on this one, and actively seek disagreement, given how convincing I've found the argument so far.
On Objective Ethics, and a bit about boats
by  — 2023-05-31T11:40:49.484Z

For a good 2,500 years or so humanity has been struggling to find quite a lot of things, but especially importantly what we should do and why. Big questions like these often seem rather irrational, and most philosophers throughout history have decided to approach this issue using mostly a form of irrational inquiry. I am a transhumanist, and I hold quite strong views on human ethics, but the one issue that I've encountered is if one says "why" enough times to a set of ethical assertions they somewhat fall to pieces. Using a form of theory reductionism, we can take a given ethical theory like virtue ethics to be blatantly false. The assertion that the ultimate good is a collection of a series of attributes such as courage, kindness, knowledge etc. is not backed up by anything. All I have to do is ask why that is and then immediately it is made clear that there is no logic underlying that claim, and the same goes for practically every established ethical theory.

And now, we come on to metaethics. Why should I even do something that's "correct", anyway? If anyone in the comments can prove to me that suffering, or premature death, can be proven to be evil using any form of logical reasoning I'll be astounded, because you can't. Using no system of a priori reasoning can I deduce that suffering or death is inherently bad. Even in the very unlikely instance that there is some form of deity, why is that deity's ethics good? Just because I see the Ten Commandments were invented by some Abrahamic God and I'll suffer some very bad consequences from not adhering to them, what is there about those commandments that makes them inherently correct? Any form of ethical system, it seems, is now doomed to fail, so why can't I just go about killing people? This is a very startling conclusion to hit upon.

However, it is not one that I quite agree with. That line of reasoning leads to some pretty dire consequences, and it commits a fundamental flaw in not making a distinction between Goal reasoning and Attainment reasoning. The line of inquiry that I pursued in the previous paragraph was an analysis of how rational thought cannot lead us to finding a goal. Neither a priori or empirical reasoning will help us find an ethical code written in the stars, and even if we found something, why would we follow it? What rationality can do, and what it is very good at doing, is he...
Against Conflating Expertise: Distinguishing AI Development from AI Implication Analysis
by Ratios — 2023-05-31T09:50:50.157Z

When discussing AI, we often associate technological proficiency with an understanding of AI's broader implications. This notion is as misguided as believing that a master printing press mechanic from Gutenberg's era would have been best equipped to predict the impacts of the printing press.

The AI development field, represented by experts like Geoffrey Hinton (proponent of safety measures and potential regulation) or Yann LeCun (an advocate of AI acceleration with fewer restrictions), primarily concerns itself with technical prowess and the creation of AI algorithms. In contrast, understanding the expansive implications of AI necessitates a comprehensive, multidisciplinary approach encompassing areas like warfare, game theory, sociology, philosophy, ethics, economics, and law, among others.

In this context, the debate between Yann LeCun and historian-author Yuval Noah Harari offers a glimpse into these diverging standpoints. While there's an understandable tendency to lend more credence to LeCun's perspective because of his technical accolades, this thought process might be a misstep when discussing the broader implications of AI.

On the other hand, Harari brings forth a more holistic perspective shaped by his wide-ranging understanding of human history, sociology, and philosophy. With his expertise in scrutinizing historical shifts and societal changes, Harari may be better prepared to explore the implications of AI because understanding implications and creating AI systems require different skill sets.

Those who can fully grasp AI's overarching impacts might not be technical experts but rather polymaths.

This post is a response to an observed bias in discussions surrounding AI: the undue weight given to technical experts' perspectives regarding the broad implications of AI. (Like in the last AI risk letter), while downplaying or ignoring the views of non-technical experts.
A rough model for P(AI doom)
by Michael Tontchev — 2023-05-31T08:58:53.313Z

(I am writing this primarily so I can reference it in another article that I'm writing to keep that article from getting longer)

Currently, my best guess is that P(AI doom by 2100) ≈ 20%. That is, there’s a 20% chance that strong AIs will be an existential challenge for humanity. But “existential challenge” hides behind academic phrasing that doesn’t drive home what this really means: there’s a 20% chance that AI will either literally eliminate humanity or permanently disempower it from choosing its future as a species, plausibly within our lifetimes.

Why 20%?

I don’t think it’s possible to give an exact calculation for this. I will start by noting that giving a number is more meaningful than just saying “wellll maybe yes, maybe no”, because then you can then start discussing “why 20%? Why not 80%? Why not 1%?”

So here's why:

My main reason is that I can find no great counterargument that knocks down all the challenges of aligning AIs. This seems to be a fundamentally hard problem, and maybe one of the hardest ones we’ll ever face, if it’s at all solvable. I don't currently see any permanent solutions that hold up against all the challenges of aligning AGIs/ASIs.

So by default, I start at 100% chance, and then make adjustments for a few reasons:

-15%: I am fallible, and there’s a chance I am wrong in my specific arguments. Even though notable experts are increasingly speaking out about AI risk, there are still smart people who disagree with me and with them. Almost no argument is bulletproof.
-20%: I don’t know everything about the topic, and there could be some big part that I'm missing. I hope so!
-30%: We might wake up (possibly due to an early, large-scale catastrophe) and 1) put in the right level of investment for the right amount of time for figuring out alignment and 2) slow down capability gain to the point where alignment significantly exceeds capabilities by the time we hit AGI.
-15%: Maybe permanent alignment turns out to be easy in a really unexpected and lucky way. Maybe?

Here's what worries me: A really big chunk of the probability of "humanity survives" comes from some form of "the AI existential risk arguments turn out to be wrong", and currently, I'm not seeing where they could realistically be wrong. If I continuously learn more about alignment but keep not finding a way out of the hard problems, the probabilities I can assign to "I turn out to be wr...
Product Endorsement: Food for sleep interruptions
by Elizabeth — 2023-05-31T01:50:02.768Z

Do you wake up before you want to and struggle with getting back to sleep when you do? One thing (out of many) that can cause this is low blood sugar*. If this is your issue, it is easily solved by eating high-fat, high-protein food.

My evidence on this is purely anecdotal; I learned this from r/adviceanimals in 2012, so the intellectual pedigree is weak. But I added ~one full sleep cycle to my night when I tried it, which lasted for years until I was forced to stop due to medication requiring an empty stomach (I now use ketone esters). A month ago my dad went from “3-5 hours/night, 3 hours awake, maybe more sleep after that” to “8-9 hours of near-continuous sleep, 1 night per fortnight a mere 6 hours” by keeping peanut butter by his bed. Downside: he’s now useless for testing the magic vibrating bracelet I’m so hyped about.

This is very easy to test. Keep some nuts by your bed; if you wake up, eat some. If you fall back asleep, congratulations on fixing your problem. There are more complicated things you can do with your evening diet to try and prevent early wake-ups entirely but I feel very muddled about these and don’t have good suggestions.

If you’re doing intermittent fasting, ketone esters (affiliate link) don’t work for many people but are great if you’re one of the lucky ones (I am), and ~don’t count as eating. MCT oils (affiliate link) work on the same principle and so may be an alternative, but I haven’t tested them personally.

I am not a doctor, anything with a real effect can hurt you, be careful not to choke or attract ants, etc.

*It’s not clear to me if this is actually low blood sugar as measured glucose in your bloodstream, or naturopathic “low blood sugar”, meaning problems you fix by eating, but I don’t care.
Optimal Clothing
by Gordon Seidoh Worley — 2023-05-31T01:00:37.541Z

I've previously argued that nice clothes are good, actually. But this was an informal claim reflective of the fact that, all else equal, nice clothes are better. But, as elsewhere in life, all else is rarely equal. Choosing clothes and designing a wardrobe is a multivariable optimization problem, and it's a problem everyone but nudists are forced to solve because we must wear something. Most people tackle the problem via intuitions, heuristics & biases, and vibes. But we're aspiring rationalists. We can do better. We can wear optimal clothing, if only we bother to try.

But before we can do better, we must first not do worse. Therefore we must identify what optimal clothing is not. Optimal clothing is not a particular style. You cannot go to the Optimal Store and buy an Optimal Fit. Anyone claiming a particular style, brand, fabric, or article of clothing is optimal, full stop, is either lying, trying to sell you something, or confused about how optimization works. Optimal must always mean optimal for something, and in the case of clothing that something is mostly up to you.

Optimal clothing is also not about optimizing for one thing to the exclusion of everything else. If you only optimize for one thing the you're unlikely to be on the Pareto frontier of fashion. Optimal clothing is not about falling for easy traps (local maxima) like "just wear whatever's comfortable" or "wear what's popular on Instagram and TikTok". You're going to have to try a little harder than that if you want your clothes to be Pareto optimized for you.

So if that's what optimal clothing is not, what does count as optimal clothing? Clothing is optimal when you've optimized it across multiple dimensions at once to find the clothes that help you to live your best life. Importantly, you need clothes for how you actually live your life (or the life you aspire to have) in the body you have and the circumstances you find yourself in, not the life you imagine you could have if your body or the world were different.

Yet many people are tempted to optimize against their own delusions. I don't say this to be mean, but to point out that we're often confused about what's going on in our lives because it takes skill and practice to notice how we actually are. Most of us have some convenient lies we tell ourselves, like that we're taller or prettier or thinner or smarter than we really are, because...