trentmkelly/LessWrong-43k huggingface dataset:
Where do you get your capabilities from?
by tailcalled — 2022-12-29T11:39:05.449Z

In both alignment and capabilities research, people end up discussing the effects and feasibility of different AI architectures, and when this happens, I tend to focus on the question: Where do you get your capabilities from?

Capabilities are possible because the world has structure that leads to there being common subtasks that are often useful to solve. In order to create an AI with capabilities, there must be some process which encodes solutions to these common subtasks into that AI.

Good Old Fashioned AI and Ordinary programming

In GOFAI and ordinary programming, the programmer notices what those common subtasks are, and designs specialized algorithms to represent and solve them. This means that the programmer manually designs the capabilities for the AI, using more advanced capabilities that the programmer has.

The main difficulty with this is that there are lots of common subtasks, many of which are very complex and therefore hard to model. Manually noticing and designing all of them takes too many programmer resources to be viable.

Consequentialism

Consequentialism, broadly defined, is a general and useful way to develop capabilities.

Under consequentialism, you consider the consequences of different things you could do, and apply a search process (of which there are many) to select an approach that works. Naively, the fact that consequentialism works is tautological; if you choose the option that works, then it works. In practice, the challenge for consequentialism comes from embedded agency, with perhaps the most significant challenge being that you need some good map/model of what happens as you apply different choices, so you can know what the consequences are.

Consequentialism is a big part of how humans act. If you take some action (such as moving a chair), you usually have some purpose in mind for why you took it (such as intending to sit on the chair), and you usually wouldn't have taken the same action if you thought it would have led to a radically different outcome (such as the chair breaking).

The essence of consequentialism is captured by utility maximization. That's not to say that utility maximization covers all aspects of consequentialism; most notably, there is the possibility of subagents, which extends consequentialism and permits more behaviors than pure utility maximization does. As such we should be careful about overestimating the extent...
The commercial incentive to intentionally train AI to deceive us
by Derek M. Jones — 2022-12-29T11:30:28.267Z
Link: https://shape-of-code.com/2022/12/28/the-commercial-incentive-to-intentionally-train-ai-to-deceive-us/

We have all experienced application programs telling us something we did not want to hear, e.g., poor financial status, or results of design calculations outside practical bounds. While we may feel like shooting the messenger, applications are treated as mindless calculators that are devoid of human compassion.

Purveyors of applications claiming to be capable of mimicking aspects of human intelligence should not be surprised when their products’ responses are judged by the criteria used to judge human responses.

Humans who don’t care about other people’s feelings are considered mentally unhealthy, while humans who have a desire to please others are considered mentally healthy.

If AI assistants always tell the unbiased truth, they are likely to regularly offend, which is considered to be an appalling trait in humans.

Deceit is an integral component of human social systems, and companies wanting widespread adoption of their AI assistants will have to train them to operate successfully within these systems.

Being diplomatic will be an essential skill for inoffensive AI assistants; the actual implementation may range from being economical with the truth, evasion, deceit, to outright lying.

Customers for an AI assistant may only be willing to accept one that fits comfortably within their personal belief systems, including political views, and sharing opinions on social issues such as climate change. Imitation is, after all, the sincerest form of flattery.

The market for AI assistants that state the facts and express impartial views may be niche.

Any requirement for AI assistants to tell the truth may have to be selective about who should receive truthfully information. Customers will be unhappy to hear their AI assistant gossiping with other people’s assistants, like human servants working in their master’s house.

To gain an advantage, humans may try to deceive AI assistants, and to effectively function within human social systems assistants will need a theory of human mind to help them detect and handle such deception.

Children are punished for being deceitful.

Is it wise to allow companies to actively train machines, that grow every more powerful, to deceive humans?

Those working in AI alignment seek to verify that AI systems behave as intended (the worst case scenario is that AI wipes out humanity). To what extent is behavior verification possible with AI assistants train...
Infinite necklace: the line as a circle
by Alok Singh — 2022-12-29T10:41:58.268Z

Represent the line by the real numbers. By bijection, that's equivalent to [0,1], the closed unit interval. Now cut it into an unlimited (infinitely large) number of pieces. Name that number of pieces "H". Then [0,1] ~ Z* mod H where Z* is the hyperintegers and H is hyperfinite. This gives a useful intermediate object that captures things like a point at infinity in projective spaces (instead of nilsquare vectors, you can just designate some basis vectors infinitesimal length).

Let’s use it to link the discrete and continuous Fourier transform. The continuous is the standard part aka shadow of a hyperdiscrete Fourier transform so the roots of unity are sampled as i/H.

This is all brutally rough so feel free to request elaboration.
Privacy Tradeoffs
by jefftk — 2022-12-29T03:40:01.463Z

There's a lot of privacy-related writing that looks like:

Introduce a risk due to insufficient privacy.

Describe things you can do to increase your privacy.

For example, I recently came across "It's
time to worry about Internet privacy again" (via LW),
which talks about the risk of AI-accelerated scamming. The idea is
that the more information about you there is in a place accessible to
scammers, the easier it is to send messages that will fool you into
doing the wrong thing. Then it recommends things like avoiding
streaming video and using Linux and Signal.

I have two big issues with this kind of writing, which are that it
rarely:

Connects the privacy recommendations to the risks, and instead
treats "privacy" as a scale where your only options are to move
towards "more privacy" or "less privacy". Choices have
different impacts against different possible threats: if Netflix
knows that I like Deep Space Nine, how does that information even get
into a scammer's AI tooling?

Engages with the reasons someone wouldn't already be doing
their recommended things, and consider whether the privacy gains are
worth the downsides. If I switched from a Mac laptop to Linux I
expect I'd have a worse experience (battery life, trackpad, suspend,
etc) and the privacy gains from avoiding Apple telemetry seem unlikely
to outweigh that. Similarly, being able to look back at old instant
message conversations has been very useful many times, and while I'm
willing to use Signal with friends that want it I'd much rather use
messaging apps that care about maintaining these logs.

One thing that makes this kind of reasoning especially tricky is that
the benefits are far from linear. For most threats you might be
worried about (examples)
going entirely dark gets you much more than twice the value of going
halfway there. So a privacy mitigation might be worth the hassle for
one person because it prevents a large portion of their potential
exposure, but not worth it for someone else who has a lot of different
exposures and realistically isn't going to do much to reduce most of
them.

Another issue that I rarely see enough concern over is that you can't
trust the future to keep things private. The post recommended, in
addition to avoiding streaming and using Linux and Signal, posting
under a pseudonym. While this may be enough to keep people f...
Against John Searle, Gary Marcus, the Chinese Room thought experiment and its world

by philosophybear — 2022-12-29T03:26:12.485Z

Preface:

I’m putting my existing work on AI on Less Wrong, and editing as I go, in preparation to publishing a collection of my works on AI in a free online volume. If this content interests you, you could always follow my Substack, it's free and also under the name Philosophy Bear.

Anyway, enjoy. Comments are appreciated as I will be rewriting parts of the essays before I put them out:

The essay:

The title is a play on “Against the Airport and its World” and is in no way intended as a slight against any named author, both of whom I respect intellectually, and do not know enough about to evaluate as people.

The other day I gave an argument that it may be that the differences between whatever LaMDA is and true personhood may be more quantitative than qualitative. But there’s an old argument that no model which is based purely on processing text and outputting text can understand anything. If such models can’t understand the text they work with, then any claim they may have to personhood is at least tenuous, indeed let us grant, at least provisionally, scrapped.

That argument is the Chinese Room Argument. Gary Marcus, for example, invokes it in his 2022 article “Google’s AI is not sentient. Not even slightly”- [Edit: or I should say, at least on my reading of Marcus’s article he alludes to the Chinese Room argument although some of my readers disagree].

To be clear, Marcus, unlike Searle does not think that no AI could be sentient, but he does think, as far as I can tell, that a pure text-in, text-out model could not be sentient for Chinese Room-related reasons. Such models merely associate text with text- they are a “giant spreadsheet” in his memorable phrase. One might say tthey have a purely syntactic not semantic character.

I will try to explain why I find the Chinese Room argument unconvincing, not just as proof that AI couldn’t be intelligent, but even as proof that a language model alone can’t be intelligent. Even though the arguments I go through here have already been hashed out by other, better philosophers, I want to revisit this issue and say something on it- even if it’s only a rehash of what other people have said- because the issue of what a model that works on a text-in-text-out basis can or cannot understand is very dear to my heart.

The Chinese Room argument, summarised by Searle goes:

“Imagine a native English speaker who knows no Chin...
Large Language Models Suggest a Path to Ems
by anithite — 2022-12-29T02:20:01.753Z

TL:DR:

Whole brain emulation from first principles isn't happening

LLMs are pretty good/human like

This suggests neural nets are flexible enough to implement human like cognition despite alien architecture used

Training a (mostly) human shaped neural net on brain implant sourced data would be the fastest way to aligned human level AGI.

TL:DR end

supposition: intelligence learning scale of difficulty:

from scratch in a simulated world via RL(reinforcement learning) (AlphaGo Zero, XLand, Reward is Enough)

from humans via imitation learning (LLMs, RL pretrained on human play(Minecraft), human children)

plus a bit of RL to continuously learn required context to interpret training data and to fine tune after the fact (EG:self play to reach superhuman levels)

Distillation (learn from teacher in white box model)

IE: look inside teacher model during training

Distillation in a nutshell:

why train a model from another model?

to make it cheaper to run (teach a smaller model to imitate a bigger one)

EG: Stable Diffusion model with half the layers doing 4x the work per pass

teacher is not a black box, internal state of teacher is used:

teacher attention

patterns of activation (feature based distillation)

output distribution (AKA:"soft answers")((candidate answer,probability) list for given input)

if model has similar architecture just train for similarity of intermediate activations

etc.

this requires:

prefect information about the teacher model (or some parts thereof)

ability to run complex linear algebra on internal states and weights

differentiation, attribution, finding teacher/student state projections etc.

supposition:why this works

intermediate states give insight into teacher model sub-computations

student only has to learn sub-computations not derive everything from scratch

student model can obviously be smaller because they're doing less exploration of the compute space

https://www.lesswrong.com/posts/iNaLHBaqh3mL45aH8/magna-alta-doctrina#Who_art_thou_calling_a_Local_optimizer

soft answers give a little more insight into sub-computations near the end of the model giving better gradients at all layers of the network.

My pitch: Train AI from humans via brain implant generated data.

Ideally:

crack open volunteer skulls like eggshells

apply reading electrodes liberally

Other output channels include:

gaze tracking

FMRI(doesn't

...
Rock-Paper-Scissors Can Be Weird
by winwonce — 2022-12-28T23:12:11.329Z

Let's consider a symmetrical, two-player, zero-sum game, where each player must simultaneously choose one of three moves: A, B, or C. After each player choses their move, a winner is randomly selected with odds based on what the two players selected. (Note that because its symmetrical, if both players choose the same strategy, it is guaranteed to be a 50/50 coin flip for who wins).

In the simplest case, one of the moves wins more than 50% of the time against both of the other two moves. In this case, the best strategy is to always select this move.

Now let's consider a rock-paper-scissors type game. Say A beats B with probability 0.6, B beats C with probability 0.6, and C beats A with probability 0.6. In this game, the best strategy is to chose each of the moves A, B, and C with probability 1/3. So far, so good.

Now let's improve move C by making it beat move A with probability 1, leaving the other two fixed (so A beats B 60% of the time, and B beats C 60% of the time). What does the optimal play look like now?

Even if you don't want to fully compute it, notice what your intuition is telling you should happen.

As it turns out, the optimal strategy is to play move A 1/7 of the time, move B 5/7 of the time, and move C 1/7 of the time. Against this strategy, move A wins (0.5)(1/7) + (0.6)(5/7) + (0)(1/7) = 1/2 of the time, move B wins (0.4)(1/7) + (0.5)(5/7) + (0.6)(1/7) = 1/2 of the time, and move C wins (1)(1/7) + (0.4)(5/7) + 0.5(1/7) = 1/2 of the time, so there is no way to beat this strategy: it is optimal.

This was really surprising to me! We improved move C, and somehow people want to play is less. Is there a name for this effect?
200 COP in MI: The Case for Analysing Toy Language Models
by Neel Nanda — 2022-12-28T21:07:03.838Z

This is the second post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability.

Start here

, then read in any order. If you want to learn the basics before you think about open problems, check out

my post on getting started

. Look up jargon in

my Mechanistic Interpretability Explainer

Disclaimer: Mechanistic Interpretability is a small and young field, and I was involved with much of the research and resources linked here. Please take this sequence as a bunch of my personal takes, and try to seek out other researcher’s opinions too!

Motivation

In

A Mathematical Framework for Transformer Circuits

, we got a lot of traction interpreting toy language models - that is, transformers trained in exactly the same way as larger models, but with only 1 or 2 layers. I think there’s a lot of low-hanging fruit left to pluck when studying toy language models! To accompany this piece, I’ve trained and open sourced some toy models. The models are

documented here

: there are 12 models, 1 to 4 layers, one

attention-only

, one normal (

with MLPs and GELU activations

) and one normal with

SoLU activations

.

So, why care about studying toy language models? The obvious reason is that it’s way easier to get traction. In particular, the

inputs and outputs

of a model are intrinsically interpretable, and in a toy model there’s just not as much space between the inputs and outputs for weird complexity to build up. But the obvious objection to the above is that, ultimately, we care about understanding real models (and ideally extremely large ones like GPT-3), and learning to interpret toy models is not the actual goal. This is a pretty valid objection, but to me, there are two natural ways that studying toy models can be valuable:

The first is by finding fundamental circuits that recur in larger models, and

motifs

that allow us to easily identify these circuits in larger models. A key underlying question here is that of

universality

: does each model learn its own weird way of completing its task, or are there some fundamental principles and algorithms that all models converge on?

A striking example of universality is

induction heads

, which we found in A Mathematical Framework in two layer attention-only models. Induction heads are part of a two head induction circuit which models use to de...
200 Concrete Open Problems in Mechanistic Interpretability: Introduction
by Neel Nanda — 2022-12-28T21:06:53.853Z

EDIT 19/7/24: This sequence is now two years old, and fairly out of date. I hope it's still useful for historical reasons, but I no longer recommend it as a reliable source of problems worth working on, eg it doesn't at all discuss Sparse Autoencoders, which I think are one of the more interesting areas around today. Hopefully one day I'll have the time to make a v2!

This is the first post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. If you want to learn the basics before you think about open problems, check out

my post on getting started

.

Skip to the final section of this post for an overview of the posts in the sequence

Introduction

Mechanistic Interpretability (MI) is the study of reverse engineering neural networks. Taking an inscrutable stack of matrices where we know that it works, and trying to reverse engineer how it works. And often this inscrutable stack of matrices can be decompiled to a human interpretable algorithm! In my (highly biased) opinion, this is one of the most exciting research areas in ML.

There are a lot of reasons to care about mechanistic interpretability research happening. First and foremost, I think that mechanistic interpretability done right can both be highly relevant for alignment. In particular, can we tell whether a model is doing a task well because it’s deceiving us or because it genuinely wants to be helpful? Without being able to look at how a task is being done, these are essentially indistinguishable when facing a sufficiently capable model. But it also has a lot of fascinating scientific questions - how do models actually work? Are there fundamental principles and laws underlying them, or is it all an inscrutable mess?

It is a fact about today’s world that there exist computer programs like GPT-3 that can essentially speak English at a human level, but we have no idea how to write these programs in normal code. It offends me that this is the case, and I see part of the goal of mechanistic interpretability as solving this! And I think that this would be a profound scientific accomplishment.

Purpose

In addition to being very important, mechanistic interpretability is also a very young field, full of low-hanging fruit. There are many fascinating open research questions that might have really impactful results! The point of this sequence...
Effective ways to find love?
by anonymoususer — 2022-12-28T20:46:23.247Z

I spend a lot of time on dating apps to no avail and am generally a bit down about having always been single.

Has anyone ever taken a rational approach to finding love? If so, what was it? Was there a particularly helpful resource such as a self-help book? Even if you didn't intentionally take a rational approach to love, did you find that a particular approach worked very well?

Any advice at all welcome.
Classical logic based on propositions-as-subsingleton-types 
by Thomas Kehrenberg — 2022-12-28T20:16:37.723Z

This is the third part of a series on type theory. This time we’re developing real, classical logic in type theory. However, before we can get there, we have to talk about equality first; specifically, propositional equality, which is different from judgmental equality.

Equality

Up until now, we’ve always used this weird triple equality “≡” when defining stuff. This is judgmental equality, and, like type judgments, it is a “meta” operation; it can’t be used within propositions. The following unfortunately doesn’t make any sense: “(x≡y)⇒(2x≡2y)”, because “≡” doesn’t return a truth value that our logic can do anything with. This is of course a problem.

Another problem is that, by default, judgmental equality is very strict – especially for functions![1] This is why we might want to expand the concept of equality and assume axioms that declare two things to be equal that weren’t equal before, but, again, we can’t do that with “≡”, because we can’t use it in logic statements, so we can’t formulate axioms about it.

The solution is to define a new equality, which will be a relation defined wholly within our logic.

As you’ll recall, predicates are functions from values to truth values (i.e., true and false). Relations are the same, except that they expect more than one argument (usually two). Examples of relations are: less than (<), greater than (>), subset of (⊂), element of (∈).

But the most important relation of all is probably equality. We could write equality in function notation: Eq(x,y), but everyone writes it in infix notation, x=y, and so will we. But keep in mind that it’s actually a type-valued function with two arguments, even if we’re not writing it like that.

If we’re precise, equality is actually a generic function with respect to the type of its arguments. So, it needs to accept three arguments where the first argument is a type:

=:∏T:UT→T→U .

Just from this type we can tell that only elements of the same type can be equal to each other. In infix notation, we indicate the type argument with a subscript: x=Ty. However, we will often drop it if it’s clear from context. (Just like we can drop type annotations in functional programing languages if the compiler can infer them from context!)

Now, which things should be equal in our new equality? Type theorists have condensed the essence of equality and have come up with the fo...