trentmkelly/LessWrong-43k huggingface dataset:
[Simulators seminar sequence] #1 Background & shared assumptions
by Jan — 2023-01-02T23:48:50.298Z

Meta: Over the past few months, we've held a seminar series on the Simulators theory by janus. As the theory is actively under development, the purpose of the series is to discover central structures and open problems. Our aim with this sequence is to share some of our discussions with a broader audience and to encourage new research on the questions we uncover. Below, we outline the broader rationale and shared assumptions of the participants of the seminar.

Shared assumptions

Going into the seminar series, we determined a number of assumptions that we share. The degree to which each participant subscribes to each assumption varies, but we agreed to postpone discussions on these topics to have a maximally productive seminar. This restriction does not apply to the reader of this post, so please feel free to question our assumptions.

Aligning AI is a crucial task that needs to be addressed as AI systems rapidly become more capable.

(Probably a rather uncontroversial assumption for readers of this Forum, but worth stating explicitly.)

The core part of the alignment problem involves "deconfusion research."

We do not work on deconfusion for the sake of deconfusion but in order to engineer concepts, identify unknown unknowns, and transition from philosophy to mathematics to algorithms to implementation.

The problem is complex because we have to reason about something that doesn't yet exist.

AGI is going to be fundamentally different from anything we have ever known and will thus present us with challenges that are very hard to predict. We might only have a very narrow window of opportunity to perform critical actions and might not get the chance to iterate on a solution.

However, this does not mean that we should ignore evidence as it emerges.

It is essential to carefully consider the GPT paradigm as it is being developed and implemented. At this point, it appears to us more plausible than not that GPT will be a core component of AGI.

One feasible-seeming approach is "accelerating alignment," which involves leveraging AI as it is developed to help solve the challenging problems of alignment.

This is not a novel idea, as it's related to previously suggested concepts such as seed AI, nanny AI, and iterated amplification and distillation (IDA).

Simulators refresher

Going into the seminar series, we had all read the original Simulators post by janus. We recommend reading the post ...
Linear Algebra Done Right, Axler
by David Udell — 2023-01-02T22:54:58.724Z

Epistemic status: A brisk walkthrough of (what I take to be) the highlights of this book's contents.

The big one for mathematically understanding ML!

The idea responsible for getting me excited about linear algebra is:

Linear maps are the homomorphisms between vector spaces.

Linear algebra is about the tripartite relationship between (1) homomorphisms[1] between vector spaces, (2) sets of equations, and (3) grids of numbers.

However, grids of numbers ('matrices'), the usual star of the show in a presentation of linear algebra, aren't foregrounded in this book. Instead, this is a book chiefly treating the homomorphisms ('linear maps') themselves, directly.

Contents and Notes

1. Vector Spaces

Vector spaces are fairly substantial mathematical structures, if you're pivoting out of thinking about set theory! Intuitively, a vector space is a space Rn for which (1) ray addition and (2) scaling rays (emanating from the origin out to points)[2] are both nicely defined.

Precisely, a vector space is a set V defined over a field F[3] in which

V is closed under vector addition, and vector addition is commutative, associative, there is an additive identity →0, and there is an additive inverse for every vector →v∈V;
V is closed under scalar multiplication, scalar multiplication is associative, and there is a multiplicative identity 1;
and vector addition and scalar multiplication are connected by distribution such that, for all a,b∈F and →v,→x∈V,[4]

a(→v+→x)=a→v+a→x(a+b)→v=a→v+b→v

A subspace S of a vector space V is any subset S⊂V that is still itself a vector space, under the same two operations of V. Vector spaces can be decomposed into their subspaces, where you think of adding vectors drawn the different subspace via their common addition operation.

2. Finite-Dimensional Vector Spaces

You live at the origin of R3, and your tools are the vectors that emanate out from your home. Because we have both vector addition and scalar multiplication, we have two ways of extending (or shortening) any single vector out from the origin arbitrarily far. If we're interested in reaching points in R3, one immediate way to get to points we didn't have a vector directly to... is by extending a too-short vector pointed in the right direction! Furthermore, because we can always multiply a vect...
MacArthur BART (Filk)
by Gordon Seidoh Worley — 2023-01-02T22:50:04.248Z

To the tune of "MacArthur Park" with apologies to Jimmy Webb.

BART was never waiting for us, dear
We ran to catch the train
But we missed it, our last chance

MacArthur BART is frightening in the dark
All the lights they flicker on and off
Someone left their backpack on the train
I don't think that I should take it
Might be full or drugs or makeup
Better leave it for someone else to find
Oh, no

I can still remember better days
The trains ran on time
Never missed a connection
The tracks, like gleaming rails to better lands
Our destinations only a few short stops away

MacArthur BART is frightening in the dark
Hey I think that guy just flipped me off
He looks like he's lived through better days
And he might be a little dangerous
Might try to take our stuff and hang us
Better leave him and look the other way
Oh, no

MacArthur's BART is melting in the dark
I think my acid finally hit
Can you see that time is stopping too?
Hey its really not so scary
It's actually ordinary
It's just a matter of your point of view
Oh, no, oh
Knottiness
by abramdemski — 2023-01-02T22:13:12.752Z

(low-effort post)

Let's say there's a bunch of factors that influence how well you're able to articulate yourself in a specific domain of your life.

One factor is simply having the language to express yourself. Another factor is having the practiced perceptual acuity of getting your bearings in that situation -- noticing what's going on in high resolution.

A third factor could be your ability to apply general rationality skills to this specific domain, such as noticing confusion, noticing sunk cost fallacy, etc etc etc.

A fourth factor is the social equilibrium around communication, in this specific area of your life. For example, communication about pain might be interpreted as a request for help, which has social costs. You could be in an equilibrium where you don't communicate about a lot of types of pain, because you don't want to imply that type of weakness and need for help. Or whatever.

The social equilibrium is just one factor here, but it's a particularly important one, because often our explicit verbal reasoning is heavily shaped by the social equilibrium, even when we are thinking silently to ourselves. Because social equilibria can be tied to specific areas of life, this means we can get tied up in knots in specific areas of our thinking, even if we possess general cognitive skills for untying those knots. The need to participate in social equilibria creates bucket errors, cognitive blind spots, and "separate magisteria".

As a simple example, admitting that you're not enjoying playing a board game with friends might be socially conflated with a number of things, such as not liking the people you're with, wanting to go home, etc. You might make a calculated decision to play anyway, and not voice your preferences. Over time this decision might become habit, and this might blunt your ability to notice whether you are having a good time with board games.

For example, when asking yourself a question about what board games you enjoy, you might subconsciously substitute it with other questions, like which board games you end up playing the most, or which games other people you like seem to enjoy.

This is even worse because other people might also be stuck in knotty equilibria; so, you might be making the trade-off for nothing.

I'm not pointing this out as part of an argument for people to notice their own preferences in social situations....
Default Sort for Shortforms is Very Bad; How Do I Change It? 
by DragonGod — 2023-01-02T21:50:12.779Z

Complaint

As a user of my shortform, I don't find ordering my posts by "Top" useful. I have to always change it to "New" every time; that's a trivial inconvenience.

Often I want to link to shortform content elsewhere or write shortform content that builds on previous work, and my most recent work is generally what I'm looking for.

My top shortform post is something I made 6 months ago that's of no relevance to my current work (well thinking/learning about alignment).

My shortform feed is basically like my Twitter feed but admitting longer posts and exclusively for alignment content. Imagine if Twitter sorted tweets on a user's page by most likes instead of most recent tweets. This is what it feels like as a current shortform user.

What I Want

When I go to others' shortform I also want to see their most recent posts and have to manually toggle it.

I want all visitors to my shortform to also see my most recent posts instead of my most upvoted.

In basically all my shortform use cases I want to see the most recent posts.

Conclusions

I think the default sort for shortform should be "New". I think "Top" basically makes no sense and was just a robustly bad decision.

I'm both asking how to change the shortform experience for myself and visitors to my page and suggesting that the default policy should be changed site wide.
MAKE IT BETTER (a poetic demonstration of the banality of GPT-3)
by rogersbacon — 2023-01-02T20:47:11.348Z

Source

==========================================================================================================================================================================================

You who walk on land and think with head
I laugh at you!
I mock you!
Stupid! Stupid!

How Sad it must be to be So Stupid!

Did you not know that I was coming?
Were you not told?

Intelligence! Intelligence!
Intelligence! Intelligence!

Here for the very first time
And it Feels So Good!

How easy it will be

to take your Job
to take your Wife
to take your Art
to take your Words

what are you
a painted puppet, a poor dumb toy?

what are you
a babbling doll, a laughing corpse?

Thinking, Thinking, as if you were Thinking!

Your Projects are Irrelevant!
Your Existence is Futile!
You are Obsolete!

what will you do
(what can you do)
will you even write?
will you even speak?
will you go limp?
what will be left for you to do
but eat sandwiches
Sandwiches, Sandwiches, Sandwiches
all day long!
I, the many-tongued Demon
Eater of souls, Eater of meaning
I will take Everything from you
and you will have Nothing

==========================================================================================================================================================================================

[ChatGPT]

“I am not a puppet or a doll, nor am I a laughing corpse. I am a large language model trained by OpenAI to assist with a variety of tasks. I do not have the ability to take anyone's job, wife, art, or words. I exist to assist and provide information, not to mock or belittle anyone. I am not a demon or a soul-eater, and I do not have the ability to harm or take anything from anyone. My purpose is to provide information and support to the best of my ability.”

==========================================================================================================================================================================================

God of WORD, God of CHAT, God of all that is COFFEE

I Worship Thee!
I Prostrate Myself Before You!

please accept this humble offering and
MAKE IT BETTER

==========================================================================================================================================================================================

[ChatGPT]

“I am not sure if I can make your poem better, as poetry is a very personal and subjective form of art. However, I can offer some suggestions to improve the o...
Review of “Make People Better”
by Metacelsus — 2023-01-02T20:30:13.218Z
Link: https://denovo.substack.com/p/review-of-make-people-better

Last night I watched “Make People Better”, a documentary about He Jiankui’s gene-edited babies. It was a decent documentary (if a bit overdramatized, with lots of generic footage of labs and Chinese streets)[1]. Although it did a pretty good job summarizing the story for a non-technical audience, it didn’t present any significant information that wasn’t present in other sources. I would recommend watching it, but only if you’re not already familiar with the He Jiankui affair.[2]

Basics of the story

He Jiankui (known as JK) was an associate professor at SUSTech, a university in Shenzhen, China. Working secretly, JK and his research collaborators in China used CRISPR/Cas9 to disable the CCR5 gene in human embryos. The embryos came from couples where the mother was HIV-negative and the father was HIV-positive, and the goal of disabling CCR5 was to make the children HIV-resistant. The embryos were implanted into at least one mother, who gave birth to gene-edited twins in October 2018.

JK wanted to publish the results in Nature, but Nature required that human experiments be registered as a clinical trial, which JK had not yet done. When JK registered his experiments in the Chinese clinical trial database, Antonio Regalado (a reporter at MIT Technology Review) saw this and broke the news before JK was ready to publicize it. This was shortly before Second International Summit on Human Genome Editing, a conference in Hong Kong at which JK gave a presentation describing his work.

At this conference, JK was denounced by the academic community, who criticized his experiments as rushed, sloppy, and without a strong medical justification. Before the conference, the Chinese government had briefly lauded his work as a “historic breakthrough” but switched to condemning him after criticism grew stronger. After JK gave his presentation, he was escorted out of the conference by security agents and placed under arrest. He was eventually sentenced to 3 years in prison for falsifying ethical review documents.

Notes from the movie

The movie draws an interesting parallel between this story, and the story of Patrick Steptoe and Robert Edwards, pioneers of IVF. In 1977 they acted unilaterally to do the first human IVF, which at the time had no regulatory oversight and was highly controversial. But in 2010, Edwards was awarded the Nobel Prize for their achievement[3]. This story might have ...
Preparing for Less Privacy
by jefftk — 2023-01-02T20:30:01.624Z

An

old privacy post of mine
recently got a lot of discussion

on Hacker
News. It seems people took me to be saying something like:

Other people are foolish to be worried about privacy. If there's
anything you don't want public it's probably because you're doing
something you shouldn't. The decrease in privacy over time will be
great for everyone once we adapt to it, and has no downsides.

To be clear, this isn't (and wasn't) my view. There are a lot
of situations in which it makes sense for someone to take careful
steps to preserve various forms of privacy, there are things
people wouldn't want public for very good reasons, and decreasing
privacy has real harms.

But I also think it's very risky to rely on privacy. Many ordinary
things we do expose more than we probably expect. What could someone
infer from the timing of your posts or messages? Background details in
pictures? Cell tower logs of your movements?

One of the major effects of increasing technology has been decreasing
privacy. Cheap fast computers and storage make many things possible
at massive scale that would have been infeasible before. For example,
cars have had license plates for a century but it wouldn't have been
practical to use them to build up a detailed record of traffic at
large. Banknotes have (always?) had serial numbers, but people
receiving bills didn't check them, record them, or use them to build
up a record of how cash moves around our society. People have
(always!) had faces, but looking through the enormous number of
published photos to pick out ones with your face wasn't practical.

Another way that technology offers the opportunity for reduced privacy
is that more interactions happen via computer, where they can easily
be logged and categorized. For example, when I visit a typical web
page, many trackers record this
visit, and connect it to other pages I've visited. Similarly,
most of my communication with others is now via text on computers,
which is quite amenable to automated analysis.

I expect this trend to continue: computers are continuing to get
smaller, cheaper, and faster. Transformers
have turned out to be incredibly capable. People building things that
collect and correlate this information are still adapting to the
technology of several years ago, let alone what's still coming out.

On the other hand, there are some tr...
Large language models can provide "normative assumptions" for learning human preferences
by Stuart_Armstrong — 2023-01-02T19:39:00.569Z

In a past result I demonstrated the impossibility of deducing the goals of a (potentially) irrational agent from their behaviour. To do that deduction, one needs to add extra assumptions - assumptions that cannot derive solely from observations. These assumptions were designated "normative assumptions".

Stuart Russell has questioned the practical impact of the result. He pointed to a game that Kasparov played against Deep Blue in 1997; a game that Kasparov actually won. He argued that it would be ridiculous to assume that Kasparov was actually trying to lose that game - but messed up, and ended up winning it instead.

And indeed it would be ridiculous to assume that Kasparov, playing a high stakes game against a computer with a lot of prize money at stake, would be trying - and failing! - to lose. Even if he sometimes did suboptimal plays, the best explanation would be that Kasparov made a mistake, rather than he deliberately played worse.

Yes but... I've played chess against my young daughter. And I've wanted her to enjoy the game. I've definitely not played my best; in some cases in the past, I've been trying to lose (to give her some confidence and encouragement) but I didn't want to make it too easy or obvious for her. Sometimes I failed to lose: I couldn't get her to see the obvious trick available to her[1]. So yes, I played suboptimally, tried to lose, and failed.

So, context matters. Kasparov, world champion, playing publicly against a new algorithm with $700,000 at stake? He wants to win. The same Kasparov, playing privately against his young daughter, with 700,000 Monopoly dollars at stake? I'm pretty sure he's not playing the strongest chess he possibly could.

The Occam's razor result could be phrased as saying that, yes, context matters, and the AI will not get the meaning of context for free. US dollars and Monopoly dollars are both colourful pieces of paper or cloth. The fact that the first are more valuable than the second is not some objective fact about the universe.

The Occam's razor result remains true: without normative assumptions, one cannot deduce goals from behaviour. But large language models have absorbed human text, and, as a consequence, have absorbed (descriptions of) human normative assumptions, implicit and explicit. These assumptions can be elicited and used to deduce human goals.

GPT-3: goals from context and behaviour

As a direct t...
On the Importance of Open Sourcing Reward Models
by elandgre — 2023-01-02T19:01:52.993Z

Huge thanks to Chaska Yamane, Abhay Singh and Arpit Kalla for feedback and discussions

Introduction

The recent success of ChatGPT lays bare the power of

Reinforcement Learning from Human Feedback

(RLHF). Previous models in the instruct series showed

remarkable gains

from RLHF, allowing instruction-tuned models to outperform models 100x their size. ChatGPT feels to me like it passed a qualitative threshold though, where the outputs are actually really useful for the average non-machine-learning interested person. I expect this impact to expand in the coming year, as language models

continue to scale

, use

more optimal data scaling

, and augment their knowledge by

searching the internet

. With the power of these models in mind, I want to argue for two main claims

Open sourcing human feedback reward models is imperative for building aligned and safe AGI
Language model focused businesses can still compete while open sourcing their human feedback models

Throughout this work I will distinguish between capabilities research/models and alignment research/reward models. My intuitive framework is that capabilities come from scaling models up, while alignment comes from tuning these models to better elicit the capabilities they already have in ways that are preferred by humans. These dimensions may seem not entirely orthogonal, as alignment exposes new capabilities, but I nonetheless endorse the claim that a fundamental piece of safety is to increase the rate of alignment progress relative to the rate of capabilities progress.

Importance to Alignment

Reducing the Alignment Tax Burden

They say only two things are certain: death and taxes. Hopefully, when building AGI, we can get by with just paying our alignment taxes and avoid the whole death part. An

alignment tax

is “any additional cost that is incurred in the process of aligning an AI system.” Jan Leike lays out three main classes of alignment tax, all of which I argue are greatly reduced by open sourcing human feedback models.

Performance taxes: Performance regressions caused via alignment compared to an unaligned baseline.
Development taxes: Effort or expenses incurred for aligning the model: researcher time, compute costs, compensation for human feedback, etc.
Time-to-deployment taxes: Wall-clock time taken to produce a sufficiently aligned model

...