trentmkelly/LessWrong-43k huggingface dataset:
The First Filter
by adamShimi — 2022-11-26T19:37:04.607Z

Consistently optimizing for solving alignment (or any other difficult problem) is incredibly hard.

The first and most obvious obstacle is that you need to actually care about alignment and feel responsible for solving it. You cannot just ignore it or pass the buck; you need to aim for it.

If you care, you now have to go beyond the traditions you were raised in. Be willing to go beyond the tools that you were given, and to use them in inappropriate and weird ways. This is where most people who care about alignment tend to fail — they tackle it like a normal problem from a classical field of science and not

an incredibly hard and epistemologically fraught problem

.

If you manage to transcend your methodological upbringing, you might come up with a different, fitter approach to attack the problem — your own weird inside view. Yet beware becoming a slave to your own insight, a prisoner to your own frame; it’s far too easy to never look back and just settle in your new tradition.

If you cross all these obstacles, then whatever you do, even if it is not enough, you will be one of the few who adapt, who update, who course-correct again and again. Whatever the critics, you’ll actually be doing your best.

This is the first filter. This is the first hard and crucial step to solve alignment: actually optimizing for solving the problem.

When we criticize each other in good faith about our approaches to alignment, we are acknowledging that we are not wedded to any approach or tradition. That we’re both optimizing to solve the problem. This is a mutual acknowledgement that we have both passed the first filter.

Such criticism should thus be taken as a strong compliment: your interlocutor recognizes that you are actually trying to solve alignment and open to changing your ways.
Respecting your Local Preferences
by Scott Garrabrant — 2022-11-26T19:04:14.252Z

In this post, I give a application of geometric rationality to a toy version of a real problem.

A Conflicted Agent

Let's say you are an agent with two partially conflicting goals. Part of you wants to play a video game, and part of you wants to save the world and tile the multiverse with computronium shaped exactly the way you like it (not paperclips). How should these conflicting interests figure out what to do? (Assume you have not yet had the idea of starting a video game company to save the world.)

We will assume that 2/3 of you wants to save the world, and 1/3 of you wants to play video games.

At first your world-saving-self has the bright idea that 2/3 is bigger that 1/3, and you should therefore devote all your time to saving the world. However, this proposal doesn't stick. Eventually, you end up Nash bargaining with your time, and devoting 2/3 of your time to saving the world and 1/3 of your time to playing video games. This works well for a while, but then your world-saving-self has a new bright idea:

"Let's look around at the world, and see how much ability we expect to have to save it. If it feels like we are in the top 60 percentile of worlds ordered by how much control we have, then we will try to save the world. If we are in the bottom 40 percentile, we will play video games!"

(The 60 percentile is arbitrarily rounding down from 2/3, so that you can both play more video games and save more worlds.)

A Nash Bargaining Model

Let's model this more carefully. Let's say there are five different types of worlds: 1, 2, 3, 4, and 5. In each world, you have two buttons in front of you. The video game button, and the save the world button. Each time step, pressing the video game button lets you play some video games, and pressing the save the world button saves the world with probability ε⋅i in the world of type i.

We have five degrees of freedom. For each i∈{1,2,3,4,5}, we have pi, which represents the proportion of our time in world i that we spend pressing the save the world button. The rest of the time is spent playing video games.

The part of you that wants to save the world has power 23, and utility equal to 1010100⋅ε⋅(p1+2p2+3p3+4p4+5p5). However, since we are Nash bargaining, the coefficient out in front does not matter.

The part of you that wants to play video games has power 13 and util...
Opinions on the sleep synaptic homeostasis hypothesis?
by Angela Pretorius — 2022-11-26T19:01:47.602Z

To a very crude approximation, the brain works via Hebbian learning. If neuron A fires immediately before neuron B then the synapse from A to B gets strengthened.

There’s just one problem with Hebbian learning. Carry on long enough and you will eventually end up in a situation where every neuron just fires like crazy all the time. Giulio Tononi’s Synaptic Homeostasis Hypothesis proposes that the function of NREM sleep is to normalise Hebbian learning by weakening synapses.

Some experiments appear to support the synaptic homeostasis hypothesis (e.g.[1]). Other experiments instead find evidence of synaptic potentiation during sleep (e.g. [2]). I’ve seen competing claims that NREM sleep instead normalises Hebbian learning by weakening synapses on neurons that fire a lot and strengthening synapses on neurons that don’t fire very often. Or that neurons use BCM theory rather than Hebbian learning and BCM theory doesn’t need normalising.

I’d like to know what other LessWrong users think about the Synaptic Homeostasis Hypothesis?

[1] Vyazovskiy, V., Cirelli, C., Pfister-Genskow, M. et al. Molecular and electrophysiological evidence for net synaptic potentiation in wake and depression in sleep. Nat Neurosci 11, 200–208 (2008). https://doi.org/10.1038/nn2035

[2] Jaclyn Durkin, BS, Sara J. Aton, PhD, Sleep-Dependent Potentiation in the Visual System Is at Odds with the Synaptic Homeostasis Hypothesis, Sleep, Volume 39, Issue 1, January 2016, Pages 155–159, https://doi.org/10.5665/sleep.5338
Why square errors?
by Aprillion — 2022-11-26T13:40:37.318Z

Mean squared error (MSE) is a common metric to compare performance of models in linear regression or machine learning. But optimization based on the L2 norm metrics can exaggerate bias in our models in the name of lowering variance. So why do we square errors so often instead of using absolute value?

Follow my journey of deconfusion about the popularity of MSE.

MSE is sensitive to outliers

My original intuition was that optimizing mean absolute error (MAE) ought to always create "better" models that are closer to the underlying reality, capture the "true meaning" of the data, be more robust to outliers. If that intuition were true, would it mean that I think most researchers are lazy and use an obviously incorrect metric purely for convenience?

MSE is differentiable, while MAE needs some obscure linear programming to find the (not necessarily unique) solutions. But in practice, we would just call a different method in our programming language. So the Levenberg–Marquardt algorithm shouldn't look like a mere convenience, there should be fundamental reasons why smart people want to use it.

Another explanation could be that MSE is easier to teach and to explain what's happening "inside", and thus more STEM students are more familiar with the benefits of MSE / L2. In toy examples, if you want to fit one line through an ambiguous set of points, minimizing MSE will give you that one intuitive line in the middle, while MAE will tell you that there are infinite number of equally good solutions to your linear regression problem:

MAE might not give you a unique solution

I see this as an advantage of MAE or L1, that it reflects the ambiguousness of reality ("don't use point estimates pf linear regression parameters when multiple lines would fit the data equally well, use those damned error bars and/or collect more data"). But I also see that other people might understand this as an advantage of MSE or L2 ("I want to gain insights and some reasonable prediction, an over-simplified fast model is enough, without unnecessary complications").

Previously, I also blamed Stein's Paradox on L2, but that does not seem true, so other people probably formed their intuitions about L1 and L2 without imaginary evidence against one of them. If both norms are equally good starting points for simplifying reality into models, then computational performance will be a bigger factor in the considera...
Three Alignment Schemas & Their Problems
by Shoshannah Tekofsky — 2022-11-26T04:25:49.206Z

This is the first journal entry for my 6 months of alignment studies. Feedback and thoughts are much appreciated! I'm hoping that documenting this process can serve as a reference for self-study methods in AIS.

I'm not sure how one goes about solving alignment. So for lack of an authoritative approach, I sat down to sketch out the problem from scratch. First off...

What happens when AI becomes more intelligent than humans?

My understanding (and hope!) is that we are currently in the green area: AI is below human range -- We're safe.

Now what are the potential scenarios from this point? Let's review the entire possibility space.

Intelligence Ceiling - AI can't get smarter than us because human intelligence is about as good as it gets. This seems extremely unlikely on first principles, but we can't empirically rule it out.
Slow Take-Off - There is no Recursive Self-Improvement so we'll have time to adjust to AGI showing up. Problems will be progressive which means we can mobilize and experiment. This is kind of okay and manageable.
Fast Take-Off Post-AGI - Recursive Self-Improvement is real and kicks in around human-level intelligence. We're sitting on a ticking time bomb but at least we know when it will go off, and it's plausibly not extremely soon.
Fast Take-Off Pre-AGI - Recursive Self-Improvement is real and kicks in significantly before human-level intelligence! This is scary, and soon, and we'd better make sure all our current AI is aligned yesterday. Let's get to work.

What does AI end up doing as it becomes more intelligent?

This question has a couple of branches. First off, we need to understand what intelligence is and what we'd like AI to do in the first place. This week I mostly dove into the latter part of that equation. Let's see what we want to align on before we figure out how to get there.

Yet the question what we want to align on also consists of two parts: who is this "we" and what are "wants"? Looking at the "we", the Minimal Viable Product (MVP) to solve for is a single human. From there, we can hopefully scale up to more humans and then all of humanity (Ideal). Notably, such scaling involves a process of value aggregation, which is a challenge in itself.

On the other hand, what are "wants"? This opens up a question on the nature of goals and if some goals might be easier or harder to align on than others. For instance, complexity of goals migh...
The many types of blog posts
by Adam Zerner — 2022-11-26T03:57:43.293Z

What are blog posts? I think they can be many different things.

Essays

For Paul Graham, they are essays. He doesn't even call that part of his page "blog". The link in the left navigation area says "Essays". Essays are different from what I think of as a prototypical[1] blog post.

But the URL is /articles. And articles are different from essays. Hm.

To Paul Graham, an essay is something that is exploratory.

To understand what a real essay is, we have to reach back into history again, though this time not so far. To Michel de Montaigne, who in 1580 published a book of what he called "essais." He was doing something quite different from what lawyers do, and the difference is embodied in the name. Essayer is the French verb meaning "to try" and an essai is an attempt. An essay is something you write to try to figure something out.

Figure out what? You don't know yet. And so you can't begin with a thesis, because you don't have one, and may never have one. An essay doesn't begin with a statement, but with a question. In a real essay, you don't take a position and defend it. You notice a door that's ajar, and you open it and walk in to see what's inside.

An essay, according to him, also doesn't take a stance on an issue. And it doesn't aim to persuade.

The other big difference between a real essay and the things they make you write in school is that a real essay doesn't take a position and then defend it.

...

The sort of writing that attempts to persuade may be a valid (or at least inevitable) form, but it's historically inaccurate to call it an essay. An essay is something else.

Opinion pieces

Ok then. An essay isn't meant to persuade, just explore. But some blog posts are meant to persuade. I guess we can call those opinion pieces.

To me, a prototypical opinion piece is pretty well researched and thought out. It isn't something the author garbled together at 3am in 20 minutes in a steaming vent of frustration. I guess the latter is a third type of blog post, but I'm not going to talk about that category here.

Educational

Some of my favorite blog posts are educational. Most recently I came across Josh W Comeau's post An Interactive Guide to Flexbox. It was pretty awesome.

It was also long. A deep dive. Not all educational blog posts are that long or deep. Some are just a three minute description of a concept or technique.

I suppose you could come up with subcategori...
New Frontiers in Mojibake
by Adam Scherlis — 2022-11-26T02:37:27.290Z
Link: https://adam.scherlis.com/2022/11/25/new-frontiers-in-mojibake/

Fun with mismatched encodings

Mojibake is the garbled text that result from character-encoding errors.

If youâ€™ve seen text that looks like this â€” and Iâ€™m sure you have â€” then youâ€™ve seen mojibake.

(You should be seeing something like this:

If you see something else, this post may be a little confusing and you need a new web browser.)

Computers represent text as a sequence of bytes, and “text encodings” are dictionaries that turn characters (i.e. symbols: letters, punctuation, etc.) into bytes and vice-versa.

The garbled text above is a pretty common species of mojibake. It’s what happens when em-dashes and curly apostrophes are encoded as bytes with UTF-8 (the now-nearly-universal text encoding) and decoded back to characters with Windows-1252 (an obsolete encoding that is still pretty widespread).

Windows-1252 is pretty straightforward: each character gets one byte, and there are only 256 characters so this works out.

UTF-8 is one of several character encodings based on Unicode, which (for our purposes) is a massive numbered list of over 100,000 characters. Unicode includes nearly every language in the world and a bunch of other nonsense like emojis.

UTF-8 turns each character into a sequence of up to four bytes based on its Unicode “codepoint” (position in the list). Codepoints are bigger than bytes, so you still need this translation step.

(I’m simplifying a little here, and you should be grateful for that.)

Specifically, an em-dash gets mangled like this:

EM DASH (—) is Unicode character #8,212, usually written (in hex) as U+2014.
UTF-8 encodes the number 8,212, which is too big to fit in a single byte, as the sequence of bytes 0xE2, 0x80, 0x94
Windows-1252 looks at each byte in turn and decodes them directly as the characters â, €, ” respectively.
Finally, your computer looks up the characters â, €, ” in some font file and draws the specific glyph for each of those characters in that font. (A “glyph” is the actual picture on your screen; a “character” is the abstract concept of the euro symbol or whatever.)

(I made this happen deliberately with python: '\u2014'.encode('utf8').decode('1252').)

This sometimes happens to the same text multiple times, and special characters turn into exponentially-growing strings of nonsense that overwhelm the text:

An astrological mystery

I saw this today on a used-book website:

Astrological symbols??

My first thought was that this w...
Semi-conductor/AI Stock Discussion.
by sapphire — 2022-11-25T23:35:57.908Z

I've been writing about investing and the EMH [1] on lesswrong/rat-discord/ea-facebook/etc. I made the first 'buy solana' post on the EA investing group when it was under 2 dollars in late 2020 (it peaked at 260 in 2021). Despite crypto crashing and FTX stealing user deposits I got my money out and I've had a very good two years investing. Post tax I multiply my net worth by about a factor of eight; my starting net worth was not small. My income during this period wasn't significant compared to my portfolio size. However some friends really wish I had been posting more systematically. I 'sold the top' but many people I got into Solana didn't. So me and some friends are going to regularly post on twitter. Please follow us [2] if you want updates systematically!

Anyone who thinks AI is going to be transformative should allow this to inform their investing. Im interested in how lesswrong is investing! Here are some brief thoughts on stocks we like for this purpose, mostly semis:

1) TSM - Significant technical lead on other semi-conductor companies. Only close competitor is Samsung. PE ratio of 17! Probably the most important company in the world right now. Only downside is risk China invades Taiwan. Pure semis play.

2) Google - Owns Deep Mind and Google Brain. Huge piles of money and data to train AI. PE of 19. Downside: Ad revenue slipping.

3) Samsung - Only serious technical competitor to TSM. Insanely low PE of 8.58. Downside: not a very pure semis play.

4) Nvidia - Still the leader in designing GPUs. In house AI research. Downsides: Doesn't manufacture their own chips, high PE ratio of 55.

5) ASML - Makes hardware for semi-conductor fabs. Almost a monopoly in lithography. Downside: pretty high PE 36+.

6) Intel - Still does good work designing chips. Huge struggles to manufacture sub 10nm chips in-house (TSM and Samsung have 3nm in production). Committed a ton of fraud. However the US government is wisely trying to increase domestic semis production and Intel is a US Company.

7) Other stocks I'm considering: Microsoft (open AI), Micron (memory play), Meta (just released diplomacy AI, priced attractively right now?), Tesla (good in-house AI).

This is a good article by @asteriskmgzn is quite good. https://asteriskmag.com/issues/1/china-s-silicon-future

Notes:
1) Please be good rationalists and don't argue about the EMH. If you believe in the EMH just don't po...
NEFFA Should Allow Small Children
by jefftk — 2022-11-25T23:00:14.637Z

When I was little we would go to the

NEFFA festival every year. I loved
seeing the dancing: contra dance, international, dance performances.
I loved listening to the music: bands, performances, hallway jamming.
I loved eating the cake the Lithuanian booth would sell. My cousins
would come up from Philadelphia, and it was one of my favorite
weekends of the year.

Those early experiences at NEFFA got me into an activity and a
community that has been a huge part of my life ever since. I've been
a dancer, musician, caller, and helped
organize a dance series and
weekend. I've been
bringing my kids with me when I go early to set up for the dance and
when I travel to play dances
elsewhere, and I'm glad they're getting to enjoy something that's
been so important to me.

Which is why I was so disappointed to learn that the first in-person
NEFFA since 2019 has
chosen a covid policy that excludes small children.

Regrettably, out of concern for everyone's safety, children who for
whatever reason are unable to remain appropriately masked as described
above, including those two years of age or younger, are not permitted
to attend this year's Festival.

When I described this policy to my cousin their response was "I guess
that means we won't be able to go this year." I think this will
likely be a common response among families with small children, and an
unfortunate one.

How seriously organizers and individuals should be trying to limit
covid spread is something there's a lot of disagreement on, mostly
tracing back to different interpretations of the limited amount we
know about the long-term impacts of getting sick. If the NEFFA
organizers want to put on a very covid-cautious event that's not
something I think people should give them grief over, and I'd rather see an
in-person NEFFA for part of the community than no in-person event at
all. But while if the precautions under which they were able to hold
the event were uniformly strict I could see excluding people to young
to mask, they aren't uniformly strict:

There will be indoor dining, people eating and talking with
masks off.

Wind instruments are (likely—there's a case-by-case
component) allowed with same-day testing.

Despite rapid tests now being cheap and
widely available, they don't require them.

They have an exception process for adults who are unable to be
vaccinated for

...
Podcast: Shoshannah Tekofsky on skilling up in AI safety, visiting Berkeley, and developing novel research ideas
by Orpheus16 — 2022-11-25T20:47:09.832Z

TLDR: I interviewed Shoshannah Tekofsky. You can listen to our conversation here or read some highlights below.

Some people in AI safety think that it’s important to have people coming up with fresh, new ideas. There’s a common meme that “thinking for yourself” and “developing your inside view” can be important. But these concepts can feel vague. What does it really mean to “think for yourself” when approaching AI alignment research?

I recently interviewed Shoshannah Tekofsky. For better or for worse, Shoshannah embodies the “think for herself” spirit. After reading List of Lethalities, she started reading about AI alignment (part-time). She then wrote

Naive Hypotheses on AI alignment

, in which she proposes her own “naive ideas” to solve the alignment problem.

In the interview, we discuss Shoshannah’s journey, her approach to starting off in AI alignment research, some stories from her recent trip to Berkeley, and some of the ways she’s currently thinking about alignment research.

You can listen to the full interview

here

. I’ve included some highlights below.

Note: I am considering having more conversations about AI and starting a podcast. Feedback is welcome. Also feel free to reach out if you’re interested in helping (e.g., audio editing, transcript editing, generating questions).

Introduction

Akash: Shoshannah is 36 years old. She has a master's in computer science and ancient technology. And she has a PhD in Player Modeling in video games. Three months ago, Shoshannah started skilling up in AI safety. I read a LessWrong post by her called

Naive hypotheses on AI alignment

. She's spent the last couple of months continuing to skill-up in AI safety, including spending one month in Berkeley, California. I'm excited to talk to Shoshana about how her skilling-up in AI safety has been going, how her experience in Berkeley was, and a bunch of other stuff that will inevitably come up.

Bad advice

Shoshannah: But the advice was absolutely terrible. And I think this is a common way of thinking. He said, “you need to spend about one or two years reading all the existing literature, following up on all the current authorities, and working through that.” And then at the end of that, you might be able to add on to some of the existing frames, or once you've like, written one thing after like two years about, you know, what you thin...