trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost] The AGI Show podcast
by Soroush Pour — 2023-05-23T09:52:29.685Z

This is a linkpost for www.theagishow.com. Also on YouTube at https://www.youtube.com/@theagishow.

Hey everyone, my name is Soroush and I work full-time on AGI alignment.

As part of my work in alignment, I've started a podcast, The AGI Show, to help discuss and promote the best and most important ideas likely to get us to a safe & positive future for humanity with AGI.

In our first episode sequence, we sit down with AI experts to talk timelines to AGI. As an example, we sat down in Episode 4 with Ryan Kupyn, the superforecaster who won Astral Codex Ten's 2022 prediction contest to discuss his timelines to AGI and how he got there.

In our upcoming episode sequence, we'll be talking through the implications of AGI & how we should collectively respond to help ensure a positive future for humanity. We'll be sitting down with technical researchers, governance & policy folks, and others working in the field.

The target audience are people who are fairly technical (e.g. have education or experience equivalent to a technical undergrad degree) but not alignment experts or researchers. As such, we'll always try to explain jargon and simplify concepts to make them accessible a general technical audience.

The podcast could be interesting to you (or a friend) if:

You're a technical person just starting to learn about AGI safety
You're a technical person who wants more ideas on how to apply your time and skills towards AGI safety (coming soon)
You're not super technical, but keen to move one notch up the technical ladder towards an AGI podcast that discusses technical topics but in a fairly accessible fashion

Note: this podcast could be especially good for technical folks outside the LessWrong and broader rationality / AI alignment community, since it doesn't assume any prior interest or exposure to this community's terminology or norms.

Please check it out and provide your honest feedback! I'm keen to make the show as positive as it can towards our future with AGI and as valuable as possible for listeners.

Audio: www.theagishow.com/
Video: https://www.youtube.com/@theagishow
Data and "tokens" a 30 year old human "trains" on
by Jose Miguel Cruz y Celis — 2023-05-23T05:34:30.731Z

I did some calculations with a bunch of assumptions and simplifications but here's a high estimate, back of the envelope calculation for the data and "tokens" a 30 year old human would have "trained" on:

Visual data: 130 million photoreceptor cells, firing at 10 Hz = 1.3Gbits/s = 162.5 MB/s over 30 years (aprox. 946,080,000 seconds) = 153 Petabytes
Auditory data: Humans can hear frequencies up to 20,000 Hz, high quality audio is sampled at 44.1 kHz satisfying Nyquist-Shannon sampling theorem, if we assume a 16bit (cd quality)*2(channels for stereo) = 1.41 Mbits/s = .18 MB/s over 30 years = .167 Petabytes
Tactile data: 4 million touch receptors providing 8 bits/s (assuming they account for temperature, pressure, pain, hair movement, vibration) = 5 MB/s over 30 years = 4.73 Petabytes
Olfactory data: We can detect up to 1 trillion smells , assuming we process 1 smell every second and each smell is represented a its own piece of data i.e. log2(1trillion) = 40 bits/s = 0.0000050 MB/s over 30 years = .000004 Petabytes
Taste data: 10,000 receptors, assuming a unique identifier for each basic taste (sweet, sour, salty, bitter and umami) log2(5) 2.3 bits rounded up to 3 = 30 kbits/s = 0.00375 MB/s over 30 years = .00035 Petabytes
This amounts to 153 + .167 + 4.73 + .000004 + .00035 = 158.64 Petabytes assuming 5 bytes per token (i.e. 5 characters) amounts to 31,728 T tokens
This is of course a high estimate and most of this data will clearly have huge compression capacity, but I wanted to get a rough estimate of a high upper bound. Here's the google sheet if anyone wants to copy it or contribute
Discussion
The motivation for this was chinchilla's wild implications post by nostalgebraist and in general the idea that humans "need" much less data to train on than AI.
According to these calculations humans "train" on a lot of Petabytes, around 5 Petabytes per year which amounts to ~1000 T tokens. Given that we're only training our current models on the order of 1 T tokens, the argument that the human brain is more efficient at learning with less data than current LLMs is not fair. Geoff Hinton changing his mind recently about the efficiency of backpropagation from a less efficient to a much more efficient learning algorithm than what the brain is doing, reinforces this point.
How I learned to stop worrying and love skill trees
by junk heap homotopy — 2023-05-23T04:08:42.022Z

[UPDATE (2023/05/26): fleshed out security analysis section more based on feedback, see Appendix.]

[UPDATE (2023/06/06): more changes to the Appendix.]

There seems to be a stupid, embarrassingly simple solution to the following seemingly unrelated problems:

Upskilling is hard: the available paths are often lonely and uncertain, workshops aren't mass-producing Paul Christianos, and it's hard for people to stay motivated over long periods of time unless they uproot their entire lives and move to London/Berkeley[1].

It takes up to five years for entrants in alignment research to build up their portfolio and do good work–too slow for short timelines.

Alignment researchers don't seem to stack.

LessWrong–and by extension greenfield alignment–is currently teetering on the edge of an Eternal September: most new people are several hundred thousand words of reading away from automatically avoiding bad ideas, let alone being able to discuss them with good truth-seeking norms.

We don't have a reliable way to gauge the potential of someone we've never met to do great work[2].

This is not a new idea. It’s a side project of mine that could be built by your average first-year CS undergrad and that I have shelved multiple times. It's just that, for some reason, like moths to a flame or a dog to its vomit I just keep coming back to it. So I figured, third time’s the charm, right?

The proposal (which I call 'Blackbelt' for obscure reasons) is really simple: a dependency graph of tests of skill.

Note that last bit: 'tests of skill'. If my intention was merely to add to the growing pile of Intro to AI Safety (Please Don't Betray Us and Research Capabilities Afterward)[3] courses out there then we can all just pack up and go home and forget this poorly-worded post ever existed. But alas, my internal model says we will not go from doomed to saved with the nth attempt at prettifying the proof of the rank-nullity theorem. The real problem is not finding better presentations or a better Chatty McTextbook explanation, but can be found by observing what does not change.

That is, let's invert the question of how to produce experts and instead ask: "What things should I be able to do, to be considered a minimum viable expert in X?"

So for instance, since we’re all trying to get more dignity points in before 2028, let’s consider the case of the empirical alignment researcher.

The m...
T-Shirt Size Distribution
by jefftk — 2023-05-23T02:40:01.529Z

Back in August when we put in an order for shirts

we didn't know what sizes
people would want, so I looked around online and made some guesses.
Now that we've sold a bunch of shirts and are looking at putting in
another order I can look at our sales and have a much better idea of
what's likely to sell in the future:

Size
Unisex
Fitted

XS
0
1

S
3
4

M
9
6

L
13
7

XL
9
1

XXL
0
1

XXXL
0
2

Going by sales isn't perfect: in addition to the noise inherent to
small samples sizes there were also a few times when someone wanted to
buy a shirt but we didn't have one in their size, and this was a bit
more common in the more popular sizes.

Here's how well this matches our initial order:

The main issue is we overestimated how many people would want shirts
in the largest sizes, which is especially unfortunate because those
ones tend to have surcharges due to the extra material. But since
we're continuing to order shirts it's not much of an issue: we'll keep
bringing them to dances until someone the right size wants them.

You might be wondering how the "sold" line could possibly be higher
than the "ordered" line? It's because "sold" includes both shirts
we've sold at dances and ones people requested in the Kickstarter and
that will be in the upcoming order, while "ordered" is just the
initial order from last year.

Separately, if you're interested in a Kingfisher shirt in an unusual
color, let me know! It's generally possible to add one-offs to an
order that wouldn't normally be economical.

Comment via: facebook, mastodon
AI self-improvement is possible
by bhauth — 2023-05-23T02:32:08.472Z

document purpose

This document mainly argues that human mental development implies that AI self-improvement from sub-human capabilities is possible.

document structure

For unambiguous referencing, sections are prefixed D: for description, L: for lemma, H: for hypothesis, or A: for argument. Descriptions are statements of facts, categorization, and definitions. Lemmas are linked arguments and conclusions that I have high confidence in. Hypotheses are views that at least some people have. Arguments are lines of reasoning that may or may not be correct.

D:abbreviations

ANN = artificial neural network

RL = reinforcement learning

SI = self-improvement (other than direct RL)

LLM = large language model (using a Transformer architecture)

SDF = signed distance field

D:data

Current LLMs are empirically data-limited, as shown by the "Chinchilla scaling laws" described in this paper.

D:efficiency

Humans are much more data-efficient than current LLMs. LLMs are trained on far more text than humans can read in their lifetime, yet their capabilities are (at least in many ways) much worse than humans. Considering D:data, this implies that humans use superior algorithms.

Humans are, for some tasks, more energy-efficient than current AI systems. A human brain uses only about 20 watts, which is much less than a modern GPU, but humans can do image segmentation faster than current ML models running on a single modern GPU.

H:plateau

Applying intelligence to improving intelligence has diminishing returns, so intelligence that becomes self-improving will plateau at a similar level of intelligence.

A:plateau

We don't see humans or institutions become ultra-intelligent by recursive SI, so we know H:plateau is correct.

Here's Robin Hanson making this argument; that's still his view today.

D:genius

The genetic and morphological changes from early mammals to early apes were much greater than those from early apes to humans, but the absolute increase in intelligence from the latter step was much greater. A few more changes make the difference between an average human and the very smartest humans. Around human-level intelligence, returns do not seem to be diminishing.

H:human_SI

D:genius happens because small improvements in genetically-specified neural systems are amplified as those initial systems generate higher-level systems or perhaps recursively self-improve.

H:human_hyperparamete

...
Worrying less about acausal extortion
by Raemon — 2023-05-23T02:08:18.900Z

I've wanted a good version of this post to exist for awhile. I'm not sure how to write that post. But here's a okay-ish first draft, which seemed hopefully better than nothing. I've made it possible for anyone to write comments on the LessWrong doc editor version of the post, so if you have ideas for how to improve it you can leave line-notes here.

Once a month or so, the Lesswrong mods get a new user who's worried about Roko's basilisk, or other forms of acausal extortion. They write a post discussing their worries. In the past, they'd often get comments explaining "Here's why you don't need to be worried about acausal exortion", but then respond with "but, what about this edge case you didn't specifically address?".

And this is a fairly sad situation – acausal trade/exortion is a bit confusing, and there are few places on the internet where people are knowledgeable enough to really explain it. But, my experience from talking to people worried about this is that no amount of explanation is really satisfying. They're left with a lingering "but, what if I didn't understand this properly and I'm still vulnerable?" and they keep looking to understand things better.

My impression is that often, they're caught in an anxiety loop that wants to reinforce itself, and the act of worrying about it mostly encourages that anxiety loop to keep going rather than reassure it. I don't think the winning move here looks like "thoroughly understand the situation until you are no longer anxious". I think the winning move is more like "find something else to think about that is interesting so that the loop can wind itself down" or "I dunno, go outside and get some fresh air and take some deep breaths."

I don't actually know what empirically works. (I'm interested in comments from anyone who was worried about acausal extortion and who eventually stopped being worried, who can talk about their experience)

It seemed good to have an FAQ that at least covered some basics.

So:

Should I worry about Roko's Basilisk or other acausal extortion stuff?

My short answer is "no", but I want to acknowledge this is a kind of tricky epistemic puzzle for the sort of person I'm imagining reading this post.

If you've been think about acausal extortion for over a week and a really distressed about it, I think you should have a few hypotheses, which might include:

Acausal ext

...
Self-leadership and self-love dissolve anger and trauma
by Richard_Ngo — 2023-05-22T22:30:06.650Z

I find the “bids and boundaries” framework from my previous post to be very useful, but also a little… sterile, perhaps. Imagine prefixing every request you ever make with “it’s okay to say no, but…”; or always refusing bids which you think you might come to resent; or always asking people if it’s okay to flirt with them before doing it. That makes relationships less risky, but also shallower and less spontaneous.[1] Can we get the best of both worlds? I think so, via the two skills I discuss in this post: self-leadership and self-love.

These terms have straightforward meanings, but also more specific connotations in the IFS framework. In IFS, the “Self” is the aspect of you which mediates between other parts, and which is present when those parts aren’t active. Self-leadership is the ability to consistently engage with the world in a way that all your parts endorse, rather than letting some of them seize control in ways that others will later resent. It’s a difficult skill because when somebody hurts you, the hurt part often feels a strong drive to express itself, and will only let you intervene if it trusts that you’ll treat it fairly, and defend its interests, even when it’s less emotionally activated.

Having said that, self-leadership doesn’t mean never getting angry—it just means never fully giving in to that anger or wielding it with the goal of hurting another person (or another part of yourself). Self-leadership might involve telling the other person that you feel angry at them, but without launching into a tirade; or telling them that you need to go on a walk to calm down, but giving them a reassuring gesture before you leave. In other words, self-leadership means that while your angry parts can’t seize full control, neither can the parts of you that want to suppress your anger. And on the receiving end, it involves being able to face someone who’s angry at you without either retreating into yourself or lashing out in response.

For some readers, expressing anger at all might be scary. But suppressing anger doesn’t make it go away; once something starts to bug you, even small examples can trigger the underlying feeling of hurt. Eventually you end up with a conversation like:

A: [finally hitting their breaking point] “I can’t stand how you never treat me with respect.”

B: “What the hell? I was just doing [X]; you’re massively overreacting.”

A

...
A Manifold market notice: Binance
by Scrooge Mcduck — 2023-05-22T22:24:07.076Z

Probably someone on LessWrong has money on Binance, being the largest cryptocurrency exchange. I'd like to draw their attention to the crime risk, and suggest they move the money elsewhere.

The CFTC has a lawsuit against both Binance and its CEO, Changpeng Zhao ("CZ"). Whatever your general views on regulating crypto, this specific situation has a practical risk to investors. Because further than just a CFTC lawsuit, my market gives high odds of criminal charges against CZ. Not just higher than the U.S. population, but actually higher than other crypto project managers.

I didn't want to prematurely declare CZ a danger to investors. So I waited until the market efficiency grew with more traders. At this time, the market gives a persistent ~70% chance he'll get criminal charges. I know it's "only" a playmoney prediction market. But at 42 traders and fairly unambiguous criteria, unless you have some special insight into his legal situation, it's rational to defer to the market.

This doesn't necessarily mean we'll see FTX-scale losses for investors. Suppose if he gets criminal charges but the exchange continues to operate, changes leadership, and gets regulated. In that scenario, investors wouldn't necessarily lose money. But the money could easily become frozen, stolen, or depreciate by large amounts. Since this ~70% is a broad aggregation of legal risks that could possibly include things like fraud, this should make you reluctant to do business with him.

Please disregard my small "NO" position on the market. I habitually place those on newly-made crime markets. Partly because most people in my markets will never be charged of anything, so that helps push it toward the base rate. And partly as a gesture of goodwill toward the people mentioned in the markets.

Consider dropping Binance as an exchange you would do business with.

Update 2023-Jun-11:

Per feedback from simon and Dagon, I've made a new series. The criteria is more relevant. Though I prefer seeing 5+ traders for reliability, this post's main subject (Binance) has 10+ now, so I feel comfortable including it. For comparison, I will include the other largest exchanges.

I'm sure there...
I don't want to talk about AI
by KirstenH — 2023-05-22T21:23:28.841Z
Link: https://ealifestyles.substack.com/p/i-dont-want-to-talk-about-ai

This is a post I wrote for my blog, EA Lifestyles.

like any good ea, I try to have a

scout mindset

. I don’t like to lie to myself or to other people. I try to be open to changing my mind.

but that kind of intellectual honesty only works if you don’t get punished for being honest. in order to think clearly about a topic, all of the potential outcomes have to be okay - otherwise you just end up with mental gymnastics trying to come up with the answer you want.

and in some cases - like existential risks from ai - none of the potential outcomes of thinking deeply about it are especially attractive.

if I look into ai safety and come to believe the world’s going to end in ten years, that would be super depressing, especially if I can’t do much to contribute. so that outcome, while tolerable, isn’t especially attractive.

you might think, “but maybe you’ll figure out that ai isn’t really a risk after all! wouldn’t that be reassuring?”

let’s think through what’s going to happen if I investigate ai safety and realize it’s not that important:

a lot of arguments. people in ea want to hear ideas that are different to their own - it’s a credit to their epistemic humility! I encountered this a lot when I first got into ea in 2017. I remember being cornered at a new year’s eve party, being interrogated in detail on my views on ai safety. it convinced me the guy I was talking to cares deeply about being correct about ai, but it wasn’t a lot of fun.
a lot of re-education. after hearing my views, I’m guessing people will want to share their own. at church, when my views deviated from the norm, I would usually get a lot of interested questions followed by the same 2 or 3 recommended readings. I’d expect the same thing here.
a lot of judgment. I’m not a scientist. if I tried to form my own view on ai safety, it might be really stupid. or it might just be kind of weird. either way, there would probably be at least a few people who would think less of me.
and maybe - MAYBE - I could convince other people to shift their resources to something else. if I were right, that would be very positive; if I were wrong, it would be very negative. but as social sciences major, my chances of being both right and persuasive on ai safety seem astronomically low.

if I wanted the best possible outcome for me personally, I’d just memorize two or three sentences from one of Ajeya or Holden’s blog posts, and qu...
Activation additions in a small residual network
by Garrett Baker — 2023-05-22T20:28:41.264Z

Abstract

Team Shard's recent activation addition methodology for steering GPT-2 XL provokes many questions about what the structure of the internal model computation must be in order for their edits to work. Unfortunately, interpreting the insides of neural networks is known to be very difficult, so the question becomes: what is the minimal set of properties a network must have in order for adding activation additions to work?

Previously, I have tried to make some progress on this question by analyzing whether number additions work for a 784-512-512-10 fully connected MNIST network I had laying around. They didn't. Generalization was destroyed, going from a loss of 0.089 for the unpatched network to an average loss of 7.4 for the modified network. Now I see whether additions work in a residual network I trained. Here's the code for the network:

class ResidualBlock(nn.Module):
def __init__(self, dim):
super(ResidualBlock, self).__init__()
self.linear = nn.Sequential(
nn.Linear(dim, dim)
)
self.relu = nn.ReLU()

def forward(self, x):
out = self.linear(x)
out += x
out = self.relu(out)
return out

# Define the ResNet
class ResNet(nn.Module):
def __init__(self, num_classes=10):
super(ResNet, self).__init__()
self.layer1 = nn.Linear(28*28, 512)
self.relu = nn.ReLU()
self.layer2 = ResidualBlock(512)
self.layer3 = nn.Linear(512, num_classes)

def forward(self, x, x_vector=None, return_midlayer_activation=False):
x = x.view(x.size(0), -1) # equivalent of doing einops.rearrange(x, 'b h w d -> b (h w d)')
x = self.layer1(x)
x = self.relu(x)
x = self.layer2(x)
if x_vector is not None: x += x_vector
activation = x
x = self.layer3(x)
if return_midlayer_activation:
return x, activation
else:
return x

Again, space is left for the interested reader to think of their predictions to themselves.

I found that when you add an a - b activation vector (using the x_vector functionality above), generalization is again destroyed, going from an average loss of 0.086 in the unmodified network to 6.31 in the patched network.

Methodology[1]

Colab notebook here

In this research, I implemented and employed a Resid...