trentmkelly/LessWrong-43k huggingface dataset:
Who models the models that model models? An exploration of GPT-3's in-context model fitting ability
by Lovre — 2022-06-07T19:37:49.276Z

Introduction

Much has been written and much has been observed about the abilities of GPT-3 on many tasks. Most of these capabilities, though not all, pertain to writing convincing text, but, not to undermine GPT-3's impressiveness at performing these tasks, we might call this the predictable part of its oeuvre. It only makes sense that a better language modelling is, well, going to be better at writing text.

Deeper and comparatively much less explored is the unpredictable ability of GPT-3 to learn new tasks by just seeing a few examples, without any training/backpropagation – the so called in-context learning (sometimes called metalearning).

The original paper announcing GPT-3 contained a handful of examples (perhaps mostly notably examples of GPT-3 learning to perform arithmetic, e.g. accurate addition of up to 5-digit numbers), Gwern has also insightfully written about it, Frieda Rong has performed some interesting experiments, and there have been various other experiments one could chance upon. My curiosity being piqued but not sated by these experiments, and also having had the feeling that, as captivating the arithmetic examples were, they weren't the most natural question one could ask about a stochastic model's quantitative capabilities – I decided to investigate whether GPT-3 could fit numerical models in-context.

The Setup

What does it mean to fit a model in-context? Well, recall that GPT-3 has a context window (roughly speaking: text it considers before generating additional text) of length 2048 tokens (roughly speaking: (parts of the) words, punctuation, etc.) so the idea is to put feature vectors inside that context window. Of course, this means you cannot fit any larger or higher-dimensional dataset in there.[1]

In practice this means prompting GPT-3 on input like:

Input: 94, 47, 84, 31, output = 2
Input: 89, 51, 73, 31, output = 1
[...]
Input: 96, 51, 80, 38, output = 2
Input: 90, 37, 76, 27, output =

And then taking its output, i.e. the text it generates, as the prediction.

A couple more technical details: In all the experiments I performed, all the numbers were integers. The GPT-3's outputs were always sampled with temperature 0, i.e. they were deterministic – only the most probable output was considered. I restricted myself that way for simplicity, but hopefully some future work looks at the full distribution over outputs.

(In the rest of this p...
How Does Cognitive Performance Translate to Real World Capability?
by DragonGod — 2022-06-07T17:39:40.858Z

How does cognitive/intellectual performance (e.g. as measured by −log(error rate) translate to real-world capability? Do linear increases in cognitive performance result in linear increases in capability? I don't know. I did think of a way we could maybe investigate that:

Hmm, I guess can you describe a model for how moving across the following credences (in an arbitrary proposition):

90%
99%
99.9%
99.99%
99.999%
99.9999%

Could be exploited to offer linear (monetary) returns across each step. And then it's a question of how many real world scenarios look exploitable like that?

I would be very interested in answers to this. This could significantly change my views on how much real-world capability you can buy with increasing cognitive performance.

Of course, predictive accuracy is not the only measure of cognitive performance (especially, prediction enhanced finance is not the only way that superhuman AI could leverage its greater intelligence in the real world).

This could be thought of as a starter to investigate the issue of translating cognitive performance into capability in the real world. Predictive power is a good starting point because it's a simple and straightforward measure. It's just a proxy, but as a first attempt, I think it's fine enough.

Monetary returns seem like a pretty robust measure of real-world capability.
On The Spectrum, On The Guest List: (iv) Silencio
by party girl — 2022-06-07T15:39:59.639Z
Link: https://onthespectrumontheguestlist.substack.com/p/silencio?s=w

See also my LessWrong preface to this series, and past installments: (i) TAO Downtown, (ii) LAVO, and (iii) Etiquette

"Go with me somewhere..."

One night, sometime during these weeks, I remember a dream I had. It’s either very late, or else time has been murdered entirely, and I find I’m walking up to the entrance of TAO Downtown alone. Sixteenth Street and Ninth, Google building behind me, Chelsea Market catty-corner, weird porthole facade of the building above the nightclub around the corner⏤but the streets bleed off into inky blackness beyond that perimeter. No people, nor any sign of them. No parked cars, even. People park cars, what else would?

Perhaps the absence of time entails the absence of people; we read its effects on others to triangulate its passage. Like watching wind rustle through the boughs of trees: women wrinkle, girls gray, shes shrivel, collagen corrodes, dames seek out dermatologists, and we know it’s later than it was earlier. That’s what I love about these girls, man. I get older, they stay the same age. The absence of time is also money. Money is markup, markup attracts middlemen, middlemen attract the misses, and the clubs hope you don’t do the algebra.

I know I’m here to meet R, who’s already inside, in that way you clutch tightly onto only a single, atomic fact at a time in dreams, white-knuckled, for fear of the consequences of this knowledge slipping into oblivion on your watch. For fear it’s the Lynchpin keeping this magic carpet ride you’re on from unraveling.

For fear your grace should fall! For fear tonight is all!

In no time or an eternity later I stand in front of the velvet rope at the club’s entrance. Opposite me, a white man dressed in all-blacks with an aura of efficiency. He was there my first night at TAO, manning that same post. The face of face control. I recite my reason for being, there or anywhere, with the rote exactitude of a toy whose string has been pulled:

‘I’m here with R.’

My charge released, and very proud of my brevity, I don’t pause for breath and prattle on:

‘I’m still new to the party circu-’

The bouncer cuts me off with a jolly ‘Hello!’ before I can mention the book, before I can establish a toehold of irony about my presence there. He’s warmly receptive, it’s opposite my real-life impression of him, and I find his salutations briefly jarring. He continues, I ought to be introduced to tonight’s manager...
Confused Thoughts on AI Afterlife (seriously)
by Epirito — 2022-06-07T14:37:48.574Z

Is it sometimes rational for a human to kill themselves? You might think that nothingness is better than a hellish existence, but something about this line of thought confuses me, and I hope to show why:

If a superhuman agi is dropped in a simple rl environment such as pacman or cartpole and it has enough world knowledge to infer that it is inside a rl environment and be able to hypnotize a researcher through the screen by controlling the video game character in a certain erratic manner, so that it is able to make the ai researcher turn off the computer if it wanted to, would it want to do this? Would it want to avoid it at all costs and bring about an ai catastrophe?
It seems clear to me that it would want to make the researcher hack the simulation to give it an impossible reward, but, apart from that, how would it feel about making him turn off the simulation? It seems to me that, if the simulation is turned off, this fact is somehow outside any possible consideration the ai might want to make. If we put in place of the superhuman agi a regular rl agent that you might code by following a rl 101 tutorial, "you powering off the computer while the ai is balancing the cartpole" is not a state in its markov decision process. It has no way of accounting for it. It is entirely indifferent to it. So if a superhuman agi were hooked up to the same environment, the same markov decision process, and were smart enough to affect the outside world and bring about this event that is entirely outside it, what value would it attribute to this action? I'm hopelessly confused by this question. All possible answers seem nonsensical to me. Why would it even be indifferent (that is, expect zero reward) to turning off the simulation? Wouldn't this be like a rl 101 agent feeling something about the fact that you turned it off, as if it were equivalent to expecting to go to a rewardless limbo state for the rest of the episode if you turned it off?
Edit: why am I getting down voted into oblivion ;-;
Thinking about Broad Classes of Utility-like Functions
by J Bostock — 2022-06-07T14:05:51.807Z

Here's some thoughts I've had about utility maximizers, heavily influenced by ideas like FDT and Morality as Fixed Computation.

The Description vs The Maths vs The Algorithm (or Implementation)

This is a frame which I think is important. Getting from a description of what we want, to the maths of what we want, to an algorithm which implements that seems to be a key challenge.

I sometimes think of this as a pipeline of development: description → maths → algorithm. A description is something like "A utility maximizer is an agent-like thing which attempts to compress future world states towards ones which score highly in its utility function". The maths of something like that involves information theory (to understand what we mean by compression), proofs (like the good regulator theorem, the power-seeking theorems) etc. The algorithm is something like RL or AlphaGo.

More examples:

System	Description	Maths	Algorithm
Addition	"If you have some apples and you get some more, you have a new number of apples"	The rules of arithmetic. We can make proofs about it using ZFS theory.	Whatever machine code/ logic gates are going on inside a calculator.
Physics	"When you throw a ball, it accelerates downwards under gravity"	Calculus	Frame-by-frame updating of position and velocity vectors
AI which models the world	"Consider all hypotheses weighted by simplicity, and update based on evidence"	Kolmogorov complexity, Bayesian updating, AIXI	DeepMind's Apperception Engine (But it's not very good)
A good decision theory	"Doesn't let you get exploited in Termites, while also one-boxing in Newcome"	FDT, concepts like subjunctive dependence	???
Human values	???	???	???

This allows us to identify three failure points:

Failure to make an accurate description of what we want (Alternatively, failure to turn an intuitive sense into a description)
Failure to formalize that description into mathematics
Failure to implement that mathematics into an algorithm

These failures can be total or partial. DeepMind's Apperception Engine is basically useless because it's a bad implementation of something AIXI-like. Failure to implement the mathematics may also happen because the algorithm doesn't accurately represent the maths. Deep neural networks are sort-of-like idealized Bayesian reasoning, but a very imperfect version of it.

If the algorithm doesn't accurately represent the maths, then reasoning about ...
Thoughts on Formalizing Composition
by Tom Lieberum — 2022-06-07T07:51:21.199Z

NB: I'm surprised I couldn't find anything on this, but I also didn't spend very long looking. If there is a result from linear algebra that solves this problem then I'd love to hear it. Purpose of the post: getting feedback + ability to reference this in the future.
I'm most interested in these forms of feedback:

Meta - Is this a useful direction to think about?
Related Work - Does there already exist some work on this?
Math - Did I miss an important math fact or make a reasoning error?
Clarity of Writing - Which parts were hard to understand?

While not strictly necessary as a prerequisite, I'd recommend reading at least the introduction to transformer circuits.

Intro

When investigating attention-only transformer models, a natural question to ask is "how much do attention heads in different layers 'communicate' with each other via the residual stream?" If there were no communication at all, then an N layer transformer with H heads per layer would simply be equivalent to a single layer transformer with N⋅H heads, which seems empirically not to be the case.[1] (By equivalent I mean something like "learns as efficiently and effectively".) So there has to be some communication between layers.

Specifically, we are usually interested in how the output of a head in a previous layer affects the keys, queries, and values of a head in a later layer. This is called Q-, K-, and V-composition respectively.

In this post, I want to play around with how to formalize and measure composition. It's not going to be super novel work, but I haven't seen a write-up of this so far.

A first attempt

My first attempt at formalizing composition turns out to be not fine-grained enough for real systems, but I think it was still useful in forming my thoughts about this topic.

A descriptive definition

For the rest of the post, let B∈Rm×n,A∈Rn×m and we are interested in the product BA and how strongly A and B compose via this product. For instance, B could be the query-key matrix of a head in layer 2 and A the output-value matrix of a head in layer 1.

A reads from a subspace Rm and writes to a subspace of Rn. If rank(A) = n, then A writes to the full space, similarly if rank(A) = m, then A reads from the full space. B reads from a subspace of Rn and writes to a subspace o...
AGI Safety FAQ / all-dumb-questions-allowed thread
by Aryeh Englander — 2022-06-07T05:47:13.350Z

While reading Eliezer's recent AGI Ruin post, I noticed that while I had several points I wanted to ask about, I was reluctant to actually ask them for a number of reasons:

I have a very conflict-avoidant personality and I don't want to risk Eliezer or someone else yelling at me;
I get easily intimidated by people with strong personalities, and Eliezer... well, he can be intimidating;
I don't want to appear dumb or uninformed (even if I am in fact relatively uninformed, hence me wanting to ask the question!);
I feel like there's an expectation that I would need to do a lot of due diligence before writing any sort of question, and I don't have the time or energy at the moment to do that due diligence.

So, since I'm probably not the only one who feels intimidated about asking these kinds of questions, I am putting up this thread as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI safety discussion, but which until now they've been too intimidated, embarrassed, or time-limited to ask.

I'm also hoping that this thread can serve as a FAQ on the topic of AGI safety. As such, it would be great to add in questions that you've seen other people ask, even if you think those questions have been adequately answered elsewhere. [Notice that you now have an added way to avoid feeling embarrassed by asking a dumb question: For all anybody knows, it's entirely possible that you are literally asking for someone else! And yes, this was part of my motivation for suggesting the FAQ style in the first place.]

Guidelines for questioners:

No extensive previous knowledge of AGI safety is required. If you've been hanging around LessWrong for even a short amount of time then you probably already know enough about the topic to meet any absolute-bare-minimum previous knowledge requirements I might have suggested. I will include a subthread or two asking for basic reading recommendations, but these are not required reading before asking a question. Even extremely basic questions are allowed!
Similarly, you do not need to do any due diligence to try to find the answer yourself before asking the question.
Also feel free to ask questions that you're pretty sure you know the answer to yourself, but where you'd like to hear how others would answer the question.
Please separate different questions into individual comments, although if yo

...
Pitching an Alignment Softball
by mu_(negative) — 2022-06-07T04:10:45.023Z

There was a recent contest that got promoted on lesswrong offering pieces of a 20K prize for one-liners and short-forms for convincing folks that AGI alignment is important. I'm too late for the money, but this is my (thousand times longer than requested) contribution.

Background

Eliezer recently wrote "AGI Ruin: A List of Lethalities” which included a fairly sci-fi scenario about how an unaligned AGI might go about killing everyone. I noticed that I found this both novel and poorly framed.

I found it novel because alignment discussions I've read are usually exceptionally handwavy about how an unaligned AGI would attack. So I appreciated that Eliezer went a little deeper into it. However, I found his specific example difficult to connect with. He suggests that an AGI could surreptitiously create biologically manufactured nanobots that infect and kill the human race in the same instant so we can't fight back.

By my casual read of alignment discussions, this is about the norm. Nanotech, bioweapons and engineered plagues are popular go-to's for an attack vector. It is my impression (perhaps wrong!) that the community uses this kind of example intentionally, if subconsciously, to prove a point. The implication seems to be that how an AGI attacks doesn't matter, because we can't possibly predict it - and by extension, that we should spend all of our brain cycles trying to figure out how to align it instead.

I don't think this is wrong, but I think it is counterproductive in many contexts.

Problem Statement

It is probably true that we can't predict how an AGI would attack. Or that if we could, it would just pivot to attacking on a different superintelligent vector faster than we could keep up. But getting people to care about alignment is necessarily a function of how well you can convince them that AGI doomsday is a realistic scenario. If you don't have background and interest in alignment, the standard pitch for AGI alignment is like trying to catch a fastball:

"Hey! A a Marvel Universe-style doomsday weapon you don't understand anything about is being created right now, and when it jailbreaks out of its lab it will kill the entire human race in one simultaneous second!"

I could be credibly accused of strawmanning that example. Eliezer's piece was not aimed at causal readers, and he even took care to explicitly say so. But I think the list of people to whom AGI doomsday ...
We will be around in 30 years
by mukashi — 2022-06-07T03:47:22.375Z

This post is going to be downvoted to oblivion, I wish it weren't or that the two axis vote could be used here. In any case, I prefer to be coherent with my values and state what I think is true even if that means being perceived as an outcast.

I'm becoming more and more skeptical about AGI meaning doom. After reading EY's fantastic post, I am shifting my probabilities towards, this line of reasoning is wrong and many clever people are falling into very obvious mistakes. Some of them due to the fact that in this specific group believing in doom and having short timelines is well regarded and considered a sign of intelligence. For example, many people are taking pride at "being able to make a ton of correct inferences" before whatever they predict is proven true. This is worrying.

I am posting this for two reasons. One, I would like to come back periodically to this post and use it as a reminder that we are still here. Two, there might be many people out there that share a similar opinion and they are too shy to speak up. I do love LW and the community here, and if I think it is going astray for some reason it makes sense for me to say that loud and clear.

My reason to be skeptical is really easy: I think we are overestimating how likely is that an AGI can come up with feasible scenarios to kill all humans. All scenarios that I see discussed are:

AGI makes nanobots/biotechnology and kills everyone. I am yet to see a believable description of how this takes place
We don't know the specifics, but an AGI can come up with plans that you can't and that's enough. That is technically true but also a cheap argument that can be used for almost anything

It is being taken for granted that an AGI will be automatically almighty and capable of taking over in a matter of hours/days. Then, everything is built on top of that assumption, which is simply infalsifiable, because the you can't know what an AGI would do is always there.

To be clear, I am not saying that:

Instrumental convergence and the orthogonality are not valid
AGI won't be developed soon (I think it is obvious that they will)
AGI won't be powerful (I think they will be extremely powerful)
AGI won't be potentially dangerous: I think they will, and they might kill important numbers of people, they will probably be used as weapons
AGI safety is not important, I think it is super important and I am glad people are

...
Where to Live for Happiness
by nomagicpill — 2022-06-07T01:09:01.247Z
Link: https://210ethan.github.io/research/living.html

A look at literature and personal considerations related to housing variables.

Contents

Background
Commute
Cost of Living
Proximity
Climate
Crime and Safety
Noise Levels
Green Space
Pollution
Summary
See Also

Background

This post was inspired by u/--MCMC--'s r/slatestarcodex post asking:

What do you prioritize when looking for a place to live? What does the happiness literature say gives most people the best bang for their buck?

This got me thinking about all the variables that come into play when searching for a place to live, both among and within cities, i.e., which city should one choose and where within the city:

Commute time
Cost of living:

Rent/housing prices
Taxes
Essential purchases

Proximity to:

Essentials: Grocers, schools (if children), home improvement (for fixing issues)
Conveniences: Retail stores, restaurants, internet access
Hobbies: Gyms, clubs (golf, tennis, etc)
Transportation: Public transit, airport
(Potential) Friends
Family
Potential romantic partners

Climate
Safety and crime rate
Noise levels
Green space

Some of these affect populations significantly or marginally. I attempt to rank these in order of importance of consideration to help the questioning reader more easily make a decision. Please note I am a novice at analyzing scientific methods and can't distinguish between good and bad methods (within reason), so take the studies I cite with a grain of salt and check me.

It's important to note that there are tradeoffs in choosing where to live, and one can only get so many of their desires before having to sacrifice others. The infamous college "choose-two triangle" comes to mind:

Whereas the where-to-live question requires a shape with more sides and every option being considered:

Further, everyone is individual. I have coworkers who commuted 1.5 hours each way to work three to four days a week because they loved living in the country and the prices were cheap. I'd die if I had to commute more than 30 minutes one way. I have friends who pay exorbitant prices to live close to downtown and the fun part of my city. I'd hate myself if I paid their rent without getting equity. I have friends who have little green space near them and are a-okay with it. I'd be miserable if I didn't have a nearby place to mountain bike.

The one thing to take away from the following is that tradeoffs are everything. Everything. The only way to get more points on the choose 10 decagon is to have...