trentmkelly/LessWrong-43k huggingface dataset:
When Giving People Money Doesn’t Help
by Zvi — 2022-07-07T13:00:00.879Z

I’ve been thinking a bunch about this study (direct link) of CBT-style techniques used to try and get criminals onto a better life path in Liberia that showed what looks like spectacular results, curious if anyone wants to share thoughts. If it would replicate seems like it crosses the EA-worthiness threshold.

I’ve been thinking more about the topic of this post, which is the cash transfer study in Chicago. That worked out less well. It seems that small cash transfers to those in need here in America made them feel worse and maybe even actually makes them worse off somehow? (paper)

This prediction seems very reasonable, and I think I would have agreed with it if I’d thought about this in advance. My model would have said that obviously cash is good and cash is helpful, in the short term getting some of it will be helpful. In the long term I’d expect some positive impact from the wealth effect. And in some cases, that money will have a particular jam that it solves or opportunity it enables (e.g. fix your car, security deposit on a new apartment, maybe even give opportunity to start a small business). In those cases it would have a permanent impact, so I would predict some overall positive impact. But in my experience there are enough implicit taxes and poverty traps for such people, and such a cultural lack of the idea of hanging onto money, that it is tough to hang onto funds, so mostly they’d be back where they started.

It looks like within about three weeks the money was mostly gone for both groups. That is not what would happen for an economically rational agent, unless they had a lot of opportunities for very high return on capital or they were paying down debt.

Elizabeth had a similar reaction to mine.

There’s something very interesting in this graph. The three groups have completely converged by the end of the 180 day period, but the average bank balance is now considerably higher. They all started out with ~$200 and now they have ~$400. That feels like a meaningfully different average amount.

This happened during Covid, so I am guessing that this is due to the Covid checks or other similar factors. This certainly complicates the experiment.

Those are pretty different things to happen. If you spend the money on bills, that’s paying down debt. It makes sense that this should happen right away. Spendi...
Wealth as a source of technological stagnation?
by alyssavance — 2022-07-07T05:46:01.663Z

(This is an essay I wrote last year, but hadn't published. I'm posting it here because some friends liked it and there's no particular reason to keep it private.)

The World Values Survey is a very comprehensive set of social science data that asks people, broadly, what they care about and how they think about life. It goes back forty years to 1981, and covers dozens of different countries.

The survey includes hundreds of questions, but the data analysts break responses down along two main axes, "traditional vs. secular/rational values" and "survival vs. self-expression values":

"Societies that emphasize survival values have relatively low levels of subjective well-being, report relatively poor health, and are low on interpersonal trust, relatively intolerant of outgroups, and low on support for gender equality. They emphasize materialist values, have relatively high levels of faith in science and technology, and are relatively low on environmental activism and relatively favorable to authoritarian government. Societies that rank high on self-expression values tend to have the opposite preferences on all of these topics. Overall, self-expression values reflect an emancipative and humanistic ethos, emphasizing human autonomy and choice." (source)

Broadly speaking, the first category seems like "the past" and the second category seems like "the present/future"... but "high levels of faith in science and technology"? Aren't science and technology supposed to be futuristic?

There's been a lot of discussion about whether innovation has slowed down or become less efficient, from Tyler Cowen's The Great Stagnation, to Robert Gordon's The Rise and Fall of American Growth, to Jason Crawford's ongoing blog The Roots of Progress. Particular causes might include things like Goodhart's Law, where any metric that is optimized ceases to be a good metric; governments, companies and universities getting larger, which makes them more bureaucratic and less focused; "institutional decay", where an organization like NASA is most effective right after its creation, and declines afterwards; and Nash equilibrium traps building up over time, where, eg., everyone knows that p < 0.05 is a lousy standard for publication, but everyone has to use it because everyone else does.

The least-cynical theory is just that discoveries have gradually "run out" - where there's a certain amount of disco...
Race Along Rashomon Ridge
by Stephen Fowler — 2022-07-07T03:20:59.701Z

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2022 Research Sprint Under John Wentworth

Two Deep Neural Networks with wildly different parameters can produce equally good results. Not only can a tweak to parameters leave performance unchanged, but in many cases, two neural networks with completely different weights and biases produce identical outputs for any input.

The motivating question:
Given two optimal models in a neural network's weight space, is it possible to find a path between them comprised entirely of other optimal models?

In other words, can we find a continuous path of tweaks from the first optimal model to the second without reducing performance at any point in the process?

Ultimately, we hope that the study of equivalently optimal models would lead to advances in interpretability: for example, by producing models that are simultaneously optimal and interpretable.

Introduction

Your friend has invited you to go hiking in the Swiss Alps. Despite extensive planning, you completely failed to clarify which mountain you were supposed to climb. So now you're both currently standing on different mountaintops Communicating via yodels, your friend requests that you hike over to the mountain he's already on; after all, he was the one who organised the trip.

How should you reach your friend? A straight line path would certainly work, but it's such a hassle. Hiking all the way back down and then up another mountain? Madness.

Looking at your map, you notice there's a long ridge that leaves your mountain at roughly the same height. Does this ridge lead to reach your friend's mountain and is it possible to follow this ridge to reach your friend?

Similarly, we can envision two optimal models in a neural net's weight space that share a "ridge" in the loss landscape. There is a natural intuition that we should be able to "walk" between these two points without leaving the ridge. Does this intuition hold up? And what of two neural networks some distance apart in the loss landscape that have both gotten close to the global minima? Can we find a way to walk from one to the other without leaving the ridge?

To begin our investigation into the structure of these "ridges" we viewed these structures as comprising the Rashomon manifold, the subset of the weight space comprised of models that minimize the loss function. In the 2018 ...
Principles for Alignment/Agency Projects
by johnswentworth — 2022-07-07T02:07:36.156Z

"John, what do you think of this idea for an alignment research project?"

I get questions like that fairly regularly. How do I go about answering? What principles guide my evaluation? Not all of my intuitions for what makes a project valuable can easily be made legible, but I think the principles in this post capture about 80% of the value.

Tackle the Hamming Problems, Don't Avoid Them

Far and away the most common failure mode among self-identifying alignment researchers is to look for Clever Ways To Avoid Doing Hard Things (or Clever Reasons To Ignore The Hard Things), rather than just Directly Tackling The Hard Things.

The most common pattern along these lines is to propose outsourcing the Hard Parts to some future AI, and "just" try to align that AI without understanding the Hard Parts of alignment ourselves. The next most common pattern is to argue that, since Hard Parts are Hard, we definitely don't have enough time to solve them and should therefore pretend that we're going to solve alignment while ignoring them. Third most common is to go into field building, in hopes of getting someone else to solve the Hard Parts. (Admittedly these are not the most charitable summaries.)

There is value in seeing how dumb ideas fail. Most of that value is figuring out what the Hard Parts of the problem are - the taut constraints which we run into over and over again, which we have no idea how to solve. (If it seems pretty solvable, it's probably not a Hard Part.) Once you can recognize the Hard Parts well enough to try to avoid them, you're already past the point where trying dumb ideas has much value.

On a sufficiently new problem, there is also value in checking dumb ideas just in case the problem happens to be easy. Alignment is already past that point; it's not easy.

You can save yourself several years of time and effort by actively trying to identify the Hard Parts and focus on them, rather than avoid them. Otherwise, you'll end up burning several years on ideas which don't actually leave the field better off. That's one of the big problems with trying to circumvent the Hard Parts: when the circumvention inevitably fails, we are still no closer to solving the Hard Parts. (It has been observed both that alignment researchers mostly seem to not be tackling the Hard Parts, and that alignment research mostly doesn't seem to build on itself; I claim that the latter is a res...
Confusions in My Model of AI Risk
by peterbarnett — 2022-07-07T01:05:01.586Z

A lot of

the reason I am worried about AI

comes from the development of optimizers that have goals which don’t align with what humans want. However, I am also pretty confused about the specifics here, especially core questions like “what actually do we mean by optimizers?” and “are these optimizers actually likely to develop?”. This means that much of my thinking and language when talking about AI risk is fuzzier than I would like.

This confusion about optimization seems to run deep, and I have a vague feeling that the risk paradigm of “learning an optimizer which doesn’t do what we want” is likely confused and somewhat misleading.

What actually is optimization?

In

my story of AI risk

I used the term ‘optimization’ a lot, and I think it’s a very slippery term. I’m not entirely sure what it means for something to ‘do optimization’, but the term does seem to be pointing at something important and real.

A definition from

The Ground of Optimization

says an optimizing system takes something from a wide set of states to a smaller set of states and is robust to perturbations during this process. Training a neural network with gradient descent is an optimization process under this definition because we could start with a wide range of initial network configurations, and the network is modified to be in one of the few configurations which do well on the training distribution, and even if we add a (reasonable) perturbation the weights will still converge. I think this is a good definition, but it is entirely defined in terms of behavior rather than a mechanistic process. Additionally, it doesn’t really match exactly with the picture where there is an optimizer which optimizes for an objective. This optimizer/objective framework is the main way that I’ve talked about optimizers, but also I would not be surprised if this framing turned out to be severely confused.

One possible way that a network could ‘do optimization’ would be for it to do some kind of internal search or internal iterative evaluation process to find the best option. For example, seeing which response best matches a question, or searching a game tree to find the best move. This seems like a broadly useful style of algorithm for a neural network to learn, especially when the training task is complicated. But it also seems unlikely for networks to implement this exactly; i...
​Some Adventures of a Curious Richard Feynman
by Dalton Mabery — 2022-07-06T23:11:32.127Z

This is cross-posted from my blog.

​Sitting around a campfire listening to stories passed on from generations brings with it a feeling of wonder and joy for the adventures told from long ago. Each one of these stories is special on its own and they could never be told in a bad way.

But you reach another level of captivation when someone who is a great storyteller holds the talking stick and their vocal cords rumble. Each word lingers in your mind as you anticipate the next set of syllables, waiting to laugh, to cry, and to love. It's a whole new experience.

This is how I felt reading What Do You Care What Other People Think? Further Adventures of a Curious Character by Richard Feynman. I have a deep admiration for Feynman's beliefs on life, knowledge, and science, but that admiration went deeper still as I followed the stories of his life told by the man himself.

Feynman, You Sexist Pig!

Shortly after giving lectures at CalTech, Feynman was accused of being anti-women because of two stories he told to illustrate different points in his Feynman Lectures on Physics. In response to the letter he received, Feynman wrote, "Don't bug me man!" and sent it off.

About a year later, he was in San Francisco receiving an award he was given from writing those books, but not all was well. A group of protesters were in front of the lecture hall giving out flyers with black, large letters titled, "FEYNMAN SEXIST PIG!"

As he started his talk, he addressed the protestor's:

I'm sorry that my short answer to your letter brought you here unnecessarily. There are more serious places to direct one's attention towards improving the status of women in physics than these relatively trivial mistakes–if that's what you want to call them–in a textbook. But perhaps, after all, it's good that you came. For women do indeed suffer from prejudice and discrimination in physics, and your presence here today serves to remind us of these difficulties and the need to remedy them.

Even though the American Association Physics Teachers has given me an award for teaching, I must confess I don't know how to teach. Therefore, I have nothing to say about teaching. Instead, I would like to talk about something that will be especially interesting to the women in the audience: I would like to talk about the structure of the proton.

Handled with grace and care.

Infinite numbers

Richard Feynman was an "uncle" to several...
Outer vs inner misalignment: three framings
by Richard_Ngo — 2022-07-06T19:46:50.902Z

A core concept in the field of AI alignment is a distinction between two types of misalignment: outer misalignment and inner misalignment. Roughly speaking, the outer alignment problem is the problem of specifying an reward function which captures human preferences; and the inner alignment problem is the problem of ensuring that a policy trained on that reward function actually tries to act in accordance with human preferences. (In other words, it’s the distinction between aligning the “outer” training signal versus aligning the “inner” policy.) However, the distinction can be difficult to pin down precisely. In this post I’ll give three and a half definitions, which each come progressively closer to capturing my current conception of it. I think Framing 1 is a solid starting point; Framings 1.5 and 2 seem like useful refinements, although less concrete; and Framing 3 is fairly speculative. For those who don’t already have a solid grasp on the inner-outer misalignment distinction, I recommend only reading Framings 1, 1.5 and 2.

For the purposes of this document, I’ll focus on the reinforcement learning setting, since that’s the case where the distinction is clearest. However, the same concepts could also apply in a supervised or self-supervised context, if we replace references to “policies” and “reward functions” with “models” and “loss functions”. The extent to which the inner alignment problem will appear in non-RL contexts is an important open question in alignment.

Framing 1: types of behavioral misalignment

Consider training an RL policy until it’s getting high reward in its training environment. Suppose we then evaluate the policy in a different test environment, without retraining it; what will happen? Consider four possibilities:

The policy behaves incompetently. This is a capability generalization failure.
The policy behaves in a competent and desirable way. This is aligned behavior.
The policy behaves in a competent yet undesirable way which gets high reward according to the original reward function.[1] This is an outer alignment failure, also known as reward misspecification.
The policy behaves in a competent yet undesirable way which gets low reward according to the original reward function.[2] This is an inner alignment failure, also known as goal misgeneralization.

Langosco et al. (2022)

provide a more formal definition and some exampl

...
Tarnished Guy who Puts a Num on it
by Jacob Falkovich — 2022-07-06T18:05:59.168Z

Why Elden Ring is great, and how not to die binomially.
Cross-posted from Putanumonit

One reason I haven’t written as much in the last two months is that I’ve been playing a lot of

Elden Ring

. It’s a video game in which you explore a vast and beautiful open world full of monsters who repeatedly kill you in gruesome ways. It’s a punishing and often frustrating experience that demands skill and discipline to persevere. You wouldn’t think that this is a formula for mass entertainment, but Elden Ring is

massively popular

. And with good reason.

I’ve written

about

some of my

favorite video games before and what drives the player in each one, whether it’s the addictive minute-by-minute gameplay or a fascinating mystery to uncover as the narrative unforlds. Elden Ring has a background story (co-written by George R. R. Martin) to uncover, but it’s not central to the experience. The gameplay is pretty standard for action-RPGs: you control a character in third-person view, with movement, dodging, and swinging of comically-oversized weapons. Watching a random minute of someone playing Elden Ring doesn’t appear impressive.

Instead, the main goal of playing Elden Ring is to get really good at Elden Ring, because being really good at a very difficult game is incredibly fun. Like other

very difficult games

, Elden Ring has a progression system with your character upgrading their attributes and gear, but this used just as much to distract the player from the hard work of “gettin gud”. Elden Ring is a rare game where I’m having more fun on my second playthrough — even though my character is weak and has basic gear, the skill I’ve built up lets me breeze through parts that took me long hours to slog through my first time around.

My friend called these games a “

stoic self-control fantasy

“. Getting good mostly isn’t about committing some complex sequence of finger movements to muscle memory like you would to play a piano piece. Each challenging boss fights requires reacting correctly to one of several possible moves by your adversary, often the opposite of your instinctive reaction the first time you face them. Success requires the self-control to overcome your initial instinct, and to let your skill overcome your fear even as you have no idea from second to second what the next move and counter-move are going to be.

The other thing that success requires is leveling up your vigor so th...
Deep neural networks are not opaque.
by jem-mosig — 2022-07-06T18:03:38.455Z

In his List of Lethalities, Eliezer writes that "matrices [in neural networks] are opaque". But I think our understanding of how neural networks actually work has improved significantly last year, with the deep learning theory book. I will briefly outline the general idea of the book below, in case closing this knowledge gap inspires any viable path towards alignment.

Everything that follows is paraphrased from Roberts, Yaida, and Hanin's book, unless I indicate otherwise. To understand the book, you need decent linear algebra and analysis skills, with a sprinkle of information theory. Most of the difficulty comes from the length of the calculations, which requires some practice to get used to. Luckily, the book is exceptionally well written and guides the reader through the math step by step.

Effective theories

Artificial neural networks are usually described in terms of activations, weights, and biases. This is the "atomic" view. But just like describing a gas in terms of the individual atoms that it is made of, the atomic view of neural networks is too fine-grained for most practical purposes. To understand neural networks, we need to know the effective degrees of freedom of the network - akin to temperature, pressure, and volume of a gas. This is not just practical, but it is essential for a true understanding of anything. You haven't fully understood a gas until you have derived the ideal gas law - even if you know all about the quantum field theory that ultimately implies it.

A beautiful thing about nature is that effective theories work. While nature always operates on the lowest level, most of these low-level things often don't matter for the high-level description. Physicists use perturbation theory and "renormalisation group flow" to successfully link the low-level description (e.g. atoms) with the high-level one (e.g. a gas). This not only gives you better concepts to work with at large scales, but also tells you when those concepts will break down and what to do in this case. The idea in the book is to apply the physicist's tools to find an effective theory description of neural networks.

The ensemble

The weights and biases of a neural network are typically initialised randomly before training. During training, all of these parameters are updated (e.g. step by step via gradient descent) until the network produces a desired output from a given inp...