trentmkelly/LessWrong-43k huggingface dataset:
Elementary Statistics
by jefftk — 2019-12-05T02:00:02.266Z

Our elementary school has a directory listing kids and parents, and
since we live in the future it's a spreadsheet, which means I can
count things. A typical family at this K-5 school has one child
enrolled (76%). The child has two parents (96%) with different last
names (59%), but they share a last name with at least one parent
(89%). The parents don't share email addresses (95%) or phone numbers
(97%), do use gmail (72%), and do have Boston

area codes (68%). Our family in the
majority for each of these, even though there's naively only a 18%
chance of that happening and they seem reasonably independent.

It's surprising to me that while parents mostly don't have the same
names as each other (59%), only 11% of their kids have hyphenated
names. I guess people realized that hyphenated names grow
exponentially? I'd like to look at how the children's last names
relate to parental gender, but that would involve annotating inferred
genders for ~500 parents.

Boring details:

There are 233 kids across six grades (K-5) with two classes per
grade, for an average of 19 kids per class. Kindergarten is the
biggest (45 kids, 22 and 23 each) while second grade is the smallest
(30 kids, 15 and 15 each).

224/233 (96%) have two parents listed.

Of the kids with two parents listed, 91/224 (41%) have the same
last name as each other.

207/233 (89%) of kids have the same name as at least one of
their parents.

Of the 26 kids who have a different name, 15 (58%) are
hyphenations, 7 (27%) have no obvious connection, 3 (12%) list only one
parent and may share a last name with the other, and 1 (4%) is an
unhyphenation (Sam Alpha-Bravo and Pat Charlie, with child Alex
Bravo).

141 (76%) of families have one child in the school, 43 (23%)
have two, and 2 (1%) have three.

All parents are listed with email addresses, but 12/224 (5%)
have the same email listed for both parents.

255/355 (72%) unique email address gmail accounts, 35 (10%) are
yahoo, 13 (4%) are hotmail, 7% (2%) are aol, and 11 (3%) are edu.

All but one parent is listed with a phone number, but 6/224
(3%) have the same phone listed for both parents.

Of 357 unique phone numbers, 242 (68%) are Boston (including
Somerville), 29 (8%) are NYC, 28 (8%) are Boston's inner suburbs, and
8 (2%) are the Bay Area. Boston's Northern suburbs (978 and 351) and
Southern suburbs (508 and 774) are each below 1%, behi

...
Kansas City Dojo meetup 11-19-19
by Bae's Theorem — 2019-12-05T00:13:32.445Z

I.) ROUND TABLE

I began by sharing an instance today where I fell for Cognitive Fusion, but successfully noticed it myself. After our catering order of subs last week, I had developed the routine “pick up food on the way to the Dojo”. This week, I changed the food to pre-packaged fruit and nuts, but I still had the algorithm “pick up food on the way to the dojo”, even though I could have purchased the food days ahead of time, saving me a significant amount of trouble in predicting my already hectic commute to the Dojo meeting space. I now plan to follow through on this in the future. I bring it up mostly as a status update on my own self awareness. I directly credit the Dojo for this particular awareness, since they have pointed out my Cognitive Fusions in the past.
I continued by bringing up the situation with my friends, which has escalated to a point where one of them had attempted suicide. I feel awful about the situation, and am trying to be there to support them as their friend. However, I wish I could do more, and I have this anxiety that if only I were smarter and more knowledgeable, there would be something more I could do. Secondarily, I have anxiety that I enjoy other people’s company more than they enjoy mine. I asked the others if they have any advice on how to deal with these feelings.

W pointed out that being a friend doesn’t usually involve “fixing” the other person. Just being present, and instilling a feeling of not being alone. Being non-judgmental. As to my secondary concern, he said “If you are enjoying something more than another person, then good for you!” Which was a helpful reframing of my feelings.
Life Engineer took a different approach to the things I had said, and wondered out loud if I felt like I was being used by certain friends. He stressed the importance of setting boundaries, and the equality of people in a relationship. He has had friendships in the past that fell apart as soon as he stopped being the initiator, and considers that a sign of an unhealthy relationship. He also emphasized the importance of Self Care. I said I will have to examine my friendships individually, but that I will take this concern seriously and endeavor to establish boundaries for my own mental health.
W brought up the difference between “being” vs “doing”

...
What are some non-purely-sampling ways to do deep RL?
by evhub — 2019-12-05T00:09:54.665Z

Conventionally in machine learning, if you want to learn to minimize some loss or maximize some expected return, you do so by sampling a bunch of losses/rewards and training on those. Since the model only ever sees the loss or reward function through the lens of those specific samples, this basic approach introduces a proxy alignment problem.

For example, suppose you train an RL agent to maximize its future discounted return according to some reward function r. Furthermore, suppose there exists some other reward function r′ such that r and r′ give equivalent samples on the training distribution, but diverge elsewhere. If you just train your agent via evaluating r on a bunch of samples, however, then even if your model is in some sense trying to do the right thing, it has no possible way of knowing whether r or r′ is the right generalization.

In many cases, however, we know exactly what r is—we have explicit code for it and everything (or at least some sort of natural language description of it)—but we still only make use of r via sampling/evaluation. Of course, in many RL settings, you actually do only know how to evaluate r, not inspect it in any other way. However, I think a system that only works in settings where you have more access to the reward function than that can still do quite a lot—even if you explicitly know an environment’s reward function, it can still be quite difficult to figure out the optimal policy (think Go, for example) such that having an ML system which can figure it out for you is quite powerful.

So, here’s my question: at least for environments in which you have a known reward function, what are some ways of making use of that information in training a deep learning model other than evaluating that reward function on a bunch of samples? I’m also interested in ways of doing this in non-RL settings, though I still mostly only want to focus on deep learning approaches—there are certainly ways of doing this in more classical machine learning, but I’m less interested in those.

Some possibilities that I’ve considered so far:

Put a differentiable copy of the reward function inside the network during training such that the network is able to arbitrarily query the reward function however it wants (credit to Nevan Wichers for this idea). For a smooth reward function you could also give your model the ability to explicitly query gradients as wel

...
Karate Kid and Realistic Expectations for Disagreement Resolution
by Raemon — 2019-12-04T23:25:59.608Z

There’s an essay that periodically feels deeply relevant to a situation:

Someday I want to write a self-help book titled “F*k The Karate Kid: Why Life is So Much Harder Than We Think”.

Look at any movie with a training montage: The main character is very bad at something, then there is a sequence in the middle of the film set to upbeat music that shows him practicing. When it's done, he's an expert.

It seems so obvious that it actually feels insulting to point it out. But it's not obvious. Every adult I know--or at least the ones who are depressed--continually suffers from something like sticker shock (that is, when you go shopping for something for the first time and are shocked to find it costs way, way more than you thought). Only it's with effort. It's Effort Shock.

We have a vague idea in our head of the "price" of certain accomplishments, how difficult it should be to get a degree, or succeed at a job, or stay in shape, or raise a kid, or build a house. And that vague idea is almost always catastrophically wrong.

Accomplishing worthwhile things isn't just a little harder than people think; it's 10 or 20 times harder. Like losing weight. You make yourself miserable for six months and find yourself down a whopping four pounds. Let yourself go at a single all-you-can-eat buffet and you've gained it all back.

So, people bail on diets. Not just because they're harder than they expected, but because they're so much harder it seems unfair, almost criminally unjust. You can't shake the bitter thought that, "This amount of effort should result in me looking like a panty model."

It applies to everything. [The world] is full of frustrated, broken, baffled people because so many of us think, "If I work this hard, this many hours a week, I should have (a great job, a nice house, a nice car, etc). I don't have that thing, therefore something has corrupted the system and kept me from getting what I deserve."

Last time I brought this up it was in the context of

realistic expectations for self improvement

.

This time it’s in the context of productive disagreement.

Intuitively, it feels like when you see someone being wrong, and you have a simple explanation for why they’re wrong, it should take you, like, 5 minutes of saying “Hey, you’re wrong, here’s why.”

Instead, Bob and Alice people might debate and doublecrux for 20 hours, making serious effort to understand each other’

...
Recent Progress in the Theory of Neural Networks
by interstice — 2019-12-04T23:11:32.178Z

It's common wisdom that neural networks are basically "matrix multiplications that nobody understands" , impenetrable to theoretical analysis, which have achieved great results largely through trial-and-error. While this may have been true in the past, recently there has been significant progress towards developing a theoretical understanding of neural networks. Most notably, we have obtained an arguably complete understanding of network initialization and training dynamics in a certain infinite-width limit. There has also been some progress towards understanding their generalization behavior. In this post I will review some of this recent progress and discuss the potential relevance to AI alignment.

Infinite Width Nets: Initialization

The most exciting recent developments in the theory of neural networks have focused the infinite-width limit. We consider neural networks where the number of neurons in all hidden layers are increased to infinity. Typically we consider networks with a Gaussian-initialized weights, and scale the variance at initialization as 1√H, where H is the number of hidden units in the preceding layer(this is needed to avoid inputs blowing up, and is also the initialization scheme usually used in real networks). In this limit, we have obtained an essentially complete understanding of both behavior at initialization and training dynamics[1]. (Those with limited interest/knowledge of math may wish to "Significance and Limitations" below).

We've actually had a pretty good understanding of the behavior of infinite-width neural networks at initialization for a while, since the work of Radford Neal(1994). He proved that in this limit, fully-connected neural networks with Gaussian-distributed weights and biases limit to what are known as Gaussian processes. Gaussian processes can be thought of the generalization of Gaussian distributions from finite-dimensional spaces to spaces of functions. Neal's paper provides a very clear derivation of this behavior, but I'll explain it briefly here.

A neural network with m real-valued inputs and 1 real valued outputs defines a function from Rm to R. Thus, a distribution over the weights and biases of such a neural network -- such as the standard Gaussian initialization -- implicitly defines a distribution over functions on Rm. Neal's paper shows that, for fully-connected neural networks, this distributio... ,
Paper-Reading for Gears
by johnswentworth — 2019-12-04T21:02:56.316Z

Lesswrong has a fair bit of advice on

how to evaluate

the claims made in scientific papers. Most of this advice seems to focus on a single-shot use case - e.g. a paper claims that taking hydroxyhypotheticol reduces the risk of malignant examplitis, and we want to know how much confidence to put on the claim. It’s very black-box-y: there’s a claim that if you put X (hydroxyhypotheticol) into the black box (a human/mouse) then Y (reduced malignant examplitis) will come out. Most of the advice I see on evaluating such claims is focused around statistics, incentives, and replication - good general-purpose epistemic tools which can be applied to black-box questions.

But for me, this black-box-y use case doesn’t really reflect what I’m usually looking for when I read scientific papers.

My goal is usually not to evaluate a single black-box claim in isolation, but rather to build a

gears-level model

of the system in question. I care about whether hydroxyhypotheticol reduces malignant examplitis only to the extent that it might tell me something about the internal workings of the system. I’m not here to get a quick win by noticing an underutilized dietary supplement; I’m here for the long game, and that means

making the investment

to understand the system.

With that in mind, this post contains a handful of thoughts on building gears-level models from papers. Of course, general-purpose epistemic tools (statistics, incentives, etc) are still relevant - a study which is simply wrong is unlikely to be much use for anything. So the thoughts and advice below all assume general-purpose epistemic hygiene as a baseline - they are things which seem more/less important when building gears-level models, relative to their importance for black-box claims.

I’m also curious to hear other peoples’ thoughts/advice on paper reading specifically to build gears-level models.

Get Away From the Goal

Ultimately, we want a magic bullet to cure examplitis. But the closer a paper is to that goal, the stronger publication bias and other memetic distortions will be. A flashy, exciting result picked up by journalists will get a lot more eyeballs than a failed replication attempt.

But what about a study examining the details of the interaction between FOXO, SIRT6, and WNT-family signalling molecules? That paper will not ever make the news circuit - ...
On decision-prediction fixed points
by jollybard — 2019-12-04T20:49:36.464Z

It seems like for embedded (reflexive, Löbian, etc) LDT agents, there ought to be a fixed point thing between decision and prediction.

Indeed, embedded agents can predict things about their own actions; but by modeling themselves sufficiently well, this should be (in the limit) equivalent to making a decision, as they will be modeling their own thoughts. Conversely, once you have decided, if you do not suffer from akrasia, then you have accurately predicted your next action. (aside: this is the source of the illusion of free will.)

This is related to the class of "metaphysical truths": truths of the form □P→P. Whenever an embedded agent believes one of those, then it must (by Löb's theorem) eventually believe P. But there are lots of such truths (perhaps each different religion offers a different set of metaphysical truths), which might then lead to spurious, or even contradictory beliefs!

The key word was "eventually", assuming LDT agents are logical inductors of some kind; in the meantime, the agent may choose its beliefs. Isn't this weird? Beliefs shouldn't be arbitrary!

But you can imagine, as an (imperfect) example, the paradox of self-confidence: if you think you are competent, then you could believe in your ability to self-improve, which will encourage your to foster your own competence; on the other hand thinking that you are incompetent may lead to not believing in your self-improvement ability, leading to a downward spiral.

Each one of these are decision-belief fixed points. Each are, in way (causally?), both true and rational.

I feel like LDT will end up being a reflexive fixed point of this sort (reminiscent of the logical induction fixed point), with the catch that there are many such fixed points. The true decision an LDT agent must make is then choosing the most effective of these fixed points.

(I'm not entirely convined of this yet since I still have no idea what logical counterfactuals will look like)

The moral of the story for us humans is that:

akrasia should not exist, not if you can predict yourself well enough;

sometimes beliefs are arbitrary. choose the most productive ones, you'll end up believing them all anyway.

EDIT: I feel this is rather important, so I would appreciate getting feedback on the presentation.
[AN #76]: How dataset size affects robustness, and benchmarking safe exploration by measuring constraint violations
by Rohin Shah — 2019-12-04T18:10:01.739Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Highlights

Self-training with Noisy Student improves ImageNet classification

(Qizhe Xie et al) (summarized by Dan H): Instead of summarizing this paper, I'll provide an opinion describing the implications of this and other recent papers.

Dan H's opinion: Some in the safety community have speculated that robustness to data shift (sometimes called "transfer learning" in the safety community) cannot be resolved only by leveraging more GPUs and more data. Also, it is argued that the difficulty in attaining data shift robustness suggests longer timelines. Both this paper and

Robustness properties of Facebook's ResNeXt WSL models

analyze the robustness of models trained on over 100 million to 1 billion images, rather than only training on ImageNet-1K's ~1 million images. Both papers show that data shift robustness greatly improves with more data, so data shift robustness appears more tractable with deep learning. These papers evaluate robustness using benchmarks collaborators and I created; they use

ImageNet-A

,

ImageNet-C

, and

ImageNet-P

to show that performance tremendously improves by simply training on more data. See

Figure 2

of the Noisy Student paper for a summary of these three benchmarks. Both the Noisy Student and Facebook ResNeXt papers have problems. For example, the Noisy Student paper trains with a few expressly forbidden data augmentations which overlap with the ImageNet-C test set, so performance is somewhat inflated. Meanwhile, the Facebook ResNeXt paper shows that more data does not help on ImageNet-A, but this is because they computed the numbers incorrectly; I personally verified Facebook's ResNeXts and more data brings the ImageNet-A accuracy up to 60%, though this is still far below the 95%+ ceiling. Since

adversarial robustness can transfer to other tasks

, I would be surprised if robustness from these models could not transfer. These results suggest data shift robustness can be attained within the current paradigm, and that attaining image classifier robustness will not require a long timeline.

Safety Gym

(Alex Ray, Joshua ...
"Fully" acausal trade
by Stuart_Armstrong — 2019-12-04T16:39:46.481Z

Acausal trade happens when two agents manage to reach a deal with each other, despite not being able to interact causally (and, in some cases, not being sure the other one exists). Consider, for example, the prisoner's dilemma played against another copy of yourself, either in the next room or the next universe.

But those two situations are subtly different. If my copy is in the next room, then we will interact after we've reached our decision; if they're in the next universe, then we won't.

It might seem like a small difference, but my simple way of breaking acausal trade succeeds in the "next universe" situation, but fails in the "next room" situation.

So it would be good to distinguish the two cases. Since the terminology is well established, I'll call the "next universe" situation - where there are no interactions between the futures of the agents - to be "fully" acausal trade.
In which ways have you self-improved that made you feel bad for not having done it earlier?
by Mati_Roy — 2019-12-04T12:33:13.143Z

It can be a decision, a skill, a habit, etc.

Can be because the improvement was very valuable, obvious in insight, a moral imperative, or any other reason.

Note that I'm specifically looking for things that made you feel bad for not having done it earlier, not a simple "ah, I guess it would have been useful to know that earlier".
A letter on optimism about human progress
by jasoncrawford — 2019-12-04T04:21:42.033Z
Link: https://letter.wiki/conversation/202#letter_1093

This open letter was originally posted on Letter.wiki and is part of a longer conversation with Andrew Glover about sustainability and progress.

Dear Andrew,

Thanks for a thoughtful reply. Reading over it, it seems the biggest difference between us is in our expectations for the future, in a word, our optimism. You agree it would be nice to give everyone the luxuries that only the rich enjoy today, but that “it doesn't seem possible to do this.” In a similar vein, on the topic of energy resources, you say you're “not aware of any principle that says new energy sources will be discovered simply by virtue of humans applying their ingenuity.”

So let's talk about that principle.

Certainly there is no law of physics that mandates inexorable progress, on any one axis or even in aggregate. Progress is not automatic or inevitable.

But human ingenuity has been so successful in such a wide variety of areas that I think, on the grounds of history and philosophy, we are justified in drawing a general principle: all problems are solvable, given enough time, effort, and thinking. Or to quote David Deutsch from The Beginning of Infinity, “anything not forbidden by the laws of nature is achievable, given the right knowledge.”

To take the historical view first, think of all the problems humanity has solved, all the magic we've created, that seemed impossible until it was invented—not just in energy, but in every field.

Our farms make an abundance of produce spring from the ground—reliably, consistently, year after year, rain or shine, flood or drought, regardless of what weeds, pests, or disease may attack our crops. We do this in many parts of the world, with different terrain, weather patterns, and growing seasons. We have done this not just through soil, fertilizer, and irrigation, but by breeding better plants—taking command of the taxonomy of species itself. And when the food is ready, we keep it fresh while it is transported all over the world; produce now knows no season or country.

We have largely conquered infectious disease. Except in the parts of the world too poor for effective water sanitation or mosquito control (and the parts of California where children aren't vaccinated), infectious disease is rare, and usually curable. Less than 200 years ago, we didn't know what caused these diseases or how they spread; today we have identified the specific microorganism behind ev

...