trentmkelly/LessWrong-43k huggingface dataset:
[AN #142]: The quest to understand a network well enough to reimplement it by hand
by Rohin Shah — 2021-03-17T17:10:04.180Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

Circuits Thread (Various people) (summarized by Rohin): We’ve previously (AN #111) summarized the three main claims of the Circuits thread. Since then, several additional articles in the series have been published, so now seems like a good time to take another look at the work in this space.

The three main claims of the Circuits agenda, introduced in Zoom In, are:

1. Neural network features - the activation values of hidden layers - are understandable.

2. Circuits - the weights connecting these features - are also understandable.

3. Universality - when training different models on different tasks, you will get analogous features.

To support the first claim, there are seven different techniques that we can use to understand neural network features that all seem to work quite well in practice:

1. Feature visualization produces an input that maximally activates a particular neuron, thus showing what that neuron is “looking for”.

2. Dataset examples are the inputs in the training dataset that maximally activate a particular neuron, which provide additional evidence about what the neuron is “looking for”.

3. Synthetic examples created independently of the model or training set can be used to check a particular hypothesis for what the neuron activates on.

4. Tuning involves perturbing an image and seeing how the neuron activations change. For example, we might rotate images and see how that impacts a curve detector.

5. By looking at the circuit used to create a neuron, we can read off the algorithm that implements the feature, and make sure that it makes intuitive sense.

6. We can look to see how the feature is used in future circuits.

7. We can handwrite circuits that implement the feature after we understand how it works, in order to check our understanding and make sure it performs as well as the learned circuit.

Note that since techniques 5-7 are based on understanding circuits, their empirical success als...
Sharing a Car
by jefftk — 2021-03-17T13:30:04.654Z

Last week

I posted about how we were
thinking of getting a car, but at least for now we've decided instead
to share a car with one of our

housemates. They have a Honda
Fit, and while it is important to them to have a car, they don't use
it very often.

We're planning on dividing costs in proportion to mileage, though if
that ends up being too much hassle we may shift to just splitting
things 50-50. This includes repairs, gas, and damage to the car that
is not the driver's fault. Damage that is the driver's fault will be
covered by the driver. We'll pay half the cost of car insurance or
the cost of adding us, which ever is more.

The Bluebook value is $5.9k
and we're going to pay them half that up front. This compensates them
for the opportunity cost of having some of their money invested in the
car. At some point, when we're no longer sharing a car, they'll
return half the remaining value. This is not quite right: if we put
more miles on the car we should be getting slightly less back, but
it's pretty close. If the car travels 30k miles over the next three
years it will be worth $4.4k,
while if it travels 40k miles it will be worth $4.2k,
a difference of just $0.02/mile.

There is also some hassle associated with having the car. When it
needs service, we'll take turns bringing it in. Our housemate prefers
to leave it on the street, despite having to move it twice a month for
street cleaning, and will plan to continue doing that. If we're
returning the car within two days of street cleaning, we'll make sure
to park it in a street-cleaning-safe spot. Our housemate will still
be responsible for moving it for street cleaning if no one has used it
in a while. If we choose to park the car in to avoid street cleaning,
we'll be responsible for freeing it after.

For managing usage, we'll keep a shared calendar, defaulting to first
come first served. If we want to use the car at the same time we'll
talk and figure out what makes sense. If no one is currently using
the car and it is not reserved, it's fine to use.

We'll keep our carseats buckled in, though if our housemate needs the
backseat they are free to remove them or move them to the trunk.
We'll make sure we have an extra set so we'll still be able to travel
by car in an emergency if our housemate is out.

If this doesn't work, or stops working, we can go ahead with getting
ou...
HCH Speculation Post #2A
by Charlie Steiner — 2021-03-17T13:26:46.203Z

The first draft of this post started with a point that was clear, cohesive, and wrong. So instead, you get this bunch of rambling that I think should be interesting.

I - Do not mess with time

Harry wrote down 181,429. He repeated what he'd just written down, and Anthony confirmed it.

Then Harry raced back down into the cavern level of his trunk, glanced at his watch (the watch said 4:28 which meant 7:28) and then shut his eyes.

Around thirty seconds later, Harry heard the sound of steps, followed by the sound of the cavern level of the trunk sliding shut. (Harry wasn't worried about suffocating. An automatic Air-Freshening Charm was part of what you got if you were willing to buy a really good trunk. Wasn't magic wonderful, it didn't have to worry about electric bills.)

And when Harry opened his eyes, he saw just what he'd been hoping to see, a folded piece of paper left on the floor, the gift of his future self.

Call that piece of paper "Paper-2".

Harry tore a piece of paper off his pad.

Call that "Paper-1". It was, of course, the same piece of paper. You could even see, if you looked closely, that the ragged edges matched.

Harry reviewed in his mind the algorithm that he would follow.

If Harry opened up Paper-2 and it was blank, then he would write "101 x 101" down on Paper-1, fold it up, study for an hour, go back in time, drop off Paper-1 (which would thereby become Paper-2), and head on up out of the cavern level to join his dorm mates for breakfast.

If Harry opened up Paper-2 and it had two numbers written on it, Harry would multiply those numbers together.

If their product equaled 181,429, Harry would write down those two numbers on Paper-1 and send Paper-1 back in time.

Otherwise Harry would add 2 to the number on the right and write down the new pair of numbers on Paper-1. Unless that made the number on the right greater than 997, in which case Harry would add 2 to the number on the left and write down 101 on the right.

And if Paper-2 said 997 x 997, Harry would leave Paper-1 blank.

Which meant that the only possible stable time loop was the one in which Paper-2 contained the two prime factors of 181,429.

If this worked, Harry could use it to recover any sort of answer that was easy to check but hard to find. He wouldn't have just shown that P=NP once you had a Time-Turner, this trick was more general than that. Harry could use it to find the combinations on ...
On Attempts to Transcend our Models of Numbers
by Chris_Leong — 2021-03-17T08:37:27.417Z

In modern times, many philosophers have dreamt of a perfect language, which would force us to speak precisely and allow us to resolve philosophical disputes by calculation rather than by argumentation. Whilst this dream appears impossible, I hope to offer a method of talking about these problems that is far more precise than would be possible otherwise by continuing in the the spirit of How An Algorithm Feels From the Inside and explaining why we say the things we say about numbers. It is not my intent to drive all the way to a conclusive solution in this post, but rather to help clarify the terms of the debate.

By the end of this post, I hope to have convinced you that I've made progress on the question of whether infinity exists and whether numbers have an existence independent of physical reality and that these two problems are strongly linked.

We want to pick a starting point that introduces as little doubt as possible, so we'll begin by examining our own minds. We quickly notice that we have a model of a thing we call "the world"; a model of a thing we call our "mind", which can mean many things, but which for our purposes we will simplify to mean our mental models and a model of some kind of "relationship" existing between the two. Since this way of describing things is rather wordy, we will shorten this to simply saying that we have a model of the relationship between our model and the world.

Two points:

"Mind" can be defined many different ways. Regardless of our precise definition, our mind contains a model which consists of all the things we can talk about precisely.
Deflationists might object to the use of Correspondence Theory, but we can avoid the metaphysical issue since we are only claiming that this relationship exists inside our mind, not in any actual sense. Undoubtedly, our mind performs some kind of translation between "the concept three" and "all groups of three objects in the world" and we can label this translation the "relationship".

Continuing on, this is already sufficient to produce a basic concept of number. For example, we might notice that according to our model our past experience was as follows: Whenever we modelled a collection of standard-weight packages as numerically being three, then we also modelled our attempt to carry these packages as having been successful afterwards, but that when we modelled the packages as numeric...
Politics is way too meta
by Rob Bensinger — 2021-03-17T07:04:42.187Z

... i.e., it doesn't spend enough time arguing about object-level things.

The way I'm using it in this post, "object-level" might include these kinds of things:

While serving as Secretary of State, did Hillary Clinton send classified information to an insecure email server? How large was the risk that attackers might get the information, and how much harm might that cause?
What are the costs and benefits of various communications protocols (e.g., the legal one during Clinton's tenure, the de facto one Clinton followed, or other possibilities), and how should we weight those costs and benefits?
How can we best forecast people's reliability on security issues? Are once-off mistakes like this predictive of future sloppiness? Are there better ways of predicting this?

"Meta" might include things like:

How much do voters care about Clinton's use of her email server?
How much will reporters cover this story, and how much is their coverage likely to influence voters?
What do various people (with no special information or expertise) believe about Clinton's email server, and how might these beliefs change their behavior?

I'll also consider discussions of abstract authority, principle, or symbolism more "meta" than concrete policy proposals and questions of fact.

This is too meta:

Meta stuff is real. Elections are real, and matter. Popularity, status, controversy, and Overton windows have real physical effects.

But it's possible to focus too much on one part of reality and neglect another. If you're driving a car while talking on the phone, the phone and your eyes are both perfectly good information channels; but if you allocate too little attention to the road, you still die.

When speaking the demon's name creates the demon

I claim:

There are many good ideas that start out discussed by blogs and journal articles for a long time, then get adopted by policymakers.

In many of these cases, you could delay adoption by many years by adding more sentences to the blog posts noting the political infeasibility or controversialness of the proposal. Or you could hasten adoption by making the posts just focus on analyzing the effects of the policy, without taking a moment to nervously look over their shoulder, without ritually bowing to the Overton window as though it were an authority on immigration law.

I also claim that this obeys the same basic causal dynamics as:

My friend Az

...
Intermittent Distillations #1
by Mark Xu — 2021-03-17T05:15:27.117Z

This is my low-budget version of Rohin's Alignment Newsletter.

A critique of pure learning and what artificial neural networks can learn from animal brains (Anthony M. Zador)

A critique of pure learning and what artificial neural networks can learn from animal brains

Summary

This paper points out that human learning algorithm must rely on nearly all unsupervised data because humans don't live that long before they become intelligent. The authors also point out that many animals, potentially including humans, are remarkably functional after only a very short time being alive. This result suggests that there are parts of animal behavior that are not learned. This observation aligns with Yann Lecun's metaphor: "If intelligence is a cake, the bulk of the cake is unsupervised learning, the icing on the cake is supervised learning, and the cherry on the cake is reinforcement learning."

Since the genome contains finite information (only about 1.5Gb in humans), there is a tradeoff between having lots of innate and fixed behaviors and having many learned and adaptable behaviors. Animals in relatively rigid environments might benefit from having lots of innate and fixed behaviors, whereas animals in adaptive environments might benefit more from learning. Additionally, animals that must be functional early to survive might favor innate behaviors, whereas animals with time to learn can achieve higher peak performance.

The authors also suggest that the genome's finite information might form a "genomic bottleneck" that serves as a regularizer, potentially improving generalization performance.

Opinion

I agree with the observation that most humans seem to learn based on unsupervised data, although I think there are denser forms of "labels" in reality than the authors claim. For example, babies can get reinforcement from their parent's facial expressions, which potentially gives them a considerable number of labeled examples.

I have already read (although I forget the source) that babies preferentially look at things with two dark spots above a single dark spot because these patterns are likely to be faces. If true, this instinct suggests that in addition to "innate behaviors", animals also have "innate learning heuristics" that direct them to places to get high-quality training data.

I am unconvinced by the claim that the size of the genome serves as a meaningful bottleneck. Th...
Comments on "The Singularity is Nowhere Near"
by Steven Byrnes — 2021-03-16T23:59:30.667Z

I followed a link on Twitter to a fun and informative 2015 blog post by Tim Dettmers:

The Brain vs Deep Learning Part I: Computational Complexity — Or Why the Singularity Is Nowhere Near

The headline conclusion is that it takes at least 1021 FLOP/s to run the algorithms of a human brain, and therefore "it is unlikely that there will be a technological singularity in this century." I disagree with that, and this post explores why.

(Specifically, I disagree with "at least 1021 FLOP/s". There's a separate step to go from "at least 1021 FLOP/s" to "it is unlikely that there will be a technological singularity in this century"—this step is related to Moore's law, bandwidth requirements for parallelization, etc. Tim's blog post has extensive discussion of this second step, and I won't say anything about that here; I'd have to think about it more.)

(I'm writing this in 2021, six years later, but Tim has a comment on this very site that says he still stands by that post; in fact he now goes even further and says "I believe that AGI will be physically impossible with classical computers.")

I highly recommend the original post. Indeed, if I didn't like the post so much, I would not have bothered writing a response. :-)

Are brain algorithms computationally expensive to simulate?

Yes! Definitely! I think it's especially telling that nobody has applied the Dileep George brain-inspired image-processing model to ImageNet, sticking to much smaller images with far fewer categories of objects (MNIST, CAPTCHAs etc.).

Likewise, this Randall O'Reilly paper has a fascinating computational exploration of (in my opinion) different and complementary aspects of the human visual system. That paper tests its theories on a set of ≈1000 256×256-pixel, 8-frame movies from 100 categories—compare to ImageNet's 14 million images from 20,000 categories ... or compare it to the number of visual categories that you can recognize. Training the model still took 512 InfiniBand-connected processor nodes running for ≈24 hours on their campus supercomputer (source: personal communication). The real human vision system is dramatically larger and more complicated than this model, and the whole brain is larger and more complicated still!

But, when I say "computationally expensive to simulate" above, I mean it in, like, normal-person-in-2021 standards of what's computationally e...
MetaPrompt: a tool for telling yourself what to do.
by abramdemski — 2021-03-16T20:49:19.693Z

MetaPrompt is a nonstandard todo list. The very bare-bones basic idea is that you add ideas to a deck of cards, and you shuffle this deck to draw things to do. This is useful for "tasks" with no priority, where you just want to be reminded of the idea at some point in the future. You can also add cards which tell you to add more cards; hence the name.

Most of this is written in a very narrative way. If you just want the how-to, skip to the how-to section.

For many people, there's something interesting about being told what to do: it's easier to make someone else a sandwich than to make one for yourself, for example. One partial explanation is decision fatigue: deciding to do things for yourself takes some sort of energy, which being told to do doesn't take.

This suggests that, in some situations, people can benefit from 'willpower partners': you each (for example) tell each other to make a sandwich. You end up with two sandwiches and no decision fatigue; a net gain over you each making a sandwich for yourself.

But what if you could be your own "willpower partner"?

In 2016, I was learning dvorak. I wanted typing practice; but, I didn't want boring typing practice. I remembered the post about WriterKata on Agenty Duck from the previous year. WriterKata was a great little website for practicing writing by responding to writing prompts. Why not practice typing and creative writing at the same time, by responding to prompts?

Unfortunately, by 2016, WriterKata no longer existed. It seems the author took the website down. I guess it wasn't worth the hosting expense? (Sad; it seemed good enough that it should have been able to support itself somehow.)

I could have just found writing prompts somewhere and typed my responses into any kind of text file, but I wanted the clean interface: a random prompt selected for me, with a text box directly below it where I could immediately type my response. I know from experience that can be enough to make the difference between vaguely intending to practice for months, vs sitting down and doing it for hours.

I went to the command line. I started a file "kata" for writing prompts, with the idea that I could pull one line from it randomly using standard command line tools. Then, I could write my responses to another file.

However, I was lazy. I didn't want to spend a bunch of time copying writing prompts to the file before beginning.

So, the...
Partial-Consciousness as semantic/symbolic representational language model trained on NN
by Joe Kwon — 2021-03-16T18:51:00.038Z

If you hook up a language model like GPT-3 to a chess engine or some other NN model, isn't a tie from semantic/symbolic level representation (words and sentences that are coherent and understandable) to distributed, subsymbolic representations in NNs being established? How likely is it that this is how the human brain works? Isn't this also progress towards causal model-building because we now have an easily manipulatable model (causal model with concepts/symbolic relations)? I don't see how someone can refute that a system like this isn't truly "understanding" the way humans "understand" (what would Searle say).

Related thought: I have spoken briefly at EA conferences to AI peoples who have spoken about their skepticism for symbolic models and NN+symbolic hybrid models. I'm curious about the reasons for why; any pointers to resources and papers if anyone is short on time, would also be really appreciated. Thank you.
An Antropic Argument for Post-singularity Antinatalism
by monkaap — 2021-03-16T17:40:24.469Z

In this post I hope to demonstrate that anyone that is able to create a superinteligent AI has the moral obligation to desing it to be Antinatalist. Failure to do so would substantially increase the probability of human extinction.

To the Garden of Eden

In his book Anthropic Bias, Nick Bostrom argues that the

Selfs-Sampling Assumpution (SSA)

is true. the SSA states that you should assume you are a random member of your reference class throughout space and time (in our case human beings). For this post we are only interested in the temporal aspect of the anthropic bias. That is to say: Whether we are to order all humans along a line, from Adam and Eve, all the way to the Last Man. where in that line (percent wise) should you expect yourself to be? If we assume the SSA to be true; you should say you have a 1% to be among the first 1%, a 10% to be among the last 10%, a 50% percent chance to be among the middle 50%, etc. This conclusion seems benign, but has some unusual consequences. Such as for example in the fable of

Serpent’s Advice

:

Eve and Adam, the first two humans, knew that if they gratified their flesh, Eve might bear a child, and that if she did, they would both be expelled from Eden and go on to spawn billions of progeny that would fill the Earth with misery. One day a serpent approached them and spoke thus: “Pssst! If you hold each other, then either Eve will have a child or she won’t. If she has a child, you will have been among the first two out of billions of people. Your conditional probability of having such early positions in the human species given this hypothesis is extremely small. If, on the other hand, Eve does not become pregnant then the conditional probability, given this, of you being among the first two humans is equal to one. By Bayes’s theorem, the risk that she shall bear a child is less than one in a billion. Therefore, my dear friends, indulge your desires and worry not about the consequences!”

To the Labs of Eden

Why am I telling you all of this? Because in reality we are Adam and Eve.

Consider Sydney, future AI researcher. Sydney has developed a superintelligent AI. If its alignment scheme works it will create a future full of joy and happiness, resulting in the long and happy life of a quintillion human beings, making use of Dyson SwarmTechnology. If it does not work, everyone will instantly die. Lucki...