trentmkelly/LessWrong-43k huggingface dataset:
Re: Fierce Nerds
by lsusr — 2021-05-19T20:18:06.035Z
Link: http://paulgraham.com/fn.html

Paul Graham just published an article titled Fierce Nerds. It's the most precise description of me I've ever read.

Most people think of nerds as quiet, diffident people. In ordinary social situations they are — as quiet and diffident as the star quarterback would be if he found himself in the middle of a physics symposium. And for the same reason: they are fish out of water. But the apparent diffidence of nerds is an illusion due to the fact that when non-nerds observe them, it's usually in ordinary social situations. In fact some nerds are quite fierce.

This is absolutely true. When I'm in my element I own the room. I am so bored in ordinary social situations I used them to drill my Chinese handwriting for three years.

The fierce nerds are a small but interesting group. They are as a rule extremely competitive — more competitive, I'd say, than highly competitive non-nerds. Competition is more personal for them. Partly perhaps because they're not emotionally mature enough to distance themselves from it, but also because there's less randomness in the kinds of competition they engage in, and they are thus more justified in taking the results personally.

Yup.

Fierce nerds also tend to be somewhat overconfident, especially when young. It might seem like it would be a disadvantage to be mistaken about one's abilities, but empirically it isn't. Up to a point, confidence is a self-fulfilling prophecy.

Overconfidence. That's familiar.

Another quality you find in most fierce nerds is intelligence. Not all nerds are smart, but the fierce ones are always at least moderately so. If they weren't, they wouldn't have the confidence to be fierce.

Yup.

There's also a natural connection between nerdiness and independent-mindedness. It's hard to be independent-minded without being somewhat socially awkward, because conventional beliefs are so often mistaken, or at least arbitrary. No one who was both independent-minded and ambitious would want to waste the effort it takes to fit in. And the independent-mindedness of the fierce nerds will obviously be of the aggressive rather than the passive type: they'll be annoyed by rules, rather than dreamily unaware of them.

I have an unusually low tolerance for rules. The first time I hired people I was disconcerted by how much they desired for someone to tell them what to do.

I used to be very aggressive when I was a teenager....
[AN #151]: How sparsity in the final layer makes a neural net debuggable
by Rohin Shah — 2021-05-19T17:20:04.453Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

Note: The newsletter will be slowing down a bit over the next month, as I’ll be fairly busy. I’m currently aiming to produce a newsletter every two weeks, but I don’t know if even that will happen.

HIGHLIGHTS

Debuggable Deep Networks: Usage and Evaluation (Eric Wong, Shibani Santurkar et al) (summarized by Rohin): One simple approach to make neural nets more understandable is to make just the final layer sparse. Neurons in the penultimate layer can be visualized using existing techniques, and the sparsity of the final layer means that it is relatively easy to understand how they are combined together to make predictions. For example, in ImageNet, the final logit for an individual class becomes a weighted combination of around 20 features, instead of 2048 as you would get with a dense model. The authors' core claim is that this makes the model more understandable and debuggable, at the cost of a small drop in performance (about 1-5 percentage points). They show this using several experiments, many with real humans:

1. The most basic test is simulation: can humans predict what the model would say (regardless of whether or not it is correct)? Unfortunately, if you show people a picture of an airplane, they are probably going to predict that the model says “airplane”, on priors. To avoid this sort of prior knowledge, they first sample a class like “airplane” that they don’t reveal. Instead, they reveal feature visualizations of five randomly chosen features that the model uses to identify images of that class. They then choose three images and ask humans which of the three images will have the highest probability of being assigned to that class, according to the model. They find that when using a sparse final layer, humans have non-trivial performance (72% when the best image really is from the sampled class, and 57% when the best image is from some different cla...
What should we expect from ARPA-H?
by ChristianKl — 2021-05-19T12:21:41.941Z

The Biden administration pretends at the moment that they want to create a new organization that works like Darpa and that's more open to innovation the way Darpa is more open to innovation. The announcement reads:

Launch the Advanced Research Projects Agency for Health (ARPA-H). The discretionary request calls for $6.5 billion to launch the Advanced Research Projects Agency for Health (ARPA-H) within the National Institutes of Health (NIH). With an initial focus on cancer and other diseases such as diabetes and Alzheimer’s, this major investment in federal research and development will drive transformational innovation in health research and speed application and implementation of health breakthroughs.

Part of what makes Darpa special that it's not told by politicians to focus on the problems that are most obvious to the politicians but is more free about chosing politicians. Focus on Cancer/Diabetes/Alzheimer's suggest that's not the case here as those are the traditional targets of NIH funding anyway. Does anybody have a more optimistic take on the annoucement and expect it to actually produce an organization that's comparable to DARPA?
Bayeswatch 4: Mousetrap
by lsusr — 2021-05-19T03:17:41.423Z

Miriam chucked the replica Salvator Mundi into the bonfire.

"We do a lot of shooting first and asking questions never," said Vi.

"Personnel are expensive. AIs are replaceable. We have standard operating procedures. It wasn't always this way. AIs used to be rare. Knowledge was precious in those early days. We didn't know what the machines could and couldn't do," said Miriam.

"You were a founder of Bayeswatch?" said Vi.

"I am not that old," said Miriam.

Miriam paused the holocaust for a moment to examine an oil painting of a vampire chained to a solar execution chamber. She tossed it into the fire.

"They had yet to standardize the architectures back then. There were overhangs all over the place. Using explicit error-entropy functions wasn't even standard industry practice. Instead they just used the implicit priors of whatever architecture got results quickly," said Miriam.

"That's like making gunpowder without atomic theory," said Vi.

"Or medicine without chemistry. Those early machines were…" Miriam trailed off.

Vi tossed a painting of a dodo tree into the fire.

"One of my first missions…it was my mentor's last. We were dispatched to explore a small compound with signs of unaligned activity," said Miriam.

"That's suicide. What was command thinking?" said Vi.

"It was the early days. Singularity breakout could have been just around the corner," said Miriam.

"But drones—" said Vi.

"Software back then was written by human beings. It had more security holes than features. Sending a drone to investigate a misaligned AI was like sending a set of lockpicks to investigate a whether a magician has broken out of his cell," said Miriam.

Miriam wore lots of foundation and concealer on her face. Vi wondered how many scars it covered up.

"We investigated a compound in the mountains of California. Kind of reminded me of Ex Machina," said Miriam.

"Was it owned by a billionaire?" said Vi.

"In your dreams. You read too many romance novels. The guy wasn't not-rich. He was an early employee of a moderately-successful software startup. I guess he built the error-entropy minimizer himself. To this day I am unsure what the thing was supposed to do. It was dead by the time we showed up," said Miriam.

"Dead?" said Vi.

"Poetic license. My point is we weren't dealing with an active threat. The inventor turned on his machine. It carried out its mission. It turned itself off. The end," said Miriam.

"There's ...
There is no No Evidence
by MalcolmOcean — 2021-05-19T02:44:49.462Z

Zvi recently coined, and has now written up this Law of No Evidence:

Law of No Evidence: Any claim that there is “no evidence” of something is evidence of bullshit.

Considered next to Eliezer's Absence of Evidence Is Evidence of Absence, it might seem like a contradiction, but as far as I can tell, it actually follows directly.

If we treat the "is" in Absence of Evidence is Evidence of Absence as an "implies" (which it seems to me to be) and then apply modus tollens to it, we get "if you don't have evidence of absence, you don't have absence of evidence" and it is precisely this bullshit that Zvi is calling. If you have evidence of absence, say so.

The Very Serious covid spokespeople spouting bullshit like "no evidence of human-to-human transmission" are doing a sort of naive equivocation version of "absence of evidence is evidence of absence", by saying "no evidence for" as if that implies "evidence against" when in fact they're trying to support some agenda or worldview when there's not very much clear evidence in any direction and someone could just as easily say "there's no evidence there isn't human-to-human transmission". At least, that's in the best case. Sometimes the evidence does favor a particular direction, they just don't like it and don't want to count it. Desperately clinging to priors?

Hm.

So Zvi's Law of No Evidence can be seen as taking Absence of Evidence is Evidence of Absence a step further and instead saying "if your absence of evidence is real, then it'll actually be evidence of absence, in which case call it that. otherwise shut up." But there's still a point to be made about a sense in which "absence of evidence" is its own thing—it's just different from no evidence. The important piece is that "no evidence" is bullshit, but "we didn't see this particular thing we would more expect to see if X were true than if X weren't true" is a vital component of successful reasoning about the world. It's the very basis of Bayes.

There is no No Evidence

If you observe {not[evidence of X]} then it makes sense to update towards {not[X]}, but the speech act of claiming "There is no evidence of X" only occurs when there exists some apparent evidence that someone wants to claim doesn't count as evidence. This is a frame battle disguised as argument, since there's some evidence for almost any hypothesis you can possibly imagine.

There's evidence for aliens on M

...
Functional Trade-offs
by weathersystems — 2021-05-19T01:06:08.518Z

I found this super cool paper from Marco Del Giudice and Bernard J. Crespi called Basic functional trade-offs in cognition: An integrative framework that I thought y'all would like. What follows is largely just a condensed version of their paper where I take the most relevant quotes and paste them here, but I highly recommend reading the full paper.

This paper takes an initial step toward a general theory of cognitive trade-offs by examining four key properties of goal-directed systems: performance, efficiency, robustness, and flexibility. These properties define a number of basic functional trade-offs that can be used to map the abstract “design space” of natural and artificial cognitive systems. Basic functional trade-offs provide a shared vocabulary to describe a variety of specific trade-offs including speed vs. accuracy, generalist vs. specialist, exploration vs. exploitation, and many others.

We could not find other examples of properties that were both universal (i.e., would apply to all functional systems) and similarly general (i.e., were not already en- compassed by the basic ones). This assertion does not mean that the classification we propose is fully exhaustive or that it cannot be extended in principle, and we encourage its growth and elaboration.

A map of basic functional trade-offs. Performance, efficiency, robustness, and flexibility are the key properties of all functional systems, including natural and artificial cognitive systems. Straight arrows represent trade-offs between properties; curved arrows represent trade-offs between different aspects of the same property.

Definitions

Performance: "The performance of a system is usually defined as its ability to produce an intended result (or some other roughly equivalent formulation)."
Efficiency: "The efficiency of a system is its ability to perform its function with minimal use of resources."
Robustness: "Robustness is the ability of a system to maintain performance in the face of perturbations. There are many possible kinds of perturbations, both external (e.g., physical damage, extreme events that exceed the system’s operating range) and internal (e.g., component failures, conflicts between subsystems). Cognitive systems are particularly exposed to perturbations caused by information corruption or noise."
Flexibility: "For cognitive systems, we suggest that flexibility can be r...
SGD's Bias
by johnswentworth — 2021-05-18T23:19:51.450Z

There’s a common hypothesis that stochastic gradient descent has some kind of built-in bias toward simpler models, or models which generalize better. (It’s essentially the opposite of the

NTK/GP/Mingard et al model

.) There is also a rough intuitive argument to expect such a thing a priori: sub-structures which are only useful sometimes will be lost on the occasions when they’re not useful, whereas sub-structures which are consistently useful will be retained. Conceptually, it’s similar to the

modularly varying goals

model in biological evolution.

Mathematically, it’s not too hard to show that SGD does indeed have a bias, and we can even write it down explicitly given some not-too-unreasonable approximations. This post will walk through that derivation, and give some intuition for where the bias comes from.

First Idea: SGD Is Approximately Brownian

“Brownian” here refers to Brownian motion, i.e. a random walk. In other words, the path taken by SGD looks qualitatively sort of like this:

2D Brownian motion with drift up and to the right.

The key idea is that each sample used to estimate the gradient is approximately independent (in the probability sense of the word), and the estimate is an average over samples, so the net effect of several steps is approximately normally distributed. That’s the defining feature of Brownian motion. (Alternatively, we can assume that steps are approximately independent and additive, which gets us to the same place with a little more generality but also a little more work.)

Let’s put some math on that.

We use SGD to choose θ to minimize EX[u(X,θ)]. Each step, we take n independent samples X1…Xn of X, use them to estimate the gradient, then take a step:

dθ=−ηn∑ni=1∇θu(Xi,θ)

… where η scales the step size. This step is an approximately-normal random variable, constructed from the IID random variables X1…Xn. It has mean −ηnEX[∇θu(X,θ)], and variance (ηn)2VarX[∇θu(X,θ)].

To formally represent this as a Brownian motion, we declare that the amount-of-time which passes during each SGD step is dt=ηn, which we assume to be small (i.e. we’ll approximate things to first order in dt). Then SGD’s path can be represented as

dθ=−EX[∇θu(X,θ)]dt+√ηnVarX[∇θu(X,θ)]dW(t)

… where W(t) is a

standard Brownian motion

, the analogue of a standard normal variable, and the square root is a m...
Does NASA Take Alien Life Seriously?
by ejacob — 2021-05-18T21:36:06.460Z

This post was inspired by recent media stories about UFOs (now rebranded as UAPs by the Very Serious People) and an episode of Neil Degrasse Tyson's Cosmos: Possible Worlds I watched on a Delta flight. Also this might look like a post about ETs but it's actually about institutional epistemic bias.

Preventing Alien Invasions

One of the job titles at NASA is "Planetary Protection Officer."

Image by Randall Munroe

The job description does not include fighting off aliens with high-concept science-fiction weapons[1]. The primary responsibility is to prevent Earth's scientific equipment from contaminating other heavenly bodies with microbial life or organic molecules. This is good science: If we find any evidence of life or its chemical precursors in space, we want to be sure that we aren't actually "discovering" stowaways from Earth brought by our own probes. It's also good conservationism: NASA has no desire to mess with possible ecosystems on other planets by introducing new species. The possibility of biological hitchhikers returning through Earth via sample return missions and then harming us is comparatively remote, but appropriate precautions are also taken for these types of missions.

From the NASA webpage about Planetary Protection:

Planetary Protection requirements for each mission and target body are determined based on the scientific advice of the Space Studies Board and on NASA or international policy guidelines. Each mission is categorized according to the type of encounter it will have (e.g., flyby, orbiter or lander) and the nature of its destination (e.g., a planet, moon, comet or asteroid). If the target body has the potential to provide clues about life or prebiotic chemical evolution, a spacecraft going there must meet a higher level of cleanliness and some operating restrictions will be imposed. Spacecraft going to target bodies with the potential to support Earth life must undergo stringent cleaning and sterilization processes and greater operating restrictions.

The episode of Cosmos that I watched mentioned this classification system and piqued my interest, so I read the page. There are five categories. Below are layperson-accessible summaries (based on my understanding) and examples of destination planets that would place missions in a particular category (taken directly from NASA).

Category I: Missions to bodies absolutely not interesting

...
Are there any methods for NNs or other ML systems to get information from knockout-like or assay-like experiments?
by J Bostock — 2021-05-18T21:33:38.474Z

As an example, suppose we have a bacterium, which we are treating as a black box, and we are training an ML system to predict various things it does (movement, secretions etc.) from its environment (chemicals around it, temperature). Instead of just measuring inputs and outputs of the cell, we might do one of two things:

1: Mutate a gene in the bacterium to see how its behaviour changes (knockout)

2: Measure the concentration of a certain protein in the bacterium under different conditions (assay)

Once we have done this, we know a little bit of the internal structure: it is no longer a pure black box. Are there any methods for enforcing this structure onto an existing ML model? (In the case of 1 we could add the knockout status of the bacterium as an extra input node but that seems to be missing the point, and there's no obvious way to do something similar for 2) Without this it seems like ML systems operating in the real world would be throwing away a lot of potential data. (I am mainly thinking of neural networks as these are undoubtedly the most commonly-used ML systems at the moment, but I would be very interested to hear about other systems)
Saving Time
by Scott Garrabrant — 2021-05-18T20:11:14.651Z

For the last few years, a large part of my research motivation has been directed at trying to save the concept of time—save it, for example, from all the weird causal loops created by decision theory problems. This post will hopefully explain why I care so much about time, and what I think needs to be fixed.

Why Time?

My best attempt at a short description of time is that time is causality. For example, in a Pearlian Bayes net, you draw edges from earlier nodes to later nodes. To the extent that we want to think about causality, then, we will need to understand time.

Importantly, time is the substrate in which learning and commitments take place. When agents learn, they learn over time. The passage of time is like a ritual in which

opportunities are destroyed and knowledge is created

. And I think that many models of learning are subtly confused, because they are based on confused notions of time.

Time is also crucial for thinking about agency. My best short-phrase definition of agency is that agency is time travel. An agent is a mechanism through which the future is able to affect the past. An agent models the future consequences of its actions, and chooses actions on the basis of those consequences. In that sense,

the consequence

causes

the action

, in spite of the fact that the action comes earlier in the standard physical sense.

Problem: Time is Loopy

The main thing going wrong with time is that it is “loopy.”

The primary confusing thing about Newcomb's problem is that we want to think of our decision as coming “before” the filling of the boxes, in spite of the fact that it physically comes after. This is hinting that maybe we want to understand some other "logical" time in addition to the time of physics.

However, when we attempt to do this, we run into two problems: Firstly, we don't understand where this logical time might come from, or how to learn it, and secondly, we run into some apparent temporal loops.

I am going to set aside the first problem and focus on the second.

The easiest way to see why we run into temporal loops is to notice that it seems like physical time is at least a little bit entangled with logical time.

Imagine the point of view of someone running a physics simulation of Newcomb’s problem, and tracking all of the details of all of the atoms. From that point of view, it seems like there is a useful sense in which the filling ...