trentmkelly/LessWrong-43k huggingface dataset:
Should theories have a control group
by George3d6 — 2020-04-24T14:45:33.302Z
Link: https://blog.cerebralab.com/Should_theories_have_a_control_group

(audio version)

It is a commonly held belief that natural selection is a process by which random modifications to a type of organism accumulate over time, based on their ability to increase their chances of survival and reproduction.

It is also a commonly held belief that an omnipotent god created all life as we know it according to this grand design.

Given the circular nature of metaphysics, I think we can just as well postulate a 3rd hypothesis:

All of the seemingly “random” changes that sustain evolution were dictated by god’s grand design, however, god was such a mediocre designer that we can’t separate his “grand design” from what would be just “random mutations”.

I find this third theory for the emergence of organic life to be far more spiritually satisfying than the other two.

Anyway…

Recently I stumbled upon this paper, which basically shows that various state of the art techniques for neural architecture search do no better than random.

It’s pretty experimentally thorough as far as ML papers go, which is to say not very better than most. They ran benchmarks on two play datasets which are used in a lot of papers (including many of the NAS papers): PTB and CIFAR-10. Also, they seem to have at least attempted to control for some of the inherent randomness:

we run 10 experiments with a different initialization of the sampling policy.

Again, this might not sound like an ideal experiment from which one can draw conclusive statements, but this is the ML literature I’m working with… as far as I’m concerned these guys are trying too hard.

Also, the reddit discussion includes a bunch of people citing various papers and personal experiences that have similar claims, but I couldn’t be bothered to follow up on those links.

Regardless, I think there’s a good reason why they are trying so hard to use best experimental practices, because the claims they are making, taken at face value, seem quite outrageous:

The most popular architecture search methods (DARTS, NAO, ENAS and Bayesian NAS) are no better than searching randomly. Not in the overall accuracy of the best model, not in the accuracy variation depending on the initial condition and (obviously) not in the time that it takes to actually train.

Is it quite the same as trying to throw away the work o...
The Puzzling Linearity of COVID-19
by Radford Neal — 2020-04-24T14:09:58.855Z
Link: https://radfordneal.wordpress.com/2020/04/23/the-puzzling-linearity-of-covid-19/

Why, recently, for many countries (and the world), are total cases and total deaths increasing in nearly a straight line, on a linear plot? Please post a comment here or there if you have a clue about this puzzle.
The Inefficient Market Hypothesis
by lsusr — 2020-04-24T07:33:08.252Z

The efficient-market hypothesis (EMH) is the idea that there are no hundred-dollar bills lying on the sidewalk because someone smarter than you would have picked them up by now. The EMH is a good tool for most people.

If you're smart enough then you should reverse this advice into "There are hundred-dollar bills lying on the sidewalk". If you are a genius then you should reverse it to the extreme. "There are hundred-dollar bills[1] lying around all over the place."

Hundred-dollar bills lying on the sidewalk are called "alpha". Alpha is a tautologically self-keeping secret. You can't read about it in books. You can't find it on blogs[2]. You will never be taught about it in school. "You can only find out about [alpha] if you go looking for it and you'll only go looking for it if you already know it exists."

Where should you look?

Abstractions

A system is only as secure as its weakest link. Cracking a system tends to happen on an overlooked layer of abstraction[3].

It's easier to install a keylogger than to break a good cryptographic protocol.

It's easier to disassemble a computer and read the hard drive directly[4] than to crack someone's password.

The best attacks (those requiring the least work) happen on an separate dimension of orthogonality entirely.

The easiest way to talk to someone powerful is just to call zir company and ask by first name[5].

Won't this technique stop working now that Tim Ferris has published it in a bestselling book? Not necessarily. Quantum mechanics has been public knowledge for decades yet most people can't do it. The hard part of pickpocketing isn't finding pockets to pick.

Perhaps you don't need to talk to anyone rich and powerful. That is a good problem to have.

I think you should find a problem that's easy for you to solve. Optimizing in solution-space is familiar and straightforward, but you can make enormous gains playing around in problem-space.

― What Startups Are Really Like by Paul Graham

Problem-space tends to have higher dimensionality than solution space.

Case study

According to Joel Spolsky, the best programmers have the ability "to think in abstractions, and, most importantly, to view a problem at several levels of abstraction simultaneously." Also according to Joel Spolsky, a business is an "abstraction [that] exists solely to create the illusion that the daily activities of a programmer (design and writing code,

...
Do dreams reinforce prior beliefs?
by ajithr — 2020-04-24T04:13:45.513Z

I've observed the following happening a couple of times to me now.

I wake up in the morning from a dream; in my dream, would be somebody I had reasons to 'hate' in the past, but I have been consciously trying in recent times to question my reasons to 'hate' them. That person would do something in the dream they hadn't even remotely done in real life to me, leaving me feeling really bad, and reinforcing all the 'hate' I had for them. When I'm awake from the dream, all I'm primarily left with is a very strong feeling of hate.

Now if I don't question why I felt so towards them, right after waking up (that is by recollecting the dream and realising its absurdity, which is an effortful exercise), I might've carried that reinforced feeling/reinforced belief and gone about with my day, which I suspect I did almost every time in the past.

This is not unlike the F.R.I.E.N.D.S episode in which Phoebe is mad at Ross and she can't remember why, and in the end she attributes it to a dream she had.

That really concerns me and I've decided to be more vigilant to largely unexplained feelings of 'hate' or similar feeling/belief.
Retracting on Facebook
by jefftk — 2020-04-24T01:50:02.580Z

Sometimes I'm in a Facebook discussion and someone says something they
wish they hadn't. Maybe it turns out to be wrong, or misleading, or
insensitive. One way I've seen this poorly handled is posting a
follow-up comment:

The problem is that the top-level post is still there, and Facebook
will often choose to show it to people without showing them the
reply. Or people will read the big text and skim over the reply.

At the other extreme, full deletion has different downsides:

If you're deleting something with replies, those comments disappear,
even ones people may have put a lot of work into. Plus people are
even less likely to know you retracted your claim.

For simple retractions, where once people know you don't endorse it
anymore it's no longer harmful, an edit can work well:

It's not confusing, and people can still read the comments if the
discussion is helpful.

If you really don't want people to gloss over your retraction, you can
edit to remove it fully:

If people really want to see the context, they can look at the edit
history:

In extreme cases, like if you accidentally outed someone, changing the
privacy setting on your post to "only me" lets you keep anyone else
from seeing it:

This way if someone has a comment they wrote that they'd like to still
have, they can write to you and ask for a copy.

If you have a comment in a nested thread where you need to clear the
edit history, delete your comment and post a
new one like "I deleted my comment above because...".

If you have a top-level comment you need to delete, there aren't any
great options. The best you can do is manually screenshot the thread
in case someone asks for their comments, and then delete it.

So, please do retract things when needed! Just be thoughtful about it.

Comment via: facebook
My Covid-19 Thinking: 4/23 pre-Cuomo Data
by Zvi — 2020-04-23T21:40:00.891Z

Previous week: My Covid-19 Thinking: 4/17

Other previous foundational Covid-19 thoughts here: On R0, Taking Initial Viral Load Seriously, Seemingly Popular Covid-19 Model is Obvious Nonsense

Spreadsheet I’m using to look at data is here, if you’d like to look.

Epistemic Status: Usual warning that I’m not an expert, just someone trying to figure things out as best he can, and am doubtless making tons of mistakes.

I’m going to try making this a weekly update while there are interesting things to say. I’ll update these thoughts as the week goes by, then publish each Friday.

There’s a whole bunch of different stuff going on. The sections play into my whole picture, but they’re also mostly distinct, so you can skip anything you don’t care about. This is the things I felt were sufficiently relevant to write down.

This is a weird post, because as I was finishing it up, Cuomo came out with antibody test results and that’s going to change everything. So these thoughts are what I got before the antibody tests and before the 4/23 data. Next time I’ll deal with the antibody results in full.

What Is Happening?

New York’s situation is rapidly improving. Other places in America, not so much. At a minimum, not so clear.

Previously I’ve been grouping New York and New Jersey, since New Jersey has the second-most infections and is right next to New York.

However, with New Jersey not getting better and New York rapidly improving, this no longer makes as much sense. I’m going to isolate only New York. I’ll still have the New Jersey columns and Ex-NJ columns, but I’ll move them off to the right in the spreadsheet.

New York alone peaked its positive test rate around 50%, and it’s now down to 26.8%. I don’t think we went too far above the point where test positive numbers stop increasing, and our testing is modestly increased since then but not dramatically so. Clearly things are much better. The naive calculation says down from peak by half, the second-level one says it’s somewhat more than that but hard to tell how much more.

We cut things somewhat more than in half in 22 days. If we assume serial interval of 5 days, which seems like the minimum, that’s 4.4 intervals. Then we get an average R0 for this period of about 0.85. Serial interval might be substantially longer, but can’t be much shorter. Our reasonably confident bounds are therefore something like 0.75 to 0.9. The more I look at i...
Do you trust the research on handwriting vs. typing for notes? 
by  — 2020-04-23T20:49:19.731Z

For a while now, I've heard the claim that "studies show" that note-takers remember content better if they take notes by hand versus on a computer. I previously took this claim on face value in part because this was before I'd heard about the replication crisis and also because I'd had personal experiences that I believed supported this claim.

In light of the replication crisis and recent experiences, I've come to be more skeptical of this research. I started to look at some of the research pop science articles on the topic cite and am skeptical of the work I've looked at so far.

In the example I link to above, they have subjects perform two tasks, a recall and recognition task for words they handwrote or typed with a multiplication task in between writing and recalling/recognizing. They find a not significant difference between recalled words for handwriting vs. typing and a barely (p-value .03) significant difference between recognized words for the two groups. However, if you look at the standard deviations for the means for the two tasks, you'll see that each mean is in the other's 1-SD range.

Furthermore, the task they describe is simple but not necessarily that relevant to what's really going on when someone takes notes on a lecture / talk. They intentionally used semantically meaningless words (for understandable reasons) whereas real-life talks hopefully have higher-level meaning and themes.

ETA (after initial posting): Just found another paper that a few pop-sci articles seem to cite. This paper covers three experiments, which are all more realistic than the one I described above. I'm only going to discuss the first here.

The first had participants watch TED talks, take notes on them (either on a laptop or by hand) and then answer a combination of "factual" and "conceptual" questions about them. At a high level, they interpret the results of this experiment as showing that laptop note-takers did as well as the by hand note-takers on factual questions but worse on conceptual questions. (However, this is only the case after they convert to z-scores. That is, the differences between raw grades are not significant.) They also find that a difference between laptop note-takers and by hand note-takers is that the laptop note-takers write more and tend to copy the lecturers words more exactly. When they control for these factors, they find a much weaker effect o

...
My experience with the "rationalist uncanny valley"
by Thomas Kwa — 2020-04-23T20:27:50.448Z

Epistemic status: Very Uncertain. Originally written with the belief rationality has harmed my everyday life; revised to reflect my current belief that it's been somewhat positive. The tone of this article, and some of the content, may constitute self-punishment behavior, signaling, or frustration rather than genuine belief.

Introduction

I'm currently a male second-year undergrad in computer science. I am not clinically depressed. My first exposure to the rationalist community was largely limited to reading HPMOR and ~40% of the original Sequences around 2013-2014; I've had rationalist/EA friends continuously since then; in mid-2019 I started following LW; in March 2020 I read almost every scrap of COVID-19 content. I'm not sure how to evaluate my strength as a rationalist, but I feel epistemically slightly weaker than the average LW commenter and guess that my applied skills are average for non-rationalists of my demographic.

Others described the phenomenon of the "rationalist uncanny valley" or "valley of bad rationality" as early as 2009:

It has been observed that when someone is just starting to learn rationality, they sometimes appear to be worse off than they were before.

I've known about the rationalist uncanny valley since 2013, and was willing to accept some temporary loss in effectiveness. Indeed, before March 2020, the damage to my everyday life was small and masked by self-improvement. However, with isolation, my life has ground to a halt, in part due to "rationalist uncanny valley" failure modes, and failures I'm predisposed to which rationality training has exacerbated. Looking back, one of these also happened during my exposure to the community in 2014. This post is an attempt to characterize the negative effects exposure to rationality has had on my life, and is not representative of the overall effect.

Examples

1. Bad form when reading LW material

I'm very competitive and my self-worth is mostly derived from social comparison, a trait which at worst can cause me to value winning over maintaining relationships, or cause me to avoid people who have higher status than me to avoid upward comparison. In reading LW and rationalist blogs, I think I've turned away from useful material that takes longer for me to grasp because it makes me feel inferior. I sometimes binge on low-quality material, sometimes even seeking out highly downvoted posts; I s

...
What's Hard About Long Tails?
by johnswentworth — 2020-04-23T16:32:36.266Z

Epistemic status: more confident in the conclusion than in any particular model.

Suppose we have a fire insurance company with $2.5M in monthly revenue, and claims following a power law:

Most months, claims are low
Once every ~1 year, they see around $5M in claims
Once every ~10 years, they see around $25M in claims
In general, every ~N^1.5 months, they see around N million dollars of claims, for large N.

Ignoring bankruptcy laws and other breakdowns of the model, what’s the expected profit of this company in a random month?

Correct answer: negative infinity. This is a classic black swan scenario: there’s a well-defined distribution of always-finite events with infinite expected value. Sooner or later this company is going to go broke.

(Math, for those who partake: when 1<k≤2, the integral ∫∞1dxxk is finite but the integral ∫∞1xdxxk is infinite. So, if events have probability density proportional to 1xk, then they’ll have a well-defined probability distribution, they’ll always be finite, but they’ll have infinite expectation.)

Let’s change the scenario a bit - let’s make our insurance company more proactive. They want to go out and prevent those big fires which eat into their profits. Whenever there’s a big fire, they find some way to prevent that sort of fire in the future. Now, different sizes of fire tend to result from different kinds of problems; preventing the once-a-year fires doesn’t really stop the once-a-decade or once-a-century fires. But within a few years, the stream of roughly-once-a-year $5M claims has all but disappeared. Within a few decades, the stream of roughly-once-a-decade $25M claims has all but disappeared.

Iterative improvement - fixing problems as they come up - has eliminated 95% of the fires. Now what’s the expected profit of this company in a random month?

Correct answer: still negative infinity. The black swans were always the problem, and they haven’t been handled at all. If anything, the problem is worse, because now the company has eliminated most of the “warning bells” - the more-frequent fires which are big but not disastrous.

Moral of the story: when the value is in the long tail, iterative improvement - i.e. fixing problems as they come up - cannot unlock the bulk of the value. Long tails cannot be handled ...
Sex, Lies, and Canaanites
by Jacob Falkovich — 2020-04-23T16:26:10.458Z

Cross-posted, as always, from Putanumonit.

The Origin of Consciousness According to Jaynes

On the right side of our cranium
Lived the voice of gods and goddesses
Who laid siege to mighty Ilium
And instructed brave Odysseus

Zombie Incas and Assyrians
Knew no humor or morality
Just unquestioning obedience
To their own bicamerality

To this day subconscious processes
Guide the bulk of our activity
So it was. Unconscious Rameses
Had no need for subjectivity

What then, our subjective faculty,
Conscious mind, what is it better for?
It’s an analog reality
We build up in verbal metaphor

We can think in counterfactuals
Play in our imaginations
A replacement for the actual
Audible hallucinations

In our heads reigns quiet loneliness
Where once voices rang, ephemeral
That’s the origin of consciousness
In the mind that was bicameral

The Bicameral History Hypothesis

This is my summary in poetry form of the first two parts of The Origin of Consciousness in the Breakdown Mind,

which we began discussing in the last post

. It is appropriate, I think, since the historical and anthropological review that takes up the bulk of the book starts with an epic poem, the Iliad. This post will look for consciousness and subjectivity in another book and another poem. One that suggests a different view of the consciousness of our ancestors.

Jaynes’ theory is that up until the end of the second millennium B.C., humans did not have a subjective mental space where they could observe their own thoughts. Instead, they did most things on habitual autopilot (as you would drive a familiar road while completely zoned out). When faced with a novel situation, the information was processed by their right hemisphere and relayed to the left as the hallucinated voice of authority figures: living and dead ancestors, personal and tribal gods. These voices were immediately obeyed, resulting in appropriate action.

This thesis is supported by a wealth of evidence from archaeology (the hierarchical structure of ancient cities and proliferation of idols), anthropology (the widespread custom of burying corpses with a supply of fresh food), and recorded history (the complete submission of the Inca Empire upon the capture of its king Atahualpa by Pizarro). Of course, it is hard to know how cherry-picked is the evidence presented in the book. History has left us myriad hints, all of which could be interpreted in different w...
Analysis of COVID-19 superspread events (linkpost)
by Jay Molstad — 2020-04-23T16:17:08.276Z

FYI, an analysis of COVID-19 superspread events is posted here. I just found it online; I was not involved with it.

Short version: The major mode of transmission (in such events, YMMV) seems to be ballistic projection of dropules while coughing, singing, or speaking loudly. Masks are recommended. Religious activities, parties, and funerals were particularly risky. Superspreading events were often upper-class, but IMHO this is probably associated with the demographics of foreign travel (as the virus is a recent import).
Training Regime Day 19: Hamming Questions for Potted Plants
by Mark Xu — 2020-04-23T16:00:10.354Z

Introduction

Sometimes, when thinking about a difficult problem, it helps to pretend that the problem is much simpler - to abstract away the complexity and consider only the core elements.

Hamming Questions are questions designed to get you to think deeply about your life problems. However, you are a human being. Human beings are complicated. Human beings have complicated life problems. Complicated life problems are both hard to think of and hard to solve.

Potted plants are simple. Potted plants have simple life problems. Simple life problems are easy to think of, although they still might be difficult to solve.

You are now a potted plant. This will help you think of major life problems by abstracting away complexity.

Questions

Water

One of the things that plants need is water. Naively, you're supposed to water your plants. However, if you water your plants too much, they suffer from being overwatered. (Maybe potted plants aren't so simple after all.)

Suppose that you-as-a-plant is receiving too much water. What is the water a metaphor for?

Are you spending too much time doing any particular activity? Consuming too much of a certain type of content? Thinking too much about a particular concept?

Sometimes, there's a problem that you think you can solve with more of something. Sometimes, this is correct. Sometimes, it is wrong. Sometimes, you should stop adding more and start adding less.

Light

Another thing plants need is light. Naively, you just put your plant next to a window. However, plants have different sides. If you don't rotate your plant, one side will grow faster than the other and it will become lopsided. (Maybe potted plants aren't so simple after all.)

Suppose that you-as-a-plant is receiving light in an uneven distribution. What is the light a metaphor for?

Are you getting too much of a resource from a particular person? Are you too dependent on a certain activity? Is there part of your routine that's too load-bearing?

Sometimes, there's something that you want, so you try to get a lot of it. Sometimes, you don't realize that there are different forms of that thing. Sometimes, optimizing for getting a lot of that thing means you only get the form that's easy to get. Sometimes, you need to drink something that's not soylent.

Fruit

A thing that plants produce is fruit. Naively, you allow your plant to produce fru...