trentmkelly/LessWrong-43k huggingface dataset:
Trace README
by johnswentworth — 2020-03-11T21:08:20.669Z

Trace is a tool for writing programs which read, write and reason about programs.
You can find it here. I wrote it as a tool for my own research, and I expect that others in this space may find the ideas interesting/useful as well. I'd be especially interested in new use-cases and other feedback!

Some kinds of things you might find Trace useful for:

Algorithms which operate on a computation graph, e.g. backpropagation, belief propagation, or other graphical inference algorithms

An intermediate data structure for static analysis, interpreters or compilers

A general-purpose non-black-box representation of objectives/constraints for optimization

A general-purpose non-black-box representation of world models for AI more broadly

Disclaimer for all of these: Trace is brand-new, and it was built with a focus on the core ideas rather than the engineering. Syntax is liable to change as we figure out what does and does not work well. Do not expect it to be easy/pleasant to use at this point, but do expect it to provide novel ways to think about programs.

One more warning: this doc is intended to be read start-to-finish. Trace does not really resemble any other tool I know of, and you will likely be confused if you just dive in.

What is Trace?

Trace is

A programming/modelling language embedded in a python library. For use as a human-facing programming language, Trace is pretty terrible, but it’s sometimes a necessary step for other use-cases.

A notation/data structure representing programs. For these use-cases, Trace is pretty good: compared to alternatives (e.g. abstract syntax trees), Trace offers a much more convenient representation of program structure.

A data structure representing the computation performed by an arbitrary program - i.e. the trace (aka execution graph aka computation graph) of a program. For this use-case, I do not know of any other tool which is anywhere near as powerful as Trace.

A prototypical use-case: suppose you want to test out a new inference algorithm. You can prototype the algorithm to operate on Trace data structures, which allows it to handle arbitrary programs (unlike e.g. pytorch graphs), with relatively little complexity (unlike e.g. python syntax trees). Then, you can write test-case world-models as programs in Trace notation. Those “programs” will themselves be fairly transparent Trace data structures, which your prototype algo

...
[AN #90]: How search landscapes can contain self-reinforcing feedback loops
by Rohin Shah — 2020-03-11T17:30:01.919Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Highlights

Demons in Imperfect Search

(John S Wentworth) (summarized by Asya): This post gives an analogy to explain optimization demons: a type of undesirable behavior that arises in imperfect search processes. In the analogy, a ball rolls down a hill trying to go as far down as possible, mimicking a gradient descent algorithm. The ball is benefited by random noise, but still basically only experiences local changes in slope-- it cannot see steep drop-offs that are a little off to the side. Small bumps in the hill can temporarily alter the ball's trajectory, and the bumps that are selected for are the ones that most effectively control its trajectory. In this way, over time the ball's trajectory selects for demons, twisty paths with high walls that keep the ball contained and avoid competing walls. Demons cause the ball to go down the hill as slowly as possible so that potential energy is conserved for avoiding competitor walls.

The general pattern this analogy is meant to elucidate is the following: In any imperfect search mechanism with a rich enough search space, a feedback loop can appear that creates a more-and-more perfect exploitation of the imperfect search mechanism, resulting in a whole new optimization process. The post gives several real world examples as proofs that this is a failure mode that happens in real systems. One example is metabolic reactions-- a chemical system searches by making random small changes to the system state while trying to minimize free energy. Biological systems exploit the search by manipulating the height of the barriers between low-free-energy states, raising or lowering the activation energies required to cross them. After enough time, some chemicals changed the barriers enough such that more copies of the chemicals were made, kicking off an unstable feedback loop that led to life on earth.

The post ends by posing an open question asking what about a system makes this kind of failure mode likely to happen.

Asya's opinion: I think it's worth spelling out how this is different from the failure modes described in

R

...
Why don't singularitarians bet on the creation of AGI by buying stocks?
by John_Maxwell — 2020-03-11T16:27:20.600Z

With the recent stock market sale, I've been looking over stocks to see which seem to be worth buying. (As background information, I'm buying stocks to have fun and bet my beliefs. If you believe in the efficient market hypothesis, buying stocks at random should perform roughly as well as the market as a whole, and the opportunity cost doesn't seem super high. Making a 40% return buying $ZM right before investors started paying attention to COVID-19 hasn't discouraged me.)

Standard investing advice is to stay diversified. I take the diversification suggestion further than most: A chunk of my net worth is in disaster preparation measures, and I'm also buying stock in companies like Facebook and Uber that I think have a nontrivial shot at creating AGI.

The stock market's "ostrich view" of AGI

AGI could be tremendously economically valuable. So one question about companies with AI research divisions is whether the possibility that they'll create AGI is already "priced in" to their stock. If the company's share price already reflects the possibility that they'll create this transformative technology, we shouldn't expect to make money from the singularity on expectation by buying their stock.

To investigate this question, let's examine Alphabet Inc's share price around the time AlphaGo defeated world Go champion Lee Sedol in March 2016. This has been referred to as a "Sputnik moment" that inspired the Chinese government to start investing billions of dollars in AI research. As a reminder of the kind of conversations that were being had at the time, here are some quotes from a Facebook post Eliezer Yudkowsky made:

...

– Rapid capability gain and upward-breaking curves.

“Oh, look,” I tweeted, “it only took 5 months to go from landing one person on Mars to Mars being overpopulated.”

...

We’re not even in the recursive regime yet, and we’re still starting to enter the jumpy unpredictable phase where people are like “What just happened?”

...

One company with a big insight jumped way ahead of everyone else.

...

AI is either overwhelmingly stupider or overwhelmingly smarter than you. The more other AI progress and the greater the hardware overhang, the less time you spend in the narrow space between these regions. There was a time when AIs were roughly as good as the best human Go-players, and it was a week in late January.

Here is Alphabet's his...
What is the best way to disinfect a (rental) car?
by Eli Tyre — 2020-03-11T06:12:32.926Z

I'm currently planning to rent a car and drive to another state.

I would like to disinfect that car as thoroughly as possible before spending 15 hours inside of it. Google gives me lots of suggestions for how to get rid of germs in a used car, but I'm hoping that I can get advice with better epistemic backing.

Does anyone have suggestions?

Thank you,

Eli
How useful are masks during an epidemic?
by Elizabeth — 2020-03-10T22:12:34.160Z

This was a pretty contentious part of the debate on the CDC's recommendations and has come up elsewhere as well: how effective are masks at preventing infection when worn by an uninfected individual? How about preventing spread when worn by an infected individual? Are there things you can do that make it more or less useful?

This is a thread for evidence to answer those questions and related ones. Obviously the more rigorous the better, but case studies like this one about one dude on a bus would be useful had they not been retracted (thanks to gwillen for pointing this out).
The Case for Privacy Optimism
by bmgarfinkel — 2020-03-10T20:30:02.152Z

(Cross-posted from my personal blog.)

This post is an edited transcript of a talk I recently gave on the past and future of privacy. I argue that the story may be a more positive and hopeful one than people often realize.

The talk stemmed from work I’ve done at the Centre for the Governance of AI. The people whose writing most influenced its ideas are Joan Feigenbaum, Aaron Segal, Bryan Ford, and Tyler Cowen. Mainly: This paper and this blog post. I’ve also especially benefitted from conversations with Andrew Trask.

Introduction

I think it’s fair to say that discussions of privacy issues tend to have a pessimistic edge to them. For a long time, the dominant narrative around privacy has been that it’s under threat. It’s either dying or it’s dead.

Here we have five decades of magazine covers announcing or predicting the death of privacy at the hands of technology. We also have the 1890 Harvard Law Review article “The Right to Privacy,” which is often seen as the starting point for modern political and legal discussions of privacy. The framing device for that article was that the camera and other new technologies posed severe threats to privacy. These threats were meant to be unprecedented enough to warrant, for the first time, the introduction of a new “right to privacy.” So it seems like we have been watching and working to stave off the death of privacy for a long time now.

Privacy Pessimism

The following narrative is a bit of a caricature, but I also do think it should be fairly recognizable. Something like it seems implicit in many discussions of privacy. The narrative goes like this: (1) People used to have a lot more privacy. (2) Unfortunately, over time, technological progress has gradually eroded this privacy. (3) Now new technologies, such as artificial intelligence, are continuing this trend. Soon we may have little or no privacy left.

Let’s call this narrative “Privacy Pessimism.”

Now, of course, the Privacy Pessimism narrative isn’t universally accepted. However, I think that even when people do decide to critique it, these critiques don’t often qualify as “heartening.” Here’s a quote from Privacy: A Very Short Introduction, pushing back against Privacy Pessimism. The author writes:

A requiem, however, is premature. The invaders are at the gate, but the citadel will not fall without a battle.

There is definitely an element of optimism here. But I st

...
Zoom In: An Introduction to Circuits
by evhub — 2020-03-10T19:36:14.207Z
Link: https://distill.pub/2020/circuits/zoom-in/

Chris Olah and the rest of the rest of the OpenAI Clarity team just published “Zoom In: An Introduction to Circuits,” a Distill article about some of the transparency research they've been doing which I think is very much worth taking a look at. I'll try to go over some of my particular highlights here, but I highly recommend reading the full article.

Specifically, I have previously written about Chris's belief that the field of machine learning should be more like the natural sciences in seeking understanding first and foremost. I think “Zoom In” is a big step towards making something like that a reality, as it provides specific, concrete, testable claims about neural networks upon which you might actually be able to build a field. The three specific claims presented in the article are:

Claim 1: Features

Features are the fundamental unit of neural networks. They correspond to directions [in the space of neuron activations]. These features can be rigorously studied and understood.

Claim 2: Circuits

Features are connected by weights, forming circuits. These circuits can also be rigorously studied and understood.

Claim 3: Universality

Analogous features and circuits form across models and tasks.

“Zoom In” provides lots of in-depth justification and examples for each of these claims which I will mostly leave to the actual article. Some highlights, however:

How do convolutional neural networks (CNNs) detect dogs in an orientation-invariant way? It turns out they pretty consistently separately detect leftward-facing and rightward-facing dogs, then union the two together.

How do CNNs detect foreground-background boundaries? It turns out they use high-low frequency detectors—which look for high-frequency patterns on one side and low-frequency patterns on the other side—in a bunch of different possible orientations.

What's particularly nice about “Zoom In”'s three claims in my opinion, however, is that they give other researchers a foundation to build upon. Once it's established that neural networks have meaningful features and circuits in them, discovering new such circuits becomes a legitimate scientific endeavor—especially if, as the third claim suggests, those features and circuits are universal across many different networks. From “Zoom In:”

One particularly challenging aspect of being in a pre-paradigmatic field is that there isn’t a shared sense of how to eval

...
COVID-19 article translations site aims to bridge the knowledge gap
by Mike Stop Continues — 2020-03-10T18:34:08.499Z

As anyone who spans two cultures know, it's often frustrating to see how wrong the media can get news in one or the other of their domains.

When faced with the elevated stakes of COVID-19 and the consequences of mismanagement imposed upon her family in China, my good friend and colleague took on the challenge of addressing the knowledge gap, setting up covid19readings.com.

The site, and its team of volunteer translators, is dedicated to surfacing quality journalism and first-person accounts of the virus, both to correct skewed perceptions in the English-speaking world and to help humanity at large weather the storm.

I hope it can be of value, at the very least, to the discussion here.
When to Reverse Quarantine and Other COVID-19 Considerations
by Benquo — 2020-03-10T16:37:41.717Z
Link: http://benjaminrosshoffman.com/when-to-reverse-quarantine-and-other-covid-19-considerations/

A lot of people in my social network have been trying to track news about the new coronavirus, COVID-19, which seems like a global pandemic that's going to kill a lot of people. I've found some of this overwhelming and difficult to figure out how to use, until I sat down with a few friends, over the phone, and worked out a simple analytic framework for thinking about some basic decisions.

I figured I'd put this up, not so much because anyone should use my numbers as-is, but because maybe the simplicity of the reasoning might help others like me get over a sort of feeling of helplessness about how to think about this sort of thing.

At the end, some less structured thoughts and links to other info sources. Please double check facts yourself before relying on them, though of course I'll fix any errors I find.

Reverse Quarantine Length

The simplest, most reliable measure available to me, to prevent myself from being infected with COVID-19 is to hole up in my apartment and isolate myself for some length of time. But, it's costly to do that - and much more costly for people who are in the short run reliant on income sources that require them to show up somewhere. So, what length of reverse quarantine has benefits that outweigh the costs?

I'm only counting selfish benefits here, not because that's all that matters, but to keep things simple. I encourage anyone who wants to, to extend this decision model elsewhere (or here in the comments).

Costs vs Benefits

Since I'm in the young-and-in-good-health category, it looks like the consensus estimate is a 0.2% chance of death if I'm infected. The costs of avoiding all potentially deadly risks are prohibitively high, but a 1 in 500 chance of dying if infected, for something that seems like it's a global pandemic, is pretty high as far as these things go. If I faced one risk like that each day, I'd probably die within a year. But "pretty high" doesn't tell me what precautions are worth it. Can I use numbers?

Let's say that a reverse quarantine is worth it on any day that it saves me more time in expectation than it costs. So, on each day, I have some probability of getting infected if I interact with the outside world - let's call that daily_infection_rate. And each day of reverse quarantine costs me some fraction of a normal day's value - l...
Vulnerabilities in CDT and TI-unaware agents
by PabloAMC — 2020-03-10T14:14:54.530Z

The aim of this post is illustrating the need to take into account decision-making and incentive considerations when designing agents. This post is also a proof that these considerations are important in order to ensure the safety of agents. Also, we will postulate that there exist some agents that are both robust to changing or having their reward function changed, although that will need a careful approach to incentive design and decision theory choosing.

The first agent we will consider is a (current Reward Function, Time Inconsistent aware, see in the second half of the post if you don't know what this means) agent that uses Causal Decision Theory (CDT). A review of different decision theories can be seen in this post. It is well known that Updateless Decision Theory (UDT) was created to correct the wrong decision a CDT agent would make when faced with Newcomb-like problems. Thus, the question we aim to answer is whether we can exploit the wrong decision making procedure in order to induce any changes in the value function of such an agent.

This is not exactly trivial since the agent could value very negatively to have its value function changed and thus opt out of games such as Newcomb. The example I propose is a modified version of Prisoner's dilemma (in which CDT is known to defect). Suppose the following problem:

It is year 2100 and Elon Musk managed to effectively colonise Mars. Meanwhile, an AI corporation, called Causal Corp, has deployed many CDT agents both in Earth and Mars. One day, Eve, known for being evil, codes a virus that if connected to one such CDT agent would arbitrarily modify the reward function. Eve makes two copies of such virus in memory sticks and sends them to arrive almost "simultaneously" to two CDT agents in Earth and Mars. With the memory stick there is a letter that tells the agents that they face a Prisoner Dilemma situation:

1. If both cooperate nothing will happen.

2. If one defects and the other cooperates, the first will receive a large amount of utilons and will subsequently modified to attack the second agent relentlessly substracting arbitrary value from the second. For example, the first agent might get cloned so that the first clone could carry on with its life (get same utility as it would normally do) + 1 utilon; and the second clone would attack the second CDT agent who has collaborated. So, i

...
Anthropic effects imply that we are more likely to live in the universe with interstellar panspermia
by avturchin — 2020-03-10T13:12:54.991Z

The sketch of the argument which favours panspermia.

1. There are around 10 billion potentially habitable planets in our galaxy.

2. Interstellar panspermia of small organisms will inseminate all of them with life in a few billion years.

3. Abiogenesis could be the main filter. Recent estimate of the minimum length of self-replicating RNA is around 100 bases. Totani thinks that only one of 10^100 Suns will generate correct RNA and start RNA-world. Thus only one of 10^80 of universes has life. Thus it looks like Rare Earth is true and we are alone.

4. However, there are many different universes in the multiverse and some of them may allow panspermia.

5. The density of planets with life in a pasnpermia-universe is 10 billion times more than such density in a non-panspermia universe as each life will colonise the whole galaxy.

6. We don't know the relation of non-panspermia to panspermia-universes. If it is less than 10 billion, we are more likely to be in panspermia-universe based on anthropic considerations, because each of them will have 10 billions times more attempts to create a civilization of a planet with life.

7. If we are in the panspermia universe, it means that the Great filter from the Fermi paradox is likely ahead, as major abiogenesis filter doesn't prevent many planets in our galaxy from having life. Rare Earth hypothesis is false locally.

8. It means that either there are a lot of aliens nearby or almost all civilizations self destruct. (Or there are some other earlier great filters like a higher level of asteroid impacts etc.) Not good from x-risks view point.
Estimated risk of death by coronavirus for a healthy 30 year old male ~ 1/190
by Roko Jelavić — 2020-03-10T12:32:02.246Z

Epistemic status: high uncertainty.

Even if I'm wrong this model could be useful to others, you can plug your own numbers. Please correct my assumptions in the comments if you think they're wrong or you have new/better data.

Chance of infection

We don't know what percentage of people will get infected. Some experts say 40-70%. Around 50% of cases are symptomatic, but for young people it's closer to 90%.[1] Asymptomatic cases seem not to be infectious. The rate of infection among young people is being underreported because the symptoms are milder and not detected as covid but are mistaken for common flu.[1] According to the stats, the chance of infection is very small for children and 4x lower for 20yo than 60yo. [2] This is wrong, but we can accept it and move on because having mild symptoms is not fatal and hence equivalent to not having covid at all. From now on, by "infection" I mean "noticable infection". The chance of being asymptomatic is already factored in. Note: say 100% of people get exposed and the risk of infection grows linearly from 0yo to 60yo, that would give us total of about 50-70% of an infected population. Best effort estimate of infection chance:

infection = 0.4

Lack of treatment

We don’t know what percentage of people who have coronavirus die. That’s because most of infected people so far had received hospital care. Once the pandemic spreads, most people will not receive hospital care because hospitals will be full. What would the mortality rate for untreated patients be?

"every demographic has approximately equal hospitalization rates, which other sources suggest are 15% to 20%."[2][3]

Those "demographics" are crude estimates, but still 0-49 had the same admittance rate as older. With regards to those 20% who require hospitalization:

"5% of people who are diagnosed with Covid require artificial respiration. Another 15% need to breathe in highly concentrated oxygen - and not just for a few days."[2]

Some studies show 6% of people in critical condition, while still 18% of people have a severe case.[4] It's not clear what exactly "severe" means. The best I can estimate with that is:

base_rate_untreated_mortality = 0.12

Adjusting for demographics

How does age affect that? The data on that is messy but seems like <50yo have around 3x smaller chance of a...