trentmkelly/LessWrong-43k huggingface dataset:
Utility functions and probabilities are entangled
by Thomas Kwa — 2022-07-26T05:36:26.496Z

Originally posted as an EA forum comment.

Suppose that there are two effective altruist billionaires, April and Autumn. Originally they were funding AMF because they thought funding AI alignment would be 0.001% likely to work and solving alignment would be as good as saving 10 billion lives, which is an expected value of 100,000 lives, lower than they could get by funding AMF.

After being in the EA community a while, they switched to funding alignment research for different reasons.

April updated upwards on tractability. She thinks research on AI alignment is 10% likely to work, and solving alignment is as good as saving 10 billion lives.
Autumn now buys longtermist moral arguments. She thinks research on AI alignment is 0.001% likely to work, and solving alignment is as good as saving 100 trillion lives.

Both of them assign the same expected utility to alignment -- 1 billion lives. As such, they will make the same funding decisions. So even though April made an epistemic update and Autumn a moral update, we cannot distinguish between them from behavior alone.

This extends to a general principle: actions are driven by a combination of your values and subjective probabilities, and any given action is consistent with many different combinations of utility function and probability distribution.

As a second example, suppose Bart is an investor who makes risk-averse decisions (say, invests in bonds rather than stocks). He might do this for two reasons:

He would get a lot of disutility from losing money (maybe it's his retirement fund).
He irrationally believes the probability of losing money is higher than it actually is (maybe he is biased because he grew up during a financial crash).

These different combinations of probability and utility inform the same risk-averse behavior. In fact, probability and utility are so interchangeable that professional traders -- just about the most calibrated, rational people with regard to probability of losing money, and who are only risk-averse for reason (1) -- often model financial products as if losing money is more likely than it actually is, because it makes the math easier.[1]

Implication

When observing that someone's behavior has changed, it's not obvious what changes are value drift vs. epistemic updates. You have to have some information besides behavior alone, or you'll have this extra degree of freedom in making i...
How Promising is Theoretical Research on Rationality? Seeking Career Advice
by Aspirant223 — 2022-07-26T01:08:02.310Z

As I come to the end of my undergraduate degree in philosophy, I am now faced with the decision of what to do next. I have spent a lot of time reading 80 000 hours, but have begun to wonder whether parts of their advice might not apply to me, or if they might be undervaluing certain kinds of research. For example, in their article on philosophy academia, they suggest that many people who are thinking about going to grad school for philosophy should instead study something like computer science, economics, or pursue a career in public policy. The problem is that I am much stronger verbally than I am quantitatively, and my scores on aptitude tests reflect this. So, subjects like computer science and economics are off the table, and I assume that one’s success in policy careers is going to depend (at least in part) on mathematical skills. The fact that I live in a smaller, less influential country could also affect the impact I would have if I went into politics or public policy.
I recently stumbled on a few LessWrong resources, such as What are the open problems in Human Rationality?, and some of the problems in this thread seem really interesting. It also seems that a person might make progress on these without strong quantitative abilities (though I will consider a counterargument to this point below). For instance, some people think that akrasia could be a promising problem to work on, and a recurring issue in the LessWrong literature is whether it makes sense to conceive of oneself as containing multiple sub-agents (in some sense), and if so, what the advantages and disadvantages of different theoretical views on this are. This is also the kind of thing you can get funding to work on for a couple of years in my country (from a philosophy department), without taking on any debt.
One advantage of doing so would be that certain aspects of this path could be really motivating. Though it is unlikely that I could rise to the level of certain heroic figures in the LessWrong community, even a small chance of one day becoming like some of my heroes would be really exciting. If I went to law school or worked for a government agency, I think this would be much less likely (though I welcome pushback on this idea— maybe I am underestimating the impact I could have on these paths).
I will now consider what I take to be the strongest counterargument to me pursuing this kind...
Alignment being impossible might be better than it being really difficult
by Martín Soto — 2022-07-25T23:57:21.488Z

Epistemic status: Thinking out loud.

TL;DR: If alignment is just really difficult (or impossible for humanity), we might end up with an unaligned superintelligence which itself solves the alignment problem, gaining exponentially more power. If it is literally impossible, the superintelligence might see its capabilities capped in some regards.

In many discussions about misalignment, the examples of what would constitute dangerously powerful capabilities for an agent to have involve fine-grained and thorough understanding of its physical context[1]. For instance, in the ELK report the following deception technique is considered: deploying undetected nanobots that infiltrate humans' brains and have their neurons fire at will (I will refer to this example throughout, but it's interchangeable with many others of similar spirit). Of course, very detailed knowledge about each particular brains' physical state must be known for this, which implies huge amounts of data and computations. This dynamic knowledge has to be either:

All contained in (or enacted directly by) the agent: This seems implausible for this kind of overly detailed specifications. Granted the agent can have a very good probabilistic model of human psychology which it exploits (just as it can model other parts of the physical world). But brainhacking more than a few people probably requires an amount of data (and system sensors near the scene, and so on) too big even for this kind of systems (accounting for the placement of almost every neuron in the present and future, etc.). This is inspired by Information Theoretic intuitions that, even with near-future hardware, any simulation of reality with that much detail will be too costly (information cannot be compressed much further, the most efficient way to simulate reality is by far reality itself, etc.)[2].
Somehow spread over systems complementary to the agent: This would very probably involve creating systems to which to delegate computations and decisions about specific parts of the physical world[3]. These systems with local knowledge will have to maximize a certain state of the world that the main agent wants to attain. If they deal with tasks as complex as manipulating humans, they can be expected to require independence and agency themselves, and so the main agent will have to solve the alignment problem for them. Failing to do so would probably b

...
Protectionism in One Country: How Industrial Policy Worked in Canada
by Davis Kedrosky — 2022-07-25T22:08:07.756Z
Link: https://daviskedrosky.substack.com/p/protectionism-in-one-country-a9a

TL;DR: the Canadian National Policy of 1879-1895 was a successful instance of deliberate infant industry protection (IIP). Selective targeting of industries with close domestic substitutes led to rapid output expansion, productivity growth, and price declines with little cost to consumer welfare. Nineteenth-century Canada is a good example of how a heavily agricultural, resource-dependent economy can use trade policy to develop a native manufacturing sector despite pressures toward primary product specialization. Main points:

Canadian protectionism worked -- it created internationally competitive industries at little cost to the domestic economy and forced US manufacturing firms to relocate inside the tariff wall.

It's possible that the result is idiosyncratic to the Belle Epoque world, in that Britain served as a "free trade sink" for Canada's export-facing economy.

IIP did not promote overall structural transformation, but mostly because it had to keep up with resource booms in the mineral and agricultural sectors.

Protection did not make Canada a less resource-dependent economy, but rather succeeded in onshoring more raw material processing

Two special announcements before we dive in. First, I’ve renamed (well, actually just removed the old placeholder title) the newsletter Great Transformations, which contains at least two book references and sort of reflects the general topics that I like to write about.

Second, I’ve added a paid subscription feature so that you can contribute to this project. The essays are staying in front of the paywall, but you’ll get a bit of additional content, including some enormous reading lists that I’m curating and weekly research posts. Maybe some other fun things in the future. The main idea, though, is that I’d like to derive some income from this project to make the time commitment worthwhile. If you can find the funds to support me, I’d really appreciate it.

Subscribe now

Also, since this is the only time I’m asking for money in the body of the newsletter, I’d be grateful if you shared around this particular edition.

A quick note: if you are receiving this post for the second time, I apologize. Substack failed to send out this issue originally and is not responding to my support requests, so I’ve decided to cut my losses and re-send it.

The basic idea behind infant industry protection is that you, as a nascent manufacturer, need t...
Mistakes as agency
by pchvykov — 2022-07-25T16:17:53.762Z

Note: this is cross posted from my blog, where the videos are posted which I can't upload here - take a look at those here.

We’re all somehow familiar with the concept that a person's actions can be “too perfect” – to the point of seeming creepy, or mechanical, or even dangerous. Classically this comes from sci-fi portrayals of humanoid robots, but can also just be an experience of watching someone do something so well that we get a strange feeling.

Early in my PhD, I spent a year in a cognitive science research group at MIT thinking about what it is that makes us perceive some things as “alive” or “agent-like,” and other things as “inanimate” or "mechanical." The classic motivation for this comes from this simple video made by researchers back in 1944. When viewers were asked to describe the video, they would talk about a fight, an abduction, a rescue, a love story – but none would say that there were some triangles and a circle moving around on a screen. The lab I worked with ran various experimental studies with young children where they would show them different sorts of behaviors of some simple shapes and checked whether the child viewed these shapes as “objects” or “agents.” I tried to develop and benchmark some simple theory for this.

The hypothesis I was working on was that the key distinguishing factor was the apparent “planning horizon.” Whenever we view some dynamics, we try to understand and infer its causes. We hypothesized that if the behavior could be explained in terms of a very short-term planner, then it would look like an object (e.g., ball rolling down hill), while if long-term planning was required to explain the dynamics, it would look like an agent.

To test this, I looked at a simple toy system: a "point-agent" exploring a utility landscape. The video here shows the motion in a 1D landscape with two local utility optima (negative utility on the y-axis). In this setting, if the short-term planner is to look like the motion of an object, then Newton’s equations need to be recovered as the short-plan limit of some more general control theory. This turns out to be the case if we set the reward function of the planner to try to minimize accelerations and maximize utility, integrated across its planning horizon. As a nice bonus, when the plan is short but not 0, the first correction just looks like an additional damping term, and so still ...
My Bitcoin Thesis @2022 - Part 1
by aysajan — 2022-07-25T15:49:25.067Z

Note that this is a pure educational post. Opinions expressed in this post are mine and mine only! they are definitely not financial advice. Even though potential future investment was a motivation for me to look into bitcoin, I neither recommend nor oppose investing in bitcoin. Always do your own research and reach your own conclusion.

Throughout this post, I use “Bitcoin” (in capital) to represent the peer-to-peer network and “bitcoin” (lower case) or “BTC” to represent the network digital currency with monetary value.

The Birth of Bitcoin and Where It Is Now

It was October 31, 2008.

Satoshi Nakomoto

published a white paper describing a peer-to-peer digital currency, titled “Bitcoin: A Peer-to-Peer Electronic Cash System”. The first sentence of the paper’s abstract says the following:

“A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution.”

Let’s fast forward to July 24, 2022. As I am writing this post, I am looking at the two charts below (my own creation) with my eyes wide open.

With its price reaching an all time high of $69,044 in November 2021 and close to a billion daily trading volume in US dollars, bitcoin sure has come a long way in such a short period of time. If we asked someone in 2010, 2015, or even 2018, it would be very unlikely that they would have foreseen what bitcoin had accomplished as a digital currency. It is truly fascinating to see that something with zero value can grow at this exponential level in a little more than a decade’s time, creating a wealth that is equivalent to about 9% of

total gold value

in the world. In terms of the Bitcoin network, it has grown exponentially as well, including huge growth in total number of transactions, unique wallet addresses, etc. (

https://www.blockchain.com/charts

provides an excellent collection of statistics on the current status of the Bitcoin network.)

Source:

https://en.wikipedia.org/wiki/File:BTC_vs_fiat_M1_and_gold.svg

As an outsider who did not pay attention to bitcoin until recently, I find the information available on the Internet about Bitcoin is filled more with noise than signal. Bitcoin is also one of those things that the majority of the public have strong opinions about. As such, we have bitcoin maximalist on one spectrum and bitcoin-zero-value-opinionis...
The Reader's Guide to Optimal Monetary Policy
by Ege Erdil — 2022-07-25T15:10:51.010Z

I've been seeing more discussion of monetary policy recently, I suspect in large part due to the wave of high inflation that's sweeping the globe right now. As such, I thought this is a good time to make a post about the wonderful website The Reader's Guide to Optimal Monetary Policy.

This website gives you access to a collection of hundreds of macroeconomics papers that are designed to specifically answer the question "what is the optimal monetary policy?" The results are then presented in a scatter-plot format, along with various properties of the optimal policy recommended in each paper, such as the first and second moments of inflation for the optimal policy.

Now that I've sufficiently advertised the website, I'll first discuss what monetary policy is about and what goals it tries to achieve, and then I'll explain the fundamental questions and tradeoffs that come up when we try to come up with an optimal monetary policy. I'll then link this to the estimates from the hundreds of papers listed in the website and broadly how each paper comes up with the recommendations they end up endorsing.

If you're already familiar with what monetary policy is about then feel free to skip the next section.

What is monetary policy?

Monetary policy is fundamentally about the issuer of a money using their power of issuing or retiring currency in circulation to achieve various aims they find to be desirable. When a country has its own currency, this currency is (usually) managed by the central bank. The central bank has the power to issue[1] new currency or to retire existing currency by buying it back and removing it from circulation, and as counterfeit money printing is illegal, they are the only institution to legally have this power.

Implementation

While central banks fundamentally exercise control over the money they manage, they can do this in ways that are somewhat less transparent. The most popular way to implement monetary policy nowadays is some form of nominal interest rate targeting. The simplified version of this is that the central bank announces a nominal[2] interest rate target, say 3% per year. The central bank then stands ready to accept unlimited deposits from banks and to make unlimited loans to banks at this interest rate.

Ideally, as the central bank has unlimited capacity to issue or retire currency, they can keep accepting deposits or making loans until a...
AGI Safety Needs People With All Skillsets!
by Severin T. Seehrich — 2022-07-25T13:32:27.011Z

(Cross-posted to EA Forum)

For quite a while, I had two major misconceptions about careers and volunteering in AGI safety. And I know others have them, too:

Only people with a background in computer science or math can help with AGI safety.
Only people

smarter than I

can help with AGI safety.

This is false, and here’s why.

If we want to win the race against AI capabilities research, we indeed need as many geniuses with a background in computer science or adjacent fields as we can get. But these people would have a hard time doing their work without others who set up the organizations they work at, who

build the funding ecosystem

, who run the retreats where they meet like-minded people, who design their websites, who wipe their office floors. In addition, they would profit from having people who do their taxes, can advise them on visa issues,

help them with health problems

, give them

productivity coaching

, and so on and so on. Over the last years, AGI safety has grown into a large ecosystem of individuals and organizations. This ecosystem depends on far more than only those who do the research.

Here is a list of skill bottlenecks I’ve found in conversations with other AI safety field builders:

Operations people

PAs

Discussion group facilitators (e.g. for

AGISF

)
Event organizers (e.g. for

AI Safety Camp

, conferences)
Lawyers (e.g. for helping to set up organizations, and advising on the tax implications of novel systems we’re designing like Impact Markets)
Founders (An alignment project incubator is in the making. Until it is ready to go public, you might want to fill out 80k’s

census of everyone who could ever see themselves doing longtermist work

)

Communicators

(written and verbal)
Everything HR-related

Mobility/Visa support
Hiring
Salary-related things: Calculating living cost, taxes, health insurance, …

Bookkeeping

Bookkeepers
Accountants

Payable
Receivable

Auditors

Software engineers

(to build projects like

Stampy

, Idea Marketplace, Impact Markets, etc.)
Particularly valuable are people who can get a broad overview of the current AGI alignment ecosystem and kickstart projects which can absorb people in a scalable manner.

…and you, too, can find your niche, even if you don’t bring any of these particular skill sets. For inspiration, here is a list of less standard career paths which turned out valuable for the ecosystem:

YouTubers (

Rob Miles

)
Architects (

Tereza Flidrova

)
Cooking

...
Is there any evidence that handwashing does anything to prevent COVID?
by mukashi — 2022-07-25T07:34:31.211Z

I live in Japan now.

Japan has been very successful in dealing with COVID, even if the cases are now on the rise, the number of dead people relative to the size and mean age of the population remains very low compared with other countries.

I guess there are different factors that can explain this, one being that people used to wear face masks in Japan well before COVID, which made it possible to keep the number of cases low until vaccines were available. However, I am starting to find annoying the social ritual of cleaning your hands. My impression is that there is little to no evidence that washing hands does anything to prevent the spread of COVID and that people keep doing it because it has become a cultural norm. Am I missing something important? I don't want to criticize something that might have some value and I am just not aware of it.
Opening Session Tips & Advice
by CFAR!Duncan — 2022-07-25T03:57:49.731Z

Meta

CFAR ran many, many workshops.

After each workshop, there would be feedback from the participants, and debrief discussions among the staff. We would talk about what had worked and what hadn't, what we wish had been said or done, what we would try differently in the future, etc.

Often, what resulted was a new addition to the opening session. Opening session, at a CFAR workshop, was largely about expectation setting, and getting everyone on the same page—making sure everyone knew what they were getting into, and what was going to be asked of them, and why.

The "tips and advice" section of opening session was often framed as "things past participants said, at the end, that they wished they'd been told at the beginning."

(This was often but not always literally true.)

Little snippets of wisdom about how to engage with the content, what to watch out for in one's own experience, where to put one's attention, etc. Often the staff would create their own tips and advice based off of watching classes fail, or watching individual participants "bounce" off the workshop, and trying to figure out why.

There were something like two dozen distinct tips, at various points, of which four or five would be presented at a given workshop. Some were added, some were removed, others morphed or mutated, yet others got more deeply baked into the structure of the workshop and were no longer needed in opening session.

Below is a selection of some of the most important and longest-lasting opening session tips. They are presented here for two purposes:

Despite the fact that this is an online sequence and not a workshop, the tips nevertheless contain valuable wisdom about how to engage with the content, and some specific ways that trying to do so tends to go wrong for people.
They may be useful advice in other contexts, such as conferences or events that you yourself may run, in the future.

Be Present

One key element of getting the most out of an experience is being present. This includes physically showing up, but it also includes having your mind in the room and your background thoughts focused on the content. The more you’re taking calls and answering texts and keeping up with social media and what’s going on back home, the more you’ll remain in your ordinary mental space, continuing to reinforce the same habits and patterns you’re here to change. There’s a sort of snowball ...