trentmkelly/LessWrong-43k huggingface dataset:
Understanding Machine Learning (I)
by Rafael Harth — 2019-12-20T18:22:53.505Z

Understanding Machine Learning is another book on Miri's research guide. I've previously written about Topology and Linear Algebra Done Right. With this one, my write-up ended up looking a lot more like a self-contained introduction into the field and a lot less like a review. I've decided to go with that, and make this a sort of shortened version of the book.

Throughout this series, I will be using bold and italics to mean "this is the first official mention of a new Machine Learning term", whereas normal italics means regular emphasis.

The Learning Model

The book begins with a sort of mission statement. Given a domain set X and a target set Y, we wish to find a function h:X→Y, which we interchangeably call a hypothesis or a predictor or a classifier. In the example used in the first chapter, X represents the set of all papayas, and Y represents a two-element set where one element is the label "tasty" and the other the label "not-tasty". To compress this into something manageable, we represent papayas by a set of features, in this case color and softness, which we both map onto the interval [0,1], and we represent the two labels by 1 and 0. I.e. X=[0,1]2 and Y={0,1}. Now the idea is that, if we learn such a predictor, we can apply it to any new papaya we encounter, say on the market, and it will tell us whether or not it is tasty before we've spent any money on it.

This model seems quite general. It covers chess, for example: one could let X be the set of all possible states of the chessboard and Y be the set of all moves; then we want h to map each x∈X onto the best possible move. In fact, it is hard to see which solvable problem could not be modeled this way. Any time we think we can in principle solve some problem, it would consist of us looking at some stuff and then presenting the solution in some way. Now we just let the set of all possible instances of such stuff be X and the set of all possible solutions be Y, and we have an instance of this model.

One would have to put in a lot more work to fully formalize this problem – and this is exactly what the book does. Recall that X is called the domain set. If Y is finite as it is for the papayas example, we also call it the label set.

We model the environment, which creates our labeled domain points (x,y) by a probability distribution D over...
Sections 5 & 6: Contemporary Architectures, Humans in the Loop
by JesseClifton — 2019-12-20T03:52:43.629Z

This post is part of the sequence version of the Effective Altruism Foundation's research agenda on Cooperation, Conflict, and Transformative Artificial Intelligence.

5 Contemporary AI architectures

Although the architectures of TAI systems will likely be quite different to existing ones, it may still be possible to gain some understanding of cooperation failures among such systems using contemporary tools [1]. First, it is plausible that some aspects of contemporary deep learning methods will persist in TAI systems, making experiments done today directly relevant. Second, even if this is not the case, such research may still help by laying the groundwork for the study of cooperation failures in more advanced systems.

5.1 Learning to solve social dilemmas

As mentioned above, some attention has recently been devoted to social dilemmas among deep reinforcement learners (Leibo et al., 2017; Peysakhovich and Lerer, 2017; Lerer and Peysakhovich, 2017; Foerster et al., 2018; Wang et al., 2018). However, a fully general, scalable but theoretically principled approach to achieving cooperation among deep reinforcement learning agents is lacking. In Example 5.1.1 we sketch a general approach to cooperation in general-sum games which subsumes several recent methods, and afterwards list some research questions raised by the framework.

Example 5.1.1 (Sketch of a framework for cooperation in social dilemmas).

The setting is a 2-agent decision process. At each timestep t, each agent i receives an observation oti; takes an action ati=πi(oti) based on their policy πi (assumed to be deterministic for simplicity); and receives a reward rti. Player i expects to get a value of Vi(π1,π2) if the policies π1,π2 are deployed. Examples of such environments which are amenable to study with contemporary machine learning tools are the "sequential social dilemmas'' introduced by Leibo et al. (2017). These include a game involving potential conflict over scarce resources, as well as a coordination game similar in spirit to Stag Hunt (Table 1).

Suppose that the agents (or their overseers) have the opportunity to choose what policies to deploy by simulating from a model, and to bargain over the choice of policies. The idea is for the parties to arrive at a welfare function w which they agree to jointly maximize; deviations from the policies which maximize the welfare function will be punishe

...
Should We Still Fly?
by jefftk — 2019-12-20T02:00:01.640Z

I've seen a lot of discussion about plane travel from a climate
perspective lately, with people arguing that we should try to
restructure our lives to fly much less. Avoid business travel,
vacation closer to home, visit relatives less, etc. After looking at
the numbers, though, I think this mostly doesn't make sense.

Let's take an example round trip flight from Boston to LA. I've flown
this many times for work and to visit relatives, and it's maybe on the
long end for a vacation flight. Taking into account that emissions at
high altitude are worse than at ground level, that's about 1.3T CO2e
[1].

The thing is, 1.3T isn't that much! For example, carbon offsets are
about $10/T, so this would add just ~$13 to your ~$500 round-trip
flight. Or, if you don't trust offsets and would rather use the full
social cost of carbon, that's ~$55/T (Wang
et. al. 2019) or ~$72. Or, if you want to go all the way to direct air
capture, that's ~$160/T (Keith
et. al. 2018) or ~$210.

If you consider a typical BOS-LAX business trip, with, say, $500
for flights, $500 for lodging, $100 for food, and 14hr time lost to
travel, a carbon cost of even $210 is rarely going to make the
difference on whether the travel is worth it. Even for a vacation,
where people tend to be more price sensitive, it's a factor but not
nearly the biggest factor.

Climate change is a real problem, and I'm not saying we shouldn't
change anything. I favor a stiff carbon tax, high enough to cover the
full social cost of emissions. But even under a high tax, most of the
things people fly for today would still be worth flying for.

[1] I tried three
different
calculators
and got 1.16T, 1.4T, and 1.36T.

Comment via: facebook
Reason and Intuition in science
by Unknown128 — 2019-12-20T01:35:58.546Z

Pbfgva Iynq Nynznevh (translate using rot13.com) who has a BA in Mathematics writes the following:

Do you imagine that men of genius or, let’s say, men of science in history walked around clear-headed, “disenchanted,” reasonable, with the tight-assed attitude of the science cultist and materialist? No great discovery has ever been made by the power of reason. Reason is a means of communicating, imperfectly, some discoveries to others, and in the case of the sciences, a method of trying to render this communication certain and precise. But no one ever made a discovery through syllogisms, through reason, through this makeshift form of transmission. Great mathematicians saw spatial relations, as great physicists saw and to some extent felt physical relations. In contemplation of mathematical forms, there is almost a physical feel of geometric relations, and all mathematicsat bottom is about geometric relations even when it doesn’t seem so. Compare the Euclidian proof of the Pythagorean theorem, based on syllogism, which helps you understand nothing that’s actually going on, with the imagistic proof of the three squares, that makes you perceive, physically perceive even in your body, why this theorem is true. Gauss, so beloved even by the tedious scientistic goblins that even Google gave him a cartoon, is famous to have said something like, “I got it…now I have to get it.” Meaning, he had seen and felt the fundamental spatial relation he was searching, but now he had to translate it into the imperfect language of mathematics for others. Thus all mathematics and all science in general—mathematics is only the prototype and most precise of the sciences—is about the definitions, not about the proof, not about the process or —absurd!—the “algorithm.” All great scientific discoveries, supposedly the great works of “reason,” are in fact the result of intuitions and sudden grasp of ideas. And all such sudden grasp and reaching is based on what, in other circumstances, would be called a kind of religious intoxication: it depends on a state of the mind where the perceiving part of the intellect is absolutely focused, limpid, yet driven by the most relentless energy, an energy to penetrate. Direct perception is already “intellectualized” and in fact ...
Causal Abstraction Intro
by johnswentworth — 2019-12-19T22:01:46.140Z

I haven't been terribly satisfied by the first few posts in this sequence; they don't do a very good job introducing things. I think part of the problem is the format, so I decided to invest in a high-end studio and try making a video instead.

It's about 10 minutes, covers similar material to the first few posts, but IMO does a better job communicating what's going on.

Feedback on the format is appreciated (including "didn't watch the video but would have read a normal post" or vice versa). So far I think the video provides better explanation per-unit-effort, and I lean toward doing more of them. Obviously both the setup and the postprocessing were pretty low-investment on this one; I'll probably put a bit more effort into production if I'm going to do this regularly.
We run the Center for Applied Rationality, AMA
by AnnaSalamon — 2019-12-19T16:34:15.705Z

CFAR recently launched its 2019 fundraiser, and to coincide with that, we wanted to give folks a chance to ask us about our mission, plans, and strategy. Ask any questions you like; we’ll respond to as many as we can from 10am PST on 12/20 until 10am PST the following day (12/21).

Topics that may be interesting include (but are not limited to):

Why we think there should be a CFAR;
Whether we should change our name to be less general;
How running mainline CFAR workshops does/doesn't relate to running "AI Risk for Computer Scientist" type workshops. Why we both do a lot of recruiting/education for AI alignment research and wouldn't be happy doing only that.
How our curriculum has evolved. How it relates to and differs from the Less Wrong Sequences. Where we hope to go with our curriculum over the next year, and why.

Several CFAR staff members will be answering questions, including: me, Tim Telleen-Lawton, Adam Scholl, and probably various others who work at CFAR. However, we will try to answer with our own individual views (because individual speech is often more interesting than institutional speech, and certainly easier to do in a non-bureaucratic way on the fly), and we may give more than one answer to questions where our individual viewpoints differ from one another's!

(You might also want to check out our 2019 Progress Report and Future Plans. And we'll have some other posts out across the remainder of the fundraiser, from now til Jan 10.)

[Edit: We're out of time, and we've allocated most of the reply-energy we have for now, but some of us are likely to continue slowly dribbling out answers from now til Jan 2 or so (maybe especially to replies, but also to some of the q's that we didn't get to yet). Thanks to everyone who participated; I really appreciate it.]
[Part 1] Amplifying generalist research via forecasting – Models of impact and challenges
by Bird Concept — 2019-12-19T15:50:33.412Z

This post covers our models of impact and challenges with our exploration in amplifying generalist research using forecasting. It is accompanied by

a second post

with a high-level description of those models, and more detailed description of experiment set-up and results.

Many of the world’s most pressing problems require intellectual progress to solve [1]. Finding ways to increase the rate of intellectual progress might be a highly promising way of solving those problems.

One component of this is generalist research: the ability to judge and synthesise claims across many different fields without detailed specialist knowledge of those fields, in order to for example prioritise potential new cause areas or allocate grant funding. For example, this skill is expected by organisations at the EA Leaders Forum to be one of the highest demanded skills for their organisations over the coming 5 years (

2018 survey

,

2019 survey

).

In light of this, we recently tested a method of increasing the scale and quality of generalist research, applied to researching the industrial revolution [2], using Foretold.io (an online prediction platform).

In particular, we found that, when faced with claims like:

“Pre-Industrial Britain had a legal climate more favorable to industrialization than continental Europe”

And

“Pre-Industrial Revolution, average French wage was what percent of the British wage?”

a small crowd of forecasters recruited from the EA and rationality communities very successfully predicted the judgements of a trusted generalist researcher, with a benefit-cost ratio of around 73% compared to the original researcher. They also outperformed a group of external online crowdworkers.

Moreover, we believe this method can be scaled to answer many more questions than a single researcher could, as well as to have application in domains other than research, like grantmaking, hiring and reviewing content.

We preliminarily refer to this method as “amplification” given its similarity to ideas from Paul Christiano’s work on Iterated Distillation and Amplification in AI alignment (see e.g.

this

).

This was an exploratory project whose purpose was to build intuition for several possible challenges. It covered several areas that could be well suited for more narrow, traditional scientific studies later on. As such, the sample size was s...
[Part 2] Amplifying generalist research via forecasting – results from a preliminary exploration
by Bird Concept — 2019-12-19T15:49:45.901Z

This post covers the set-up and results from our exploration in amplifying generalist research using predictions, in detail. It is accompanied by

a second post

with a high-level description of the results, and more detailed models of impact and challenges. For an introduction to the project, see that post.

___

The rest of this post is structured as follows.

First, we cover the basic set-up of the exploration.

Second, we share some results, in particular focusing on the accuracy and cost-effectiveness of this method of doing research.

Third, we briefly go through some perspectives on what we were trying to accomplish and why that might be impactful, as well as challenges with this approach. These are covered more in-depth in

a separate post

.

Overall, we are very interested in feedback and comments on where to take this next.

Set-up of the experiment

A note on the experimental design

To begin with, we note that this was not an “experiment” in the sense of designing a rigorous methodology with explicit controls to test a particular, well-defined hypothesis.

Rather, this might be seen as an ”exploration” [3]. We tested several different ideas at once, instead of running a unique experiment for each separately. We also intended to uncover new ideas and inspiration as much as testing existing ones.

Moreover, we proceeded in a startup-like fashion where several decisions were made ad-hoc. For example, a comparison group was introduced after the first experiment had been completed; this was not originally planned, but later became evidently useful. This came at the cost of worsening the rigor of the experiment.

We think this trade-off was worth it for our situation. This kind of policy allows us to execute a large number of experiments in a shorter amount of time, quickly pivot away from bad ones, and notice low-hanging mistakes and learning points before scaling up good ones. This is especially helpful as we’re

shooting for tail-end outcomes

, and are looking for concrete mechanisms to implement in practice (rather than publishing particular results).

We do not see it as a substitute for more rigorous studies, but rather as a complement, which might serve as inspiration for such studies in the future.

To prevent this from biasing the data, all results from the experiment are public, and we try to note when decisions were made post-hoc. ...
When Goodharting is optimal: linear vs diminishing returns, unlikely vs likely, and other factors
by Stuart_Armstrong — 2019-12-19T13:55:28.954Z

In this post, I'll argue that some of the behaviours that seem to be clear examples of the Goodhart problem, are in fact not. And they are, in some sense, perfectly optimal.

And, somewhat conversely, that the reason we fear Goodharted results are because we implicitly know that there is some key information about the reward functions that we have not been able to include.

Intro: a Goodhart example?

Consider the following situation, with a robot that can go left or right one square each turn (or stay put):

The reward function UL rewards the robot by 1 each turn it spends in the L (left) square, while UR similarly rewards the robot for each turn spend in the R square.

Suppose the robot puts 50.1% probability on UL being correct, and 49.9% probability on UR. Its optimal policy is then to go left, and stay in the left square forever.

So far, so Goodhart; this feels wrong that, because of such a small probability different, UR is essentially completely irrelevant. But, in fact, both UL and UR think that that policy is optimal!

That statement needs clarification, obviously. The reward UR would much prefer that the robot went right; failing that, it would prefer the robot alternated between L and R. But still, in a sense, it sees that the policy of going left is optimal.

To see how, define two reward functions to be mutually exclusive if one is always zero when the other is non-zero, like UL and UR are. Then, suppose you asked the following question to a UR-optimiser:

"Suppose that a robot is uncertain between UR and one other reward function, which is mutually exclusive with UR. The robot has no further way of distinguishing between the two. The UR may be the most probable function or the least probable, both options are equally likely. Is it optimal for the robot to maximise the highest probability reward function (equivalently, to maximise the weighted sum of the reward functions)?"

And the answer to that, from the UR-optimiser, is a bold YES!! That is an optimal policy, and better, from UR's expected perspective, from moving back and forth between L and R (since that looses reward while on the centre square).

Given those constraints, both reward functions would agree that policy is optimal ex ante (before knowing the probabilities); UR would only disagree with it ex post (after knowing the probabilities).

When to Goodhart, when not to

There are two other optimal pol

...
Neural networks as non-leaky mathematical abstraction
by George3d6 — 2019-12-19T12:23:17.683Z
Link: https://blog.cerebralab.com/Neural_networks_as_non-leaky_mathematical_abstraction

I find it rather weird that mathematics is usually thought to people on a historical basis. It's likely possible for a 4th century Alexandrian mathematician to teach it in a modern primary school and get decent results. An 18th century Venetian tutor could carry the curriculum all the way up to high school. I'll grant they might need a week or two to brush up on sets and some modern notation.

On the other hand, Darwin or Huxley would most get fired within the week if they tried to teach high school biology. The great Aristotle would be unable to introduce a 1st grader to any of the sciences; Despite his works in the precursors of modern science remaining foundational more than a millennium after his death.

Mathematical knowledge retains validity over time to a great extent. When it seldom become irrelevant or poorly formulated, it’s nonetheless still valid.

The obvious hypothesis as to why this is true is that mathematics is rather “pure” compared to many/all other fields.

That's boring, I’ve come up with a second, more interesting and much more scandalous hypothesis.

Mathematical abstractions are very leaky

In case the terms is unfamiliar to you, a leaky abstraction is an abstraction that leaks the details that it's supposed to abstract away.

Take something like sigma summation (∑), at first glance it look like an abstraction, but it’s too leaky to be a good one. Why is it leaky ? Because all the concept it “abstracts” over need to be understood in order to use the sigma summation. You need to "understand" operations, operators, numbers, ranges, limits and series.

For another example take integrals (∫), where the argument for leakiness can be succinctly state. To master the abstraction one needs to understand all of calculus and all that is foundational to calculus (also known as "all of math" until 60 years ago or so).

A leaky abstraction is the rule rather than the norm in mathematics. The only popular counterexample that comes to mind are integral transform (think Laplace and Fourier Transforms). Indeed, any mathematician in the audience might scoff at me for using the word "abstraction" for what they think of as shorthand notations.

Why ? Arguably because mathematics has no such thing as an "observation" to abstract over...
CFAR: Progress Report & Future Plans
by Adam Scholl — 2019-12-19T06:19:58.948Z

Context: This is the first in a series of year-end updates from CFAR. This post mainly describes what CFAR did over the past two years, and what it plans to do next year; other posts in the series will describe more about its mission, financials, and impact.

(12/29/19: Edited to mention CFAR's mistakes regarding Brent, and our plans to publicly release our handbook).

Progress in 2018 and 2019

For the last three years, CFAR held its total number of workshop-days[1] roughly constant:

But the distribution of workshops has changed considerably. Most notably, in 2018 we started co-running AIRCS with MIRI, and since then we’ve been allocating roughly ⅓ of our staff time[2] to this project.[3]

AIRCS

AI Risk for Computer Scientists (AIRCS) workshops are designed to recruit and train programmers for roles doing technical alignment research. The program name isn’t perfect: while most attendees are programmers, it also accepts mathematicians, physicists, and others with significant technical skill.

We co-run these workshops with MIRI—the staff is mostly a mix of current and former CFAR and MIRI staff, and the curriculum is roughly half rationality material and half object-level content about the alignment problem. So far, MIRI has hired six AIRCS graduates (more are in the pipeline). While these workshops are designed primarily to benefit MIRI’s research programs, the curriculum includes a variety of other perspectives (e.g. Paul Christiano’s IDA), and AIRCS graduates have also subsequently accepted positions at other alignment orgs (e.g. FHI, Ought, and the OpenAI alignment team).

We’re excited about these workshops. So far, 12 AIRCS graduates have subsequently taken jobs or internships doing technical safety research,[4] of which (from what we can tell), AIRCS played a non-trivial causal role[5] in 9 of these outcomes. AIRCS also seems to help participants learn to reason better about the alignment problem, and to give MIRI greater familiarity with candidates (thus hopefully improving average quality of hires).

Overall, we’ve been impressed by the results of this program so far. We’re also excited about the curricular development that’s occurred at AIRCS, parts of which have been making their way into our curriculum for other programs.

European Workshops

Another notable strategic change over this period was that we decided to start running many more workshops in Europe. I

...