trentmkelly/LessWrong-43k huggingface dataset:
Why you should minimax in two-player zero-sum games
by Nisan — 2020-05-17T20:48:03.770Z

Disclaimer: I'm surely reinventing the wheel here.

Last week I wrote that I was dissatisfied with von Neumann and Morgenstern's argument for playing the minimax strategy in a two-player zero-sum game. Here's an argument I like:

We have two players A,B who play strategies a,b. A is trying to maximize V(a,b). Since it's a zero-sum game, B is trying to minimize V.

A's minimax strategy is a∗:=argmaxaminbV(a,b)

B's minimax strategy is b∗:=argminbmaxaV(a,b)

The minimax theorem gives us v∗:=maxaminbV(a,b)=minbmaxaV(a,b)

We assume that A believes that no matter what A does, B will be epistemically and instrumentally rational. Epistemic rationality means that B's factual beliefs are accurate — i.e., B's beliefs conditional on playing b are accurate if b is the strategy that B actually plays; but B's beliefs conditional on playing other strategies (his properly counterfactual beliefs) are unconstrained. Instrumental rationality means that if B believes an action is better than all other actions, then B will take that action.

I claim that given the above assumption, it's instrumentally rational for A to play minimax. Here's why:

B believes that if he plays minimax, V(a,b)≤v∗. This follows from the definitions of b∗ and v∗.

If B is epistemically rational, he knows the value of V(a,b). If V(a,b)>v∗, then B would be instrumentally irrational (because he knows he would be better off playing b∗). So if B has both kinds of rationality, V(a,b)≤v∗.

Since A believes B is epistemically and instrumentally rational, A believes that V(a,b)≤v∗, no matter what strategy A plays.

Also, A believes that if she plays minimax, V(a,b)≥v∗. This follows from the definitions of a∗ and v∗.

Thus A believes that minimax is at least as good as any other strategy.

Note that A can still play something other than minimax, as long as she's sure that V(a,b)=v∗.

This argument does not justify playing minimax in non-zero-sum games or games with more than 2 players. In those games it's not generally true that one player is trying to minimize another player's reward.

If you squint, this is basically the argument that von Neumann and Morgenstern use, with all the steps and assumptions written out. And if you formalize this argument, I guess you'd get Kripke semantics for games.
Outbuilding Thoughts
by jefftk — 2020-05-17T12:10:02.533Z

Behind our house there's a small building:

The previous owners had used it for storage, but it's in bad shape.
There was a hole in the roof when we bought it, which has slowly been
getting larger:

There are cracks in the wall, which look like they might be from
settling:

Though at least it doesn't look too different from 5 1/2 years ago:

It doesn't look like it has a proper footing. In this part of the
country the top few feet of ground freeze every winter, so you need to
go down four feet to get something stable. It looks to me like this
is built on cinderblocks, and I would be surprised to learn that they
dug a four-foot trench for them.

We've mostly ignored it since we bought the house five years ago; there was so much else that needed doing. Now that I have more time, we're starting to think about
what might make sense to do with it. First step is taking
measurements:

This is a scale drawing, because that seemed fun. I measure the
external dimensions as 10'10" by 16'8", which is consistent with the
176 sqft that the assessor's database has for it.

The main options are fixing it, replacing it, removing it, and leaving
it alone. Because of the foundation issue I suspect fixing it doesn't
make much sense unless we just wanted to use it for storage though I
could talk to some contractors. Replacing it could make sense if we
could have something that was actually useful there. Removing it
would give us more space in the back yard, though I don't know if we
would do much with it. Leaving it alone keeps our options open a bit,
though at some point it would start to get in danger of falling over.

Somerville adopted new
zoning in December 2019. While the previous zoning would not
allow most existing buildings to be built today, the revised zoning
tries to be a closer match for the city. I think our current shed is
allowed under 10.2.2.t:

A roofed structure used as a storage space. The
following standards apply:

Sheds may be up to twelve (12) feet in height.

Except that it's too close to the rear lot line: I measure
18" from the back fence, and 10.2.1.e requires three feet.

There are two other categories we could potentially have a backyard
structure like this under. One is an "outbuilding" under 10.2.2.o,
which has a bit more permissive use:

A free-standing, fully-enclosed struct

...
Learning to build conviction
by benkuhn — 2020-05-16T23:30:01.272Z

People sometimes tell me that they want to join a startup, so that they can learn how it works, and eventually start one themselves. I usually end up suggesting that they skip straight to step 2 and start one themselves.

Why is that? Isn’t it better to learn from someone else’s mistakes than to have to make all of them yourself? At least for me, the answer’s been sometimes yes, but sometimes no.

For my first few years as an engineer, I was lucky to work with some mentors who taught me a ton about engineering. Things like:

How to name things well, and why that’s so important.

How to organize code so that it’s easy for future readers to navigate.

How to factor code into cohesive, decoupled modules.

How to keep my software designs simple and understandable.

How to write documentation that other people can understand.

By giving me fast, frequent feedback on these things, they taught me much more quickly than I could learn on my own. Eventually I’d end up building models of particular people in my head—I could ask myself “what would Ping think of this name?” or “how would Abeer factor this code?”

Of course, my predictions wouldn’t be perfect, because I only partially understood the mental models that my mentors were applying when they gave me feedback. But I’d predict correctly, say, 95% of the time, which was enough to help me become a much better engineer.

After a few years, though, I ended up in a situation where I didn’t have access to those mentors. We’d split Wave into two divisions, and I ended up in the smaller division (“mobile money”) where I was usually the most technically opinionated engineer.

I spent a long time building the mobile money team and codebase without much feedback from more-experienced engineers. That required me to make a bunch of higher-stakes, longer-term decisions on my own:

How to prioritize between different potential large-scale code improvement projects.

What kind of hiring bar is reasonable, and how to build an interview process that reliably upholds that bar.

How to decide whether a non-boring technology is worth adopting.

How to staff teams so that they have the right mix of skills and personalities to gel and build great things together.

When to keep working on the plan for a tricky technical project, and when to bias towards action and ship it.

How to help coworkers navigate towards the role that reso

...
Do any mammal species exhibit an immune response in some of the herd in response to the infection in other herd members?
by jmh — 2020-05-16T17:33:24.946Z

I am not even sure how to search for an answer on that question -- tried a simple search but results seemed off topic.

The question is prompted from two rather different observations. Long ago I read an article suggesting trees "talked". The sense in which they talk was chemical signals. Trees that were infested with some disease or parasite produces chemicals that were then picked up by nearby trees. The other trees then started producing the defense response to the threat infecting the other tree.

A recent story (via MR) suggests 10% of Boston has the antibodies. So one might take that to mean 10% of Boston was already infected. But the same study says only 1 in 40 are currently infected. These two statistic may be fully inline with antibodies = infection and we just have a lot of asymptomatic people.

However, what if human immune system have some response elements similar to the tress and we respond to signals from others being infected as well as the infection itself?

Anyone know if that is just a crazy thought or not?
What are your greatest one-shot life improvements?
by Mark Xu — 2020-05-16T16:53:40.608Z

Sometimes, people have life problems that can be entirely solved by doing one thing. (doing X made my life 0.1% better, PERMANENTLY!) These are not things like "This TAP made me exercise more frequently", but rather like "moving my scale into my doorway made me weigh myself more, causing me to exercise more frequently" - a one-shot solution that makes a reasonable amount of progress in solving a problem.

I've found that I've had a couple of life problems that I couldn't solve because I didn't know what the solution was, not because it was hard to solve - once I thought of the solution, implementation was not that difficult. I'm looking to collect various one-shot solutions to problems to expand my solution space, as well as potentially find solutions to problems that I didn't realize I had.

Please only put one problem-solution pair per answer.
Making a Crowdaction platform
by B Jacobs — 2020-05-16T16:08:11.383Z

In 2017 Eliezer Yudkowsky wrote:

Coordination isn’t as simple as everyone jumping simultaneously every time one person shouts “Jump!” For coordinated action to be successful, you need to trust the institution that says what the action should be, and a majority of people have to trust that institution, and they have to know that other people trust the institution, so that everyone expects the coordinated action to occur at the critical time, so that it makes sense for them to act too.

That’s why we have policy prediction markets and… there doesn’t seem to be a word in your language for the timed-collective-action-threshold-conditional-commitment…hold on, this cultural translator isn’t making any sense. “Kickstarter”? You have the key concept, but you use it mainly for making video games?

I thought this was very clever and was happy he broadcasted this idea to a community of capable, well-off computernerds. "I can't wait to see what this community cooks up" I thought. We are currently halfway into 2020 and the dream has not yet been realized. Let's see where we stand.

CollAction

I've searched the web and the closest I could come is CollAction. These people have gone ahead and build a functional website with actual projects on it. The makers of this site call these kinds of websites 'Crowdacting websites'. Since 'inadequate equilibria' didn't name this idea I shall be calling it that from here on out.

The site allows you to upload your own projects (with their approval) and allows you to join other projects. The visually clean lay-out shows how many people have already joined a project and what the desired threshold is. The projects have clear deadlines and goals and once they are reached they are closed forever. I encourage you to check it out for yourself since it's a good way to tickle your imagination about what such a website may look like in the future.

While the site is very pretty, it's not made by this community and as far as I can tell also not used by this community. Not very surprising since it's very barebones. Let's run through a list of features I am missing that I expect to see on a Crowdacting website.

Expected Features of a Crowdaction website

Broad range of projects

The site currently only has projects about crowda...
Learning Rhythms
by jefftk — 2020-05-16T11:00:02.361Z

In February I recorded myself playing a rhythm on mandolin,

measured the subbeat timings, and
programmed them explicitly into my

rhythm stage
setup. In addition to being slow and annoying, this approach
doesn't feel fluid. I can imitate one specific mandolin pattern, but
I can't smoothly vary what I'm doing over time, or perform a new
pattern live.

I decided to try and make my system learn the rhythm from my piano
playing. My system divides the beat into 72 subbeats, so I started by
making counters for each subbeat, incremented whenever I play a note
starting there, and aging slowly over time:

This is essentially a probability distribution over subbeats: how
often have I started a note on each?

I initially thought that this would work well mapped to the volume of
a bass, and it sort of does. The curve is nice, but it ends up
sounding ahead of the beat, like it's rushing. I think what's going
wrong is that our brains place the note by when it first becomes
noticeable, and not by the timing of the peak.

Instead we want to look at the notes and find the peak. I thought
about a bunch of ways of processing the distribution to find peaks,
but ended up with something much simpler since I primarily care about
finding the upbeat: use the average timing of the last eight notes
that fell between 3/8 and 3/4 of the way through the beat. This will
cover everything from a robotic upbeat falling exactly at 1/2,
something pushing it a little at ~15/36, something swung at ~5/8,
down to a relaxed jig at ~2/3. Since this doesn't need many notes to
make a decision, it can follow you smoothly if you start adding a bit
of swing or firm things up some.

YouTube Demo

The next thing I want to look at is whether this will let me let the
system maintain its idea of the current tempo. Right now I need to
tap every downbeat on a pedal, but instead I could do that for a
little bit to clarify the tempo and then it could follow the piano
from there. This would free up my feet a lot, and be easier on my
knees.
Multi-agent safety
by Richard_Ngo — 2020-05-16T01:59:05.250Z

Note: this post is most explicitly about safety in multi-agent training regimes. However, many of the arguments I make are also more broadly applicable - for example, when training a single agent in a complex environment, challenges arising from the environment could play an analogous role to challenges arising from other agents. In particular, I expect that the diagram in the 'Developing General Intelligence' section will be applicable to most possible ways of training an AGI.

To build an AGI using machine learning, it will be necessary to provide a sequence of training datasets or environments which facilitate the development of general cognitive skills; let’s call this a curriculum. Curriculum design is prioritised much less in machine learning than research into novel algorithms or architectures; however, it seems possible that coming up with a curriculum sufficient to train an AGI will be a very difficult task.[1] A natural response is to try to automate curriculum design. Self-play is one method of doing so which has worked very well for zero-sum games such as Go, since it produces tasks which are always at an appropriate level of difficulty. The generalisation of this idea to more agents and more environments leads to the concept of multi-agent autocurricula, as discussed by Leibo et al. (2019).[2] In this framework, agents develop increasingly sophisticated capabilities in response to changes in other agents around them, in order to compete or cooperate more effectively. I'm particularly interested in autocurricula which occur in large simulated environments rich enough to support complex interactions; the example of human evolution gives us very good reason to take this setup seriously as a possible route to AGI.

One important prediction I would make about AGIs trained via multi-agent autocurricula is that their most interesting and intelligent behaviour won’t be directly incentivised by their reward functions. This is because many of the selection pressures exerted upon them will come from emergent interaction dynamics.[3] For example, consider a group of agents trained in a virtual environment and rewarded for some achievement in that environment, such as gathering (virtual) food, which puts them into competition with each other. In order to gather more food, they might learn to generate theories of (simulated) physics, ...
Conjecture Workshop
by johnswentworth — 2020-05-15T22:41:31.984Z

The conjecture workshop is an activity I’ve run three times now in a conference/meetup-style setting. Feedback has been unusually positive, so I’m writing it up here to codify the basic idea and hopefully get independent feedback from others who try it.

The overall goal is to translate intuitive ideas into mathematical conjectures.

Format

People break into groups of 2-3. Within each group, one person serves as “conjecturer”, and the other one or two serve as “questioners”. Roles can rotate over the course of the allotted time (~1 hr for sessions so far).

To start, the conjecturer should have some intuitive claim in mind that they want to formalize. Some AI-oriented examples from previous sessions:

In beneficial comprehensive AI services, security services and planning services necessarily have to be "agenty"
There exists some “universal” algorithm which performs about-as-well as any other on optimization problems with bounded runtime
Coarse-grained models of the world form a category, and we can always construct pullback/pushout models within that category

Note that these are quite fuzzy and leave out a lot of the idea; fully explaining the ideas intuitively (even without formalization) takes much longer than the short blurbs above. Indeed, the ideas usually start out even fuzzier than the blurbs above - just summarizing them in a sentence is hard.

Once the conjecturer has something in mind, they try to explain the claim to the questioners, intuitively. The questioners’ job is to regularly interrupt, and ask questions to help formalize what the conjecturer is saying. Common examples include:

“You’ve mentioned <thing> a couple of times. Should we give it a name?”
“What type of thing is <thing we just named>? Any constraints on it?”
“Ok, so we claim/assume that <thing> is <better/larger/simpler/etc> than <other thing> in some intuitive sense. How do we quantify that?”
“So we want to assume <intuitive assumption>. What does that mean, mathematically?”
“Does the model so far accurately capture your intuitions, at least the parts which intuitively seem relevant to the claim?”

The questioners should also ask general explanation-support questions, like “can you give an example?”, &#x...
How to choose a PhD with AI Safety in mind
by kwiat.dev — 2020-05-15T22:19:13.800Z

I'm about to begin doctoral studies in multiagent RL as applied to crowd simulation, but somewhere on the horizon, I see myself working on AI Safety-related topics. (I find the Value Alignment problem to be of particular interest)

Now, I'm asking myself the question - if my PhD is in a roughly related area of AI, but not really closely compatible with AI Safety, does that make anything more difficult further down the line? Or is it still perfectly fine?
Movable Housing for Scalable Cities
by Eliezer Yudkowsky — 2020-05-15T21:21:05.395Z

First posted on Steemit on July 30, 2016.

0: Summary.

The non-scalability of current major cities poses a challenge to economic growth; it is a burden that falls disproportionately on those least able to bear it.

Unfortunately, any would-be startup city must confront the network effects that pose a tremendous potential energy barrier to the attractiveness of a city that isn't yet mature. From day one, you need to compete with the hedonic attractiveness of San Francisco, in the eyes of the person who decides where to locate the company.

Movable houses mated to modular foundations – nice large customized houses, not small ugly trailers – can make cities more scalable and more hedonically attractive:

By making it easier for groups to relocate and untangle themselves in a coordinated way, movable houses can increase the natural dispersion of the city as it grows larger.
Since people could buy and retain customized houses manufactured with economies of scale, movable houses could have better technology and amenities not available today. (Contrast your current house to a modern car.)
Movable houses could allow unprecedented opportunities to live next to your friends, or to groups of similar-minded people. You and your friends just need to find a set of available modular foundations close together.
Movable housing might help on key points of governance and realpolitik, for example by making land value taxes more attractive, and decreasing exit costs.

(This was written in response to Y Combinator's request for ideas about the design of new cities.)

1: Introduction: The challenge of scalable cities.

Manhattan doesn't scale well. San Francisco doesn't scale well. Tokyo and London don't scale well. On the present system, each added resident comes with some tiny added degree in inconvenience for every other resident already living in a city, most notably in the form of traffic jams and housing prices. Yes, every added programmer also provides some tax revenue; but the city infrastructure, qua infrastructure, is not scaling well.

This is not all due to anti-housing housing policy, though we've certainly gone to every possible length to shoot ourselves in that foot. Limited road bandwidth is a real issue. The air in Manhattan is genuinely dirtier than in Berkeley – I acquire a sore throat every time I visit New York City.

Building cities that can scale better is a problem of overwhelm

...
The EMH Aten't Dead
by Richard Meadows — 2020-05-15T20:44:49.402Z

Cross-posting from my personal blog, but written primarily for Less Wrong after recent discussion here.

There are whispers that the Efficient-Market Hypothesis is dead. Eliezer's faith has been shaken. Scott says EMH may have been the real victim of the coronavirus.

The EMH states that “asset prices reflect all available information”. The direct implication is that if you don’t have any non-available information, you shouldn’t expect to be able to beat the market, except by chance.

But some people were able to preempt the corona crash, without any special knowledge! Jacob mentioned selling some of his stocks before the market reacted. Wei Dai bought out-of-the-money 'put' options, and took a very handsome profit. Others shorted the market.

These people were reading the same news and reports as everyone else. They profited on the basis of public information that should have been priced in.

And so, the EMH is dead, or dying, or at the very least, has a very nasty-sounding cough.

I think that rumours of the death of efficient markets have been greatly exaggerated. It seems to me the EMH is very much alive-and-kicking, and the recent discussion often involves common misunderstandings that it might be helpful to iron out.

This necessarily involves pissing on people's parade, which is not much fun. So it's important to say upfront that although I don't know Wei Dai, he is no doubt a brilliant guy, that Jacob is my favourite blogger in the diaspora, that I would give my left testicle to have Scott's writing talent and ridiculous work ethic, that Eliezer is a legend whose work I have personally benefited from greatly, etc.

But in the spirit of the whole rationality thing, I want to gently challenge what looks more like a case of 'back-slaps for the boys' than a death knell for efficient markets.

First: how the heck did the market get the coronavirus so wrong?

The Great Coronavirus Trade

Lots of people initially underreacted to COVID-19. We are only human. But the stockmarket is not only human—it’s meant to be better than this.

Here’s Scott, in A Failure, But Not of Prediction:

The stock market is a giant coordinated attempt to predict the economy, and it reached an all-time high on February 12, suggesting that analysts expected the economy to do great

...