trentmkelly/LessWrong-43k huggingface dataset:
The Dualist Predict-O-Matic ($100 prize)
by John_Maxwell — 2019-10-17T06:45:46.085Z

This is a response to Abram's The Parable of Predict-O-Matic, but you probably don't need to read Abram's post to understand mine. While writing this, I thought of a way in which I think things could wrong with dualist Predict-O-Matic, which I plan to post in about a week. I'm offering a $100 prize to the first commenter who's able to explain how things might go wrong in a sufficiently crisp way before I make my follow-up post.

Dualism

Currently, machine learning algorithms are essentially "Cartesian dualists" when it comes to themselves and their environment. (Not a philosophy major -- let me know if I'm using that term incorrectly. But what I mean to say is...) If I give a machine learning algorithm some data about itself as training data, there's no self-awareness there--it just chugs along looking for patterns like it would for any other problem. I think it's a reasonable guess that our algorithms will continue to have this "no self-awareness" property as they become more and more advanced. At the very least, this "business as usual" scenario seems worth analyzing in depth.

If dualism holds for Abram's prediction AI, the "Predict-O-Matic", its world model may happen to include this thing called the Predict-O-Matic which seems to make accurate predictions -- but it's not special in any way and isn't being modeled any differently than anything else in the world. Again, I think this is a pretty reasonable guess for the Predict-O-Matic's default behavior. I suspect other behavior would require special code which attempts to pinpoint the Predict-O-Matic in its own world model and give it special treatment (an "ego").

Let's suppose the Predict-O-Matic follows a "recursive uncertainty decomposition" strategy for making predictions about the world. It models the entire world in rather fuzzy resolution, mostly to know what's important for any given prediction. If some aspect of the world appears especially relevant to a prediction it's trying to make, it "zooms in" and tries to model that thing in higher resolution. And if some part of that thing seems especially relevant, it zooms in further on that part. Etc.

Now suppose the Predict-O-Matic is trying to make a prediction, and its "recursive uncertainty decomposition" algorithms say the next prediction made by this Predict-O-Matic thing which happens to occupy its world model appears especially relevan...
A Money Dojo
by Elo — 2019-10-16T20:35:11.718Z

Original post: A money Dojo

First up though, some ontology to make it easier to talk about this – “concrete, subtle, causal”.

The concrete world is the world of atoms. It’s physical and literal. (sometimes called the gross world)

The subtle is the world that sits on top of that. Things like emotion and goodness or the felt/body sense of things. If I’m having a “good day” technically that’s in the atom world, but practically when it’s described its not literal, nor is it always touchable from inside one mind to another (my good day is lightly transferable to you but I can’t force you to take my good day).

The causal realm, (yes they are more like “realms” than “worlds”) (sometimes called the very subtle). Is described as the place where other realms come from phenomenologically. Where thoughts come from. When I remember where I put the keys, that’s a potentially generative thought that seems to arise from nowhere. It’s causal in that it can cause me to go “aha!” (subtle emotional experience) and get the keys (concrete action), but when talking about the causal we are definitely talking about something different from the atom world.

Ideas and concepts exist in a causal sense and map down to the atom world. (although the way people use causal can sometimes vary and there’s endless debates on it). As subjective categories they are a bit variable. There’s also the edge where some experiences happen at the edge. A particular way of breathing might be more like an emotion than a concrete despite being both.

I was talking to a friend about money. And I noticed that trading money feels very odd. It always has, transactions feel strange. All my life transactions have been this odd little awkward dance of retrieving the cash, handing over the cash and receiving the change. There’s a little bit of social interaction in the process but it’s a bit more odd too.

My friend talked about a skill swap where the feeling of the trade felt more right and they enjoyed the experience much more because it felt like both sides were giving from the emotional part of themselves.

When I make a financial trade it seems to have a concrete realm swap of coinage but does not carry the subtle “made with love” e...
Epistemic Spot Checks: The Fall of Rome
by Elizabeth — 2019-10-16T18:40:00.958Z

Introduction

Epistemic spot checks are a series in which I select claims from the first few chapters of a book and investigate them for accuracy, to determine if a book is worth my time. This month’s subject is The Fall of Rome, by Bryan Ward-Perkins, which advocates for the view that Rome fell, and it was probably a military problem.

Like August’s The Fate of Rome, this spot check was done as part of a collaboration with Parallel Forecasting and Foretold, which means that instead of resolving a claim as true or false, I give a confidence distribution of what I think I would answer if I spent 10 hours on the question (in reality I spent 10-45 minutes per question). Sometimes the claim is a question with a numerical answer, sometimes it is just a statement and I state how likely I think the statement is to be true.

This spot check is subject to the same constraints as The Fate of Rome, including:

Some of my answers include research from the forecasters, not just my own.

Due to our procedure for choosing questions, I didn’t investigate all the claims I would have liked to.

Claims

Claim made by the text: “[Emperor Valerian] spent the final years of his life as a captive at the Persian Court”

Question I answered: what is the chance that is true?

My answer: I estimate a chance of (99 – 3*lognormal(0,1)) that Emperor Valerian was captured by the Persians and spent multiple years as a prisoner before dying in captivity.

You don’t even have to click on the Wikipedia page to confirm this is the common story: it’s in the google preview for “emperor valerian”. So the only question is the chance that all of history got this wrong. Wikipedia lists five primary sources, of which I verified three. https://www.ancient-origins.net/history/what-really-happened-valerian-was-roman-emperor-humiliated-and-skinned-hands-enemy-008598 raises questions about how badly Valerian was treated, but not that he was captive.

My only qualm is the chance that this could be a lie perpetuated at the time. Maybe Valerian died and the Persians used a double, maybe something weirder happened. System 2 says the chance of this is < 10% but gut says < 15%.

Claim made by the text: “What had totally disappeared, however, were the good-quality, low-value items, made in bulk, and available so widely in the Roman period”

Question I answered: What is the chance mass-produced, low-value items availabl...
Make more land
by jefftk — 2019-10-16T11:20:03.381Z

We used to make land. We built long wharves for docking ships, and
then over time filled in the areas between them. Later we built up
mudflats wholesale to make even larger areas. Here's a map of Boston
showing how much of the land wasn't previously dry:

(Map
reproduction courtesy of the Norman B. Leventhal Map & Education
Center at the Boston Public Library)

In expensive areas, converting wetlands and shallow water into usable
land is a very good thing on balance, and we should start doing it
again. To take a specific example, we should make land out of the San
Francisco Bay, at least South of the Dumbarton Bridge:

This is about 50mi2, a bit bigger than San Fransisco. This would be
enough new central land to bring
rents down dramatically across the region. It can be built to a
higher density than SF, because no one is having their neighborhood
Manhattanized. Millions of people could live there.

So, ok, let's address some likely objections:

This would be an environmental disaster. Some of that area is a
wildlife refuge, and all of it should be protected.

The world is very large, and cities are a very small portion of it.
The land we set aside for animals should be outside of cities, where
far more land is available at far less impact to people.

Sprawl has a much larger impact on wildlife than infill, and allowing
people to live closer in is the most powerful way to address sprawl.
Additionally, sprawl leads to much higher carbon emissions through
less efficient transportation. While development of the Bay would be
harmful to the specific animals that live there today, it would be
better for animals (and people) overall.

The Bay is beautiful and this would ruin it.

This part of the Bay is primarily industrial salt ponds.

This is just a few miles from a major fault line, and made land
can liquify in earthquakes.

You do need to take fill into account to build in an earthquake-safe
way, but modern engineering is well up to the task.

Traffic would be even worse.

The biggest contribution to traffic in the Bay Area is that people
aren't allowed to live where the jobs are. The second biggest is that
BART doesn't have enough coverage to make living without a car practical
in most of the area. This would help with both of these, since this
project would allow millions of people to live closer in and would...
Full toy model for preference learning
by Stuart_Armstrong — 2019-10-16T11:06:03.746Z

1. Toy model for greater insight

This post will present a simple agent with contradictory and underdefined "preferences", and will apply the procedure detailed in this research project to capture all these preferences (and meta-preferences) in a single utility function[1].

The toy model will start in a simpler form, and gradually add more details, aiming to capture almost all of sections 2 and 3 of the research agenda in a single place.

The purposes of this toy model are:

To illustrate the research agenda in ways that are easier to grasp.

To figure out where the research agenda works well and where it doesn't (for example, working through this example suggests to me that there may be a cleaner way of combining partial preference than the one detailed here).

To allow other people to criticism the toy model (and through it, the research agenda) in useful ways.

And, thus, ultimately to improve the whole research agenda.

So, the best critiques are ones that increase the clarity of the toy model, those that object to specific parts of it (especially if alternatives are suggested), and those that make it more realistic for applying to actual humans.

1.1 How much to read this post

On a first pass, I'd recommend reading up to the end of Section 3, which synthesises all the base level preferences. Then Sections 4, 5, and 6 deal with more advanced aspects, such as meta-preferences and identity preferences. Finally, sections 7 and 8 are more speculative and underdefined, and deal with situations where the definitions become more ambiguous.

2. The basic setting and the basic agent

Ideally, the toy example would start with some actual agent that exists in the machine learning literature; an agent which uses multiple overlapping and slightly contradictory models. However, I haven't been able to find a good example of such an agent, especially not one that is sufficiently transparent and interpretable for this toy example.

So I'll start by giving the setting and the agent myself; hopefully, this will get replaced by another ML agent in future iterations, but it serves as a good starting point.

2.1 The blegg and rube classifier

The agent is a robot that is tasked with sorting bleggs and rubes.

Time is divided into discrete timesteps, of which there will be a thousand in total. The robot is in a room, with a conveyor belt and four bins. Bleggs (blue eggs) and rubes (red cubes)...
Gradient hacking
by evhub — 2019-10-16T00:53:00.735Z

"Gradient hacking" is a term I've been using recently to describe the phenomenon wherein a deceptively aligned mesa-optimizer might be able to purposefully act in ways which cause gradient descent to update it in a particular way. In Risks from Learned Optimization, we included the following footnote to reflect this possibility:

Furthermore, a deceptively aligned mesa-optimizer would be incentivized to cause there to be a systematic bias in the direction of preventing the base optimizer from modifying its mesa-objective. Thus, in the context of a local optimization process, a deceptive mesa-optimizer might try to “hack” its own gradient (by, for example, making itself more brittle in the case where its objective gets changed) to ensure that the base optimizer adjusts it in such a way that leaves its mesa-objective untouched.

Outside of that footnote, however, there hasn't been much other discussion of this phenomenon, and I think that's a shame, as I think it's definitely something that's worth looking into.

For example, here's a gradient hacking story that I'm particularly concerned about. Suppose you're training a model and it becomes deceptive. You're not worried about that, though, because you have sufficient transparency tools that you're confident you can detect any deceptive cognition it might be doing. If it's thinking about how to deceive you, your transparency tools will pick that up, and you'll know to throw out that model and try again. However, your transparency tools are quite expensive, so you can only afford to run them once, which you're planning on doing after training before you deploy your model. If your model is deceptive, though, then it might know all of that—in particular, it might know that if it stays deceptive, it'll get caught.

However, there is still something it can do in that situation to be able to defect against you and cause whatever it wants to happen during deployment. That is, it could hack its own gradient to produce a new model which will still exhibit the same deceptive behavior, but without doing any of the detectable deceptive cognition. Perhaps the new model exhibits the deceptive behavior not because it's trying to act deceptive, but just because it's a hard-coded part of its algorithm, or because it has some bad proxy, or any other defect that might be more difficult to detect than outright deception.

How might suc...
Always Discard Fascist Policies
by David Spies — 2019-10-15T22:23:48.383Z

Burden of proof

I enjoy the game Secret Hitler. If you're unfamilliar with the game, the rules
can be found here: https://secrethitler.com/assets/Secret_Hitler_Rules.pdf
In this game, like in many social deception games, players love to make claims
about what you should always or never do when playing. In general, I reject
such claims since most pure strategies in hidden information games are
exploitable. The true mixed-strategy Nash equilibrium for Secret Hitler
inevitably will involve non-zero and non-100% probabilities for most actions.

Some examples of bad 100% policies that I've heard:

As president, if you pass two fascist policies to your chancellor, always
claim you drew three fascist policies rather than going head-to-head against
your chancellor.

Always assume presidents follow the above bullet point (and thus suspect only
the chancellor when two players go head-to-head).

As Hitler, always play policies identically to a liberal unless it would cause
the liberals to win the game immediately.

As a Fascist, if given the chance to view Hitler's party affiliation, always
claim Hitler is liberal.

So with this in mind, I recognize that if I make the claim "as a liberal, Always
Discard Fascist Policies", I'm placing a heavy burden of proof on myself.
Nonetheless, I intend to prove that this is indeed always the optimal
strategy.

Assumption

However I need to make one assumption. I've never played with a group who
rejected this assumption. But I'd be curious to hear if readers feel
differently. The assumption is that liberal players always immediately and
truthfully reveal all information they have available to them.

The Setup:

Imagine Alice and Bob are playing Secret Hitler. The group has approved
Alice as president and Bob as chancellor. Alice is a liberal and does not
concretely know Bob's party affiliation. Alice draws three tiles. Two are
liberal policies and one is fascist. Alice has the option to either:

Discard the fascist tile and pass off the two liberal tiles to Bob, denying
him any choice in what he does.

OR

Discard one of the liberal tiles and allow Bob to select whether to play a
liberal or fascist tile.

Proof

Since liberals always honestly reveal all information available to them, we know
that Bob knows everything Alice claims to know. Therefore Bob's information is a
superset of Alice's (or any liberal's) information.

Now we'll sample a ra...
Philadelphia SSC Meetup
by Majuscule — 2019-10-15T17:41:40.028Z

Philadelphia SSC typically meets the last Wednesday of every month at 6:30pm, locations TBA. To avoid upcoming holidays, we've moved up some of our dates; below are our scheduled meetups through the end of 2019:

Wednesday, Oct. 30, 6:30pm - Tir Na Nog Irish Pub, 1600 Arch St., Philadelphia

Wednesday, Nov. 20, 6:30pm - TBA

Friday, Dec. 13 - Year-end celebration in Ardmore, PA

If you're interested, please join our Google Group or send an e-mail to dmcbriggs(at)gmail.com or wfenza(at)gmail.com

https://groups.google.com/forum/#!forum/ssc-philadelphia
Transportation Tracking
by jefftk — 2019-10-15T11:00:01.987Z

Several times lately I've seen people say it's not practical to
continue with walking and public transit in the US once you have kids
unless you live in Manhattan. I've also had people think that us not
having a car means that we never use cars. Both of these aren't the
case, and I thought it might be helpful to talk some about how we do
transportation.

First, here are some numbers for the most recent thirty days. They're
not completely representative, but it's as typical as any thirty-day
period is likely to get. The data comes from going through my location history,
collected automatically by my
phone, supplemented by memory when I noticed it had missed something:

Number of (one-way) trips: 101

Trips by purpose:

Commute: 40 (40%; subway to and from work)

Errand: 25 (25%; groceries, school drop-off)

Social: 23 (23%; visiting my dad)

Leisure: 13 (13%; going for a walk or to the park)

Trips by mode:

Transit: 49 (49%; bus and subway)

Foot: 39 (39%; walking, with some scooter and trike)

Car: 13 (10%; rides, renting, taxis)

Days with any:

Foot: 24 (80%)

Transit: 21 (70%)

Car: 9 (30%)

(sheet)

Our family is two adults and two children, 3y and 5y. We live in Somerville,
one of the most dense cities in the country (#16,
though less so if you look at population-weighted density), and
about a ten minute walk from a subway station. Julia works from home,
I work downtown. We don't have a car, but we do have housemates and
two of them have cars.

On a typical day it's just walking and public transit. I'll walk (run) to the subway, ride to work, and reverse in
the evening. When the kids go out, generally to the park, it will be
stroller, wagon, or on foot. Most places we
want to go are pretty practical traveling this way. If it's cold or
rainy we'll make sure the kids are dressed appropriately, and the
stroller has a rain cover.

For grocery shopping we do a mix of things. For perishables I'll
generally stop at the supermarket on my way home from work. For
non-perishables I'll do a big
shop at Market Basket or Wegmans about once a quarter when I
happen to have access to a car. (We have a large chest freezer, and
plenty of basement space.) We also occasionally order groceries delivered when
that seems like a better use of time. And occasionally we'll walk
over with t...
TAISU 2019 Field Report
by Gordon Seidoh Worley — 2019-10-15T01:09:07.884Z

Last summer I delivered a "field report" after attending the Human Level AI multi-conference. In mid-August of this year I attended the Learning-by-doing AI Safety Workshop (LBDAISW? I'll just call it "the workshop" hereafter) and the Technical AI Safety Unconference (TAISU) at the EA Hotel in Blackpool. So in a similar spirit to last year I offer you a field report of some highlights and what I took away from the experience.

I'll break it down into 3 parts: the workshop, TAISU, and the EA Hotel.

The workshop

The learning by doing workshop was organized by Linda Linsefors and led by Linda and Davide Zagami. The zeroth day (so labeled because it was optional) consisted of talks by Linda and Davide explaining machine learning concepts. Although this day was optional I found it very informative because machine learning "snuck up" on me by becoming relevant after I earned my Masters in Computer Science so there have remained a number of gaps in my knowledge about how modern ML works. Having a day full of covering basics with lots of time for questions and answers was very beneficial to me, as I think it was for many of the other participants. Most of us had lumpy ML knowledge, so it was worthwhile to get us all on the same footing so we could at least talk coherently in the common language of machine learning. As I said, though, it was optional, and I think it could have easily been skipped for someone happy with their level of familiarity with ML.

The next three days were all about solving AI safety. The approach Linda took was to avoid loading people up with existing ideas, which was relevant because some of the participants had not previously thought much about AI safety, and instead asked us to try to solve AI safety afresh. The first day we did an exercise of imagining different scenarios and how we would address AI safety under those scenarios. Linda called this "sketching" solutions to AI safety, with the goal being to develop one or more sketches of how AI safety might be solved by going directly at the problem. For example, you might start out working through your basic assumptions about how AI would be dangerous, and then see where that pointed to a need for solutions, then you'd do it again but choosing different assumptions and see where it lead you. Once we had done that for a couple hours we presented our ideas about how to address AI safety. The ideas...