trentmkelly/LessWrong-43k huggingface dataset:
Platonic rewards, reward features, and rewards as information
by Stuart_Armstrong — 2019-11-12T19:38:10.685Z

Contrast these two expressions (hideously mashing C++ and pseudo-code):

argmaxxr(x),

argmaxx∗(&r)(x).

The first expression just selects the action x that maximises r(x) for some function r(), intended to be seen as a reward function.

The second expression borrows from the syntax of C++; (&r) means the memory address of r, while ∗(&r) means the object at the memory address of r. How is that different from r itself? Well, it's meant to emphasise the ease of the agent wireheading in that scenario: all it has to do is overwrite whatever is written at memory location (&r). Then ∗(&r) can become - whatever the agent wants it to be.

Let's dig a bit deeper into the contrast between reward functions that can be easily wireheaded and those that can't.

The setup

The agent A interacts with the environment in a series of timesteps, ending at time t=N.

There is a 'reward box' R which takes observations/inputs oRt and outputs some numerical reward amount, given by the voltage, say. The reward function is a function of oRt; at timestep t, that function is Rt(). The reward box will thus give out a reward of Rt(oRt). Initially, the reward function that R() implements is r()=R0().

The agent also gets a separate set of observations oAt; these observations may include full information about oRt, but need not.

Extending the training distribution

Assume that the agent has had a training phase, for negative values of t. And, during that training phase, Rt() was always equal to r().

If A is trained as a reinforcement agent, then there are two separate value functions that it A can learn to maximise:

E∑Nt=0r(oRt), or

E∑Nt=0Rt(oRt).

Since Rt()=r() for t<0, which is all the t that the agent has ever seen, both fit the data. The agent has not encountered a situation where it can change the physical behaviour of R() to anything other than r() - how will it deal with that?

Wireheading is in the eye of the beholder

Now, it's tempting to call r(oRt) the true reward, and Rt(oRt) the wireheaded reward, as the agent redefines the reward function in the reward box.

But that's a judgement call on our part. Suppose that we wanted the agent to maintain a high voltage coming out of R, to power a device. Then Rt(oRt) is the 'true' reward, while r() is just information about what the current design of R is.

This is a key insight, and a reason that avoiding wireheading is so hard. ...
Attach Receipts to Credit Card Transactions
by jefftk — 2019-11-12T16:30:01.678Z

If you log into your credit card account you'll see a list of charges,
each with a date, amount, and merchant. It would be helpful if this
also included receipt data:

If you didn't recognize a charge, seeing what it was for could
remind you.

If you needed a receipt for taxes or reimbursement one could be
captured automatically.

Personal finance tools (or corporate equivalents for company
cards) could track spending with higher granularity.

Because the credit card company knows what the items are they
can better detect fraud.

Receipt data isn't currently part of the protocol used for charges;
you'd need to spec out something that let companies communicate
everything a receipt can communicate today. This would be a very
large change, but everyone who would need to make changes can have
incentives in the right direction:

The card company likes it because it can market their card as
supporting receipts and better detect fraud.

The merchant likes it because they see fewer chargebacks and
the credit card company probably gives them slightly better
rates.

Point-of-sale makers like it because they get to sell a lot of
upgrades.

The main downside I can think of, aside from it being a lot of work,
is that people might not want their credit card company knowing the
particular products they bought. The company would probably want to
sell this to marketers, though there would be plenty of time to pass
regulations limiting that if we wanted to. Personally, I don't mind:
the merchant is probably already selling my purchase information or
will soon. And the money the credit card company gets from selling
the data, since it's a competitive market, probably mostly gets passed
on as higher cardholder incentives or higher incentives to merchants
to adopt receipt sharing.

Comment via: facebook
An optimal stopping paradox
by Yuxi_Liu — 2019-11-12T08:51:03.199Z

Consider an optimal stopping problem: a company at each time step grows by some constant, and has a certain probability of shutting down. You decide when to sell the company.

Since the math is cleaner in continuous time, we consider the continuous time. Then the company has a linearly increasing value βt, and an exponentially decaying survival curve e^(-αt).

Another framing of the paradox: Schrodinger wants to make a new record for the longest surviving cat, so he put a cat in the box with an atom that might decay and kill the cat, and waits. When should he open the box?

Since at each moment in time, you face the exact same problem (linearly increasing reward, α-exponentially decaying survival rate), if you decide to wait at t=0, you would decide to wait forever, and thus receive no reward.

There are several possible replies to this paradox, none of which is satisfactory to me:

"This looks like St. Petersburg Paradox.". No, because at time t=0, the expectation is β/α^2. In fact, the payoff can grow faster than βt, such as like t^3, and it would still have finite expectation.

Claim that expectation maximization decision theory is flawed. This doesn't stop the procrastination. As long as your decision is purely based on the future, and your rational decision process is constant in time, you either immediately sell the company or never sell the company.

Try some kind of discounting, like exponential discounting. This doesn't stop the procrastination., since at any time, selling the company gives you 0 extra expected reward, and waiting gives you some positive extra expected reward, no matter how much you discount the future.

Claim that there should be a finite lifetime. You can't wait forever. If there is a finite lifetime, then the same decision analysis would tell you to procrastinate until the very end. This effectively is procrastinating forever. It does not converge to a reasonable finite waiting time as your lifetime goes to infinity.

Claim that one should stick to past decisions even when they don't make sense from a purely future-looking decision theory. Such decision theory seems to be
just sweeping time-inconsistency under the rug, and I'm sure would suffer from serious paradoxes of their own.

Claim that there is no paradox, and procrastination is really the rational action. I'd not claim a strategy that guarantees 0 reward to be rational.

Option 5 seem

...
Can indifference methods redeem person-affecting views?
by David Scott Krueger (formerly: capybaralet) — 2019-11-12T04:23:10.011Z

My uninformed paraphrase/summary of "the person-affecting view" is: "classical utilitarianism + indifference to creating/destroying people".

These views seem problematic (e.g. see Hillary Greeves interview on 80k), and difficult to support.

Indifference methods (e.g. see Stuart Armstrong's paper) seem like they might be a way to formalize the person-affecting view in a rigorous way.

If we have a policy, we can always reverse engineer a corresponding reward function (see our Reward Modelling agenda, bottom page 6).

So while there might still be highly counter-intuitive bullets that need to be bitten, this might provide a way of cashing out person-affecting views in a way that is mathematically coherent/consistent.

What do you think? Does it work?

And is that even an open problem, or an interesting result to people in ethics?
Operationalizing Newcomb's Problem
by ErickBall — 2019-11-11T22:52:52.835Z

The standard formulation of Newcomb's problem has always bothered me, because it seemed like a weird hypothetical designed to make people give the wrong answer. When I first saw it, my immediate response was that I would two-box, because really, I just don't believe in this "perfect predictor" Omega. And while it may be true that Newcomblike problems are the norm, most real situations are not so clear cut. It can be quite hard to demonstrate why causal decision theory is inadequate, let alone build up an intuition about it. In fact, the closest I've seen to a real-world example that made intuitive sense is Narrative Breadcrumbs vs Grizzly Bear, which still requires a fair amount of suspension of disbelief.

So, here I'd like to propose a thought experiment that would (more or less*) also work as an actual experiment.

A psychologist contacts you and asks you to sign up for an experiment in exchange for a payment. You agree to participate and sign all the forms. The psychologist tells you: "I am going to administer a polygraph (lie detector) test in which I ask whether you are going to sit in our waiting room for ten minutes after we finish the experiment. I won't tell you whether you passed, but I will give you some money in a sealed envelope, which you may open once you leave the building. If you say yes, and you pass the test, it will be $200. If you say no, or you fail the test, it will be $10. Then we are done, and you may either sit in the waiting room or leave. Please feel no obligation to stay, as the results are equally useful to us either way. The polygraph test is not perfect, but has so far been 90% accurate in predicting whether people stay or leave; 90% of the people who stay for ten minutes get $200, and 90% of those who leave immediately get $10."

You say you'll stay. You get your envelope. Do you leave the building right away, or sit in the waiting room first?

Does the answer change if you are allowed to open the envelope before deciding?

*I don't know if polygraphs are accurate enough to make this test work in the real world or not.
[Health] [Math] Proofs, forgetting, and an eldritch god
by aaq — 2019-11-11T18:31:05.289Z

This post made me realize something very important, and in hindsight very obvious about myself. I'm leaving it up for future reference. I don't think it's actually very well written.

Originally, it was tagged [Math], not [Health] and [Math].

I envy computers sometimes for their memory.

My basic intuition from learning signals and Fourier transforms in my EE major is that any limited-space memory system that has to adapt to new surroundings, also has to have some factor by which old memories decay over time -- otherwise the system would become saturated with information content. (I am not being precise with these terms. Look where I point.)

But computers don't usually have as ruthless a "use-it-or-lose-it" forgetting mechanism as human brains do built into them. Their decay is mostly due to (i) other conscious agents going in and rearranging their memories; (ii) the slow encroaching wave of entropy on their hard disks. (i) can be dealt with by locking the computer in a room away from human hands, and praying the Poincare recurrence theorem combined with thermodynamic noise doesn't imply it will eventually birth an AGI all on its own. But that's a symptom of (ii), which can be mostly dealt with via RAID-5 and an influx of hard drives. With those two in place, on a human timescale, computers essentially never have to forget something once it's encoded in memory.

Human beings, we aren't like that. We forget. We forget so easily. And we warp the memories we do have to fit our twisted little narratives in the moment. I'm not a terribly existential person, but the fact that we ship of Theseus ourselves according to the ridiculous whims of future us, people just like us now but ever-so-slightly slower, crazier, and more tied down to the local flora for their survival, you have to admit -- it's a little unsettling.

I've been having a bit of a crisis over the last few days. It's about math, as all my crises are.

The problem is simple: I prove $X$ once. I walk away for a few days. When I go back to look at $X$ again, I suddenly realize the steps of the proof don't immediately spring to mind any more. I have to prove it again, don't I; or accept it on faith that $X$ is a fact, that former me proved $X$ to be a fact, that no, there's really nothing much to be gained from re-proving something in roughly the same way that I did before.

The cognition is contrarian: Proofs take effo...
The randomness/ignorance model solves many anthropic problems
by Rafael Harth — 2019-11-11T17:02:33.496Z

(Follow-up to Randomness vs Ignorance and Reference Classes for Randomness)

I've argued that all uncertainty can be divided into randomness and ignorance and that this model is free of contradictions. Its purpose is to resolve anthropic puzzles such as the Sleeping Beauty problem.

If the model is applied to these problems, they appear to be underspecified. Details required to categorize the relevant uncertainty are missing, and this underspecification might explain why there is still no consensus on the correct answers. However, if the missing pieces are added in such a way that all uncertainty can be categorized as randomness, the model does give an answer. Doing this doesn't just solve a variant of the problem, it also highlights the parts that make these problems distinct from each other.

I'll go through two examples to demonstrate this. The underlying principles are simple, and the model can be applied to every anthropic problem I know of.

1. Sleeping Beauty

In the original problem, a coin is thrown at the beginning to decide between the one-interview and the two-interview version of the experiment. In our variation, we will instead repeat the experiment 2n times and have n of those run the one-interview version, and another n run the two-interview version. Sleeping Beauty knows this but isn't being told which version she's currently participating in. This leads to 2n instances of Sleeping Beauty waking up on Monday, and n instances of her waking up on Tuesday. All instances fall into the same reference class, because there is no information available to tell them apart. Thus, Sleeping Beauty's uncertainty about the current day is random with probability 23 for Monday.

2. Presumptuous Philosopher

In the original problem, the debate is about the question of how the size of the universe influences the probability that the universe is large, but it is unspecified whether our current universe is the only universe.

Let's fill in the blanks. Suppose there is one universe at the base of reality which runs many simulations, one of them being ours. The simulated universes can't run simulations themselves, so there are only two layers. Exactly half of their simulations are of "small" universes (say with 1015 people), and the other half are of "large" universes (say with 1021 people). All universes look identical from the inside.

Once again, there is only one reference cla...
Ban the London Mulligan
by Zvi — 2019-11-11T11:10:00.443Z

Previously: On The London Mulligan

Oko, Thief of Crowns is a highly messed up Magic card and needs to be banned in Standard. On that we can all agree. Throne of Eldraine contains many other messed up Magic cards. Some of them, like Once Upon a Time, Wicked Wolf and Gilded Goose, are not getting the appreciation they deserve because Oko, Thief of Crowns is stealing the spotlight. If both of those cards are Standard legal when they rotate out, I will be quite surprised. Then there are Fires of Invention, Caldron Familiar / Witch’s Oven, Embercleave, Emry, Lurker of the Lock. Then there’s Feasting Troll King, Questing Beast, Bonecrusher Giant, Lovestruck Beast, Edgewall Inkeeper and the list goes on. And yes, it has a lot of green in it. There are also messed up Magic cards one can choose from previous sets, although the density of them is far lower.

You are not going to succeed in Standard, for a long time, no matter what is banned, without building around at least one messed up Magic card from Throne of Eldraine. If design does not make large adjustments, and likely even if they do, every good Standard deck for a long time is going to have a key messed up Magic card.

Even more than a single messed up Magic card, these decks have central play patterns. Gilded Goose into Oko, Thief of Crowns. Third turn Nissa, Who Shakes the World. Fires of Invention into Cavalier of Flame. Witch’s Oven sacrificing Caldron Familiar sacrificing food triggering Trail of Crumbs or Mayhem Devil. Embercleave on my Questing Beast or Rotting Regisaur or knight. First turn Pelt Collector or else your deck doesn’t quite work. Explosive or super efficient adventure or Veteran Loxodon starts. Growth Spiral into Wilderness Reclamation. And so on.

If you get to do these things, a seventh card you had to put on the bottom will not be overly missed. If you don’t get these things, an extra card will not much matter.

Thus, the first player is forced to mulligan hands that look perfectly good, but which cannot pull off their key play pattern. Decks get designed with a key consideration being what you can and cannot keep as an opening hand. The deck I would have run at Mythic Championship V, Elk Blade, keeps no configuration of cards that lacks Arboreal Grazer, Gilded Goose or Once Upon a Time against an unknown opponent, even at six cards. Because once you see that such hands are going to be bad no m...
Pieces of time
by KatjaGrace — 2019-11-11T07:00:03.264Z

My friend used to have two ‘days’ each day, with a nap between—in the afternoon, he would get up and plan his day with optimism, whatever happened a few hours before washed away. Another friend recently suggested to me thinking of the whole of your life as one long day, with death on the agenda very late this evening. I used to worry, when I was very young, that if I didn’t sleep, I would get stuck in yesterday forever, while everyone else moved on to the new day. Right now, indeed some people have moved on to Monday, but I’m still winding down Sunday because I had a bad headache and couldn’t sleep. Which is all to say, a ‘day’ does not just mean a 24 hour measure of time, in our minds. Among its further significance, we treat it as a modular unit: we expect things within it to be more continuous and intermingled with each other than they are with things outside of it. What happens later today is more of a going concern at present than something that happens after sleeping. The events of this morning are more part of a continuous chapter, expected to flavor the present, than what happened yesterday. The same is true to some extent for weeks, months and years (but not for fortnights or periods of 105 hours).

I think days are well treated as modular like this because sleeping really separates them in relevant ways. I notice two other kinds of natural modular time-chunks that seem worth thinking in terms of, but which I don’t have good names for:

Periods during which you are in one context and stream of thought (usually a minute to a few hours long). For instance the period of going for a walk, or the period between getting home and receiving a phone call that throws you into a new context and set of thoughts. During one such chunk, I can remember a lot about the series of thoughts so far, and build upon them. Whereas if I try to go back to them later, they are hard to bring back to life, especially the whole set of thoughts and feelings that I wandered around during a period, rather than just a single insight brought from it. Within chunks like this, my experience seems more continuous and intermingled with other experience within the chunk. Then I get an engaging message or decide to go out, and a new miniature chapter begins, with new feelings and thoughts. (Though I’m not sure how much other people’s thoughts depend on their surroundings, so maybe for other

...
Epistemic Spot Check: Unconditional Parenting
by Elizabeth — 2019-11-10T20:10:00.932Z

Epistemic spot checks started as a process in which I investigate a few of a book’s claims to see if it is trustworthy before continuing to read it. This had a number of problems, such as emphasizing a trust/don’t trust binary over model building, and emphasizing provability over importance. I’m in the middle of revamping ESCs to become something better. This post is both a ~ESC of a particular book and a reflection on the process of doing ESCs and what I have and should improve(d).

As is my new custom, I took my notes in Roam, a workflowy/wiki hybrid. Roam is so magic that my raw notes are better formatted there than I could ever hope to make them in a linear document like this, so I’m just going to share my conclusions here, and if you’re interested in the process, follow the links to Roam. Notes are formatted as follows:

The target source gets its own page

On this page I list some details about the book and claims it makes. If the claim is citing another source, I may include a link to the source.

If I investigate a claim or have an opinion so strong it doesn’t seem worth verifying (“Parenting is hard”), I’ll mark it with a credence slider. The meaning of each credence will eventually be explained here, although I’m still working out the system.

Then I’ll hand-type a number for the credence in a bullet point, because sliders are changeable even by people who otherwise have only read privileges.

You can see my notes on the source for a claim by clicking on the source in the claim

You may see a number to the side of a claim. That means it’s been cited by another page. It is likely a synthesis page, where I have drawn a conclusion from a variety of sources.

This post’s topic is Unconditional Parenting (Alfie Kohn) (affiliate link), which has the thesis that even positive reinforcement is treating your kid like a dog and hinders their emotional and moral development.

Unconditional Parenting failed its spot check pretty hard. Of three citations I actually researched (as opposed to agreed with without investigation, such as “Parenting is hard”), two barely mentioned the thing they were cited for as an evidence-free aside, and one reported exactly what UP claimed but was too small and subdivided to prove anything.

Nonetheless, I thought UP might have good ideas kept reading it. One of the things Epistemic Spot Checks were designed to detect was “science washi...