trentmkelly/LessWrong-43k huggingface dataset:
Intuitive examples of reward function learning?
by Stuart_Armstrong — 2018-03-06T16:54:18.466Z

Can you help find the most intuitive example of reward function learning?

In reward function learning, there is a set of possible non-negative reward functions, R, and a learning process ρ which takes in a history of actions and observations and returns a probability distribution over R.

If π is a policy, Hm is the set of histories of length m, and Pπ(hm) is the probability of hm∈Hm given that the agent follows policy π, the expected value of π at horizon m is:

Vπ=∑R∈R,hm∈HmPπ(hm)R(hm)ρ(R|hm),

where R(hm) is the total R-reward over the history hm. Problems can occur if ρ is riggable (this used to be called "biasable", but that term was over-overloaded), or influenceable.

There's an interesting subset of value learning problems, which could be termed "constrained optimisation with variable constraints" or "variable constraints optimisation". In that case, there is an overall reward R, and every R′∈R is the reward R subject to constraints C. This can be modelled as having C(hm) being 1 (if the constraints are met) and 0 (if they are not).

Then if we define RC(hm)=R(hm)C(hm), and let ρ be a distribution over C, the set of constraints, the equation changes to:

Vπ=∑C∈C,hm∈HmPπ(hm)RC(hm)ρ(C|hm).

If ρ is riggable or influenceable, similar sorts of problems occur.

Intuitive examples

Here I'll present some examples of reward function learning or variable constraints optimisation, and I'm asking for readers to give their opinions as to which one seems the most intuitive to you, and the easiest to explain to outsiders. You're also welcome to suggest new examples if you think they work better.

Classical value learning: human declarations determine the correctness of a given reward R. The reward encodes what food the human prefers, and some foods are much easier to get than other.
As above, but the reward encodes whether a domestic robot should clean the house or cook a meal.
As above, but the reward encodes the totality of human values in all environments.
Variable constraint optimisation: the agent is writing an unoriginal academic paper (or a patent), and must maximise the chance it gets accepted. The paper must include a literature review (constraints), but the agent gets to choose the automated process that produces the literature review.
Variable constraint optimisation: p-hacking. The agent chooses which hypothesis to formula

... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Murphy’s Quest Ch 5: Fail Gracefully
by alkjash — 2018-03-06T05:10:00.635Z

“I need healing!”

“Murph, over here! I’m wounded!”

“Heal me!”

“I need healing!”

I aim my Initiate Chimes at Plun and cast Heal. The spell sends a circular golden orb crawling through the air. It misses my party by a wide margin and vanishes into the dungeon wall.

“I need healing!”

Chimes must be the silliest weapons ever invented: a combination Menora/Wind Chime, with candles on top and musical instrument on bottom. I dangle the Holy weapon away from my body, trying not to light my loose-fitting Initiate Robes on fire.

“I need healing!”

“Murph, top me up!”

A second golden orb leaves my Chimes. It grazes Plun’s buttock before connecting with an enemy Bandit, saving her from the brink of death.

“Whose side are you on, dipshit?!”

At first, the enemy tried to focus-fire me. They soon realized I can self-Heal, and furthermore that I can’t hit anyone else with this slow-moving skillshot.

That’s why I, LVL 3 Cleric Murphy, am standing in the center of the fight bodyblocking for our Mage Vant.

“I need healing!”

By divine providence I finally land a Heal on our Rogue Tess as she weaves around. Tess and I have … philosophical disagreements about the proper way to play Rogue. I’d rather she STAY IN ONE PLACE and shoot arrows. Instead she insists on running in dagger style.

“I need healing!”

“I JUST HEALED YOU!”

—

I met a few minor setbacks since last week.

Class Choice is an archaic name. Once upon a time, nobody picked Cleric. The Church complained. Now “The Goddess” chooses for you. Class Choice is permanent.

Clerics are members of the Church who specialize in Healing and Purity magic. Main stat FTH, secondary INT. Stats are determined by LVL and equipment, but also can be trained individually.

The basic Heal spell is a skill-shot single-target heal. The size and power of the orb scale off FTH, and the speed of the orb scales off INT.

XP sharing is out of whack in this world. XP for a kill is divided equally among all players who dealt damage to the enemy and all players who healed them. With my weak and impossible-to-aim Heal spell, this becomes an issue.

One might ask, “Why is your Heal so weak, Murphy? Have you been slacking off?”

To make a long story short:

—

INT is trained by reading the Training Books in the Library. The moment I got a free hour I went over to get some INT buffs. Here’s the first line of Training Book 1:

Yberz vcfhz qbybe fvg nzrg, pbafrpgrghe nqvcvfpvat ryvg...
Book Review: Consciousness Explained
by Charlie Steiner — 2018-03-06T03:32:58.835Z

The trouble with brains, it seems, is that when you look in them, you discover that there’s nobody home.

I.

This is a book I've long been aware of, but never got that itch to read. Maybe I trusted the field of philosophy too little, assuming that a book called "Consciousness Explained" was probably not very good. Maybe I trusted the field of philosophy too much, assuming that if someone had actually explained consciousness while I was a toddler, I would have been informed somehow before now. Either way, I was wrong, and the book is great.

I'm going to try to give what is either a short tour, or a long compilation of quotes. I'm leaving out several whole chapters, nearly every thought experiment, most of the examples and science, and some very nice language. And yet, this is still long enough that I encourage you, even if you do like reading Dan Dennett on consciousness, if you don't like long things, maybe don't read this all in one sitting - stop at V or VI and pretend that's the end of part one.

II.

Dennett quickly warns the reader that he's aware that the contents may sound counterintuitive.

We shouldn’t expect a good theory of consciousness to make for comfortable reading — the sort that immediately “rings bells,” that makes us exclaim to ourselves, with something like secret pride: “Of course! I knew that all along! It’s obvious, once it’s been pointed out!” The mysteries of the mind have been around for so long, and we have made so little progress on them, that the likelihood is high that some things we all tend to agree to be obvious are just not so.

This is not the mysterian claim that his ideas about consciousness are likely because they are counterintuitive, but it does signal a core claim of the book: the intuitive view of the problem of consciousness is broken from the foundation up. Naturally, if the intuitive theory is wrong, the right theory is counterintuitive.

Where, exactly, is our intuition going wrong? The most important example is introduced by considering how, on macroscopic scales, it is convenient to treat observers as point-like entities:

We explain the startling time gap between the sound and sight of distant fireworks by noting the different transmission speeds of sound and light. They arrive at the observer (at that point) at different times, even though they left the source at the same time.

Wh

...
set of cards 
by November H — 2018-03-05T22:37:49.240Z

I think it'd be helpful to me (maybe others) to make a set of cards to aid in problem solving, but I'd like to get feedback on what people think and suggestions (as well as find similar projects if they're around). What I'm thinking is a few different types - one would be a set of heuristics for problem solving when you've gotten stuck.

The idea would be to pull a card, try the approach it suggests if acceptable, and move it to the 'tried' pile if the problem is still unsolved. There might even be some process of promotion, like adding heuristics that worked for you to the top of the deck, and putting the 'dead' pile on the bottom of the deck. Another might be a set of fallacies.

I was thinking one for emotions, although I'm less sure about this one, to check in with your feelings - things to be mindful of regarding each of them, like I notice anger seems to be kind of a defensive emotion that comes up when I feel ripped off or infringed upon. With anger I feel like it'd be helpful to ask, "what was infringed?", etc. So with those it might be helpful to pick all the ones you feel about a situation. I guess the analogy I'm kind of going for here is a sort of 'tarot' deck that rather than predicting the future, helps you deal with uncertainty constructively and somewhat methodically.

Obviously not a substitute for thinking but a supplement to it. I was thinking for the starting heuristics to go with the ones from polya's "how to solve it" and maybe some others. I'd like it to include some that a person might forget in focusing on solving the problem but might be important to consider, like, "is it acceptable to cheat? Is there a way to cheat?", "Do you have to solve this problem? Can you give up?" etc.
Ancient Social Patterns: Comitatus
by ryan_b — 2018-03-05T18:28:35.765Z

Purpose: I've been thinking about problems of goal advancement in spite of death, which brings us to honor. I am also keen on the history of Central Asia and the Steppe, and this is really cool: hence the blog section.

The word comitatus is Latin for retinue, or armed escort. In Germania, Tacitus uses it to describe the relationship between a lord and his sworn warrior. Through textual and archaeological evidence, this relationship has been shown to be widespread, and extended back much further to Proto-Indo-European times. It has been observed as far west as Spain, and as far east as Korea and Japan. I have most of this information from the book

Empires of the Silk Road: A History of Central Eurasia from the Bronze Age to the Present

, by Christopher Beckwith.

The comitatus is a lord and a warband of his friends, who swear to defend him to the death, forsaking all other obligations. In life they are bodyguards and constant companions, asking for and receiving gifts of wealth and weapons. If the lord dies first then his comitatus would be interred with him after ritual suicide or execution, and they were buried armed for battle in order that they could fight in the afterlife. Such was honor in those days.

The detail about this arrangement that seems most important to me is that the oath superseded obligations to their kin-group or people. Near as I can tell it is the first social institution to do this: while we have evidence of urbanization and religious organization which is earlier, they are not separated from kin-group concerns. It proved extremely resilient, lasting as late as early modern times in the Scandinavian and Islamic contexts (though I note that the pattern had been adapted - for example ritual execution and suicide fell into disuse with the rise of current world religions).

Seems to have arisen concurrent with chariots; apart from methods of making tools, may be the earliest military innovation.
I would expect success in competing against clan-oriented power bases, which was probably especially important for intra-clan competition.
Reliant on a shared culture of honor.
Bound up with commercial interests: being able to attract brave companions required wealth to give them, which meant it required profitable commerce, taxes, or plunder.

This pattern of oath-bound behavior has a variety of local terms, but because it comes from Central Asia and the ...
Takeoff Speed: Simple Asymptotics in a Toy Model.
by Aaron Roth — 2018-03-05T17:07:12.866Z
Link: http://aaronsadventures.blogspot.com/2018/03/how-unlikely-is-intelligence-explosion.html

I've been having fun recently reading about "AI Risk". There is lots of eloquent writing out there about this topic: I especially recommend Scott Alexander's Superintelligence FAQ for those looking for a fun read. The subject has reached the public consciousness, with high profile people like Stephen Hawking and Elon Musk speaking publicly about it. There is also an increasing amount of funding and research effort being devoted to understanding AI risk. See for example the Future of Humanity Institute at Oxford, the Future of Life Institute at MIT, and the Machine Intelligence Research Institute in Berkeley, among others. These groups seem to be doing lots of interesting research, which I am mostly ignorant of. In this post I just want to talk about a simple exercise in asymptotics.

First, Some Background.

A "superintelligent" AI is loosely defined to be an entity that is much better than we are at essentially any cognitive/learning/planning task. Perhaps, by analogy, a superintelligent AI is to human beings as human beings are to Bengal tigers, in terms of general intelligence. It shouldn't be hard to convince yourself that if we were in the company of a superintelligence, then we would be very right to be worried: after all, it is intelligence that allows human beings to totally dominate the world and drive Bengal tigers to near extinction, despite the fact that tigers physiologically dominate humans in most other respects. This is the case even if the superintelligence doesn't have the destruction of humanity as a goal per-se (after all, we don't have it out for tigers), and even if the superintelligence is just an unconscious but super-powerful optimization algorithm. I won't rehash the arguments here (Scott does it better) but it essentially boils down to the fact that it is quite hard to anticipate what the results of optimizing an objective function will be, if the optimization is done over a sufficiently rich space of strategies. And if we get it wrong, and the optimization has some severely unpleasant side-effects? It is tempting to suggest that at that point, we just unplug the computer and start over. The problem is that if we unplug the intelligence, it won't do as well at optimizing its objective function compared to if it took steps to prevent us from unplugging it. So if it's strategy ... ,,,,,,,,,,,,,,,,,,,,,,,,,,,
Murphy’s Quest Ch 4: Noticing Confusion
by alkjash — 2018-03-05T07:20:01.112Z

Two muscle heads stand in the doorway, clubs out. Let’s call them Crabbe and Goyle. Were those really their names? Might as well have been.

“Hand ‘em over,” Crabbe smacks his Wood Club against his other hand like a police baton.

Now it came to me: these two had been missing from Training almost every day since the Sleep Refresh was revealed. My spidey sense went off just as they’d snuck off to steal my hard-earned ears.

Cornered, my adrenaline kicks in. Mind starts racing.

What would Harry do?

They messed with the wrong interdimensional traveler. Imma go Robert Downey Junior on these fuckers.

All the possibilities lay themselves clearly before my eyes. Knock out their knees. One running jump. A feint to the side. I’ll point behind them wide-eyed to distract –

—

“Murphy! Murphy!”

“Wh— What happened?”

“Crabbe and Goyle knocked you out and stole your ears!”

I sit bolt upright, but already know it’s too late. Plun helps me up and we shuffle shamefaced into the Drill Sergeant’s office.

—

“ – and that’s the last thing I remember, Sir.”

Sarge studies me in contempt.

He’ll find the bastards. They can’t get away with this.

The silence draws out like a string of mozzarella cheese from a hot pizza.

“Let that be a lesson, boy. Out there, in the dungeons, you won’t get a second – ”

I run outside. I’m in no mood to take shit from this midget. The other boys are gone now, probably already left for Class Choice. Only Murphy left. Without me, they’d still be slaving away at 9 ears a day.

A Kobold spawns nearby. I bash its head savagely.

“I will enjoy watching you die.”

I don’t remember what that’s from, but the words flow smoothly out of my mouth. I barely notice the Kobold Scratch as I kill the pathetic little beast.

Crabbe, Goyle, I’m coming for you!

—

“Murph! Hey!”

A voice pulls me out of my rage.

“Plun? What are you doing here?”

“You thought I’d abandon you, Murph? What kind of friend do you take me for?”

The word disorients me. Here, in this fantasy world, I plan to acquire plenty of Core Party Members, but …

“Friend?”

“You asshole, Murph. You think just because I’m an army-raised orphan I don’t know how to be a friend? Listen, take half my ears. It’s only gonna take four more days if we farm these fuckers together.”

There’s times when you have no words to say, so you just hold the other person as tight as you can and cry. This was one of those times. In ani...
God Help Us, Let’s Try To Understand Friston On Free Energy
by Scott Alexander — 2018-03-05T06:00:01.132Z

I’ve been trying to delve deeper into predictive processing theories of the brain, and I keep coming across Karl Friston’s work on “free energy”.

At first I felt bad for not understanding this. Then I realized I wasn’t alone. There’s an entire not-understanding-Karl-Friston internet fandom, complete with its own parody Twitter account and Markov blanket memes.

From the journal Neuropsychoanalysis (which based on its name I predict is a center of expertise in not understanding things):

At Columbia’s psychiatry department, I recently led a journal club for 15 PET and fMRI researhers, PhDs and MDs all, with well over $10 million in NIH grants between us, and we tried to understand Friston’s 2010 Nature Reviews Neuroscience paper – for an hour and a half. There was a lot of mathematical knowledge in the room: three statisticians, two physicists, a physical chemist, a nuclear physicist, and a large group of neuroimagers – but apparently we didn’t have what it took. I met with a Princeton physicist, a Stanford neurophysiologist, a Cold Springs Harbor neurobiologist to discuss the paper. Again blanks, one and all.

Normally this is the point at which I give up and say “screw it”. But almost all the most interesting neuroscience of the past decade involves this guy in one way or another. He’s the most-cited living neuroscientist, invented large parts of modern brain imaging, and received of the prestigious Golden Brain Award for excellence in neuroscience, which is somehow a real thing. His Am I Autistic – An Intellectual Autobiography short essay, written in a weirdly lucid style and describing hijinks like deriving the Schrodinger equation for fun in school, is as consistent with genius as anything I’ve ever read.

As for free energy, it’s been dubbed “a unified brain theory” (Friston 2010), a key through which “nearly every aspect of [brain] anatomy and physiology starts to make sense” (Friston 2009), “[the source of] the ability of biological systems to resist a natural tendency to disorder” (Friston 2012), an explanation of how life “inevitably and emergently” arose from the primordial soup (Friston 2013), and “a real life version of Isaac Asimov’s psychohistory” (description here of Allen 2018).

I continue to hope some science journalist takes up the mantle of explaining this comprehensively. Until that happens, I’ve been working to gather as many perspectives as I c...
Explanation of Paul's AI-Alignment agenda by Ajeya Cotra
by habryka — 2018-03-05T03:10:02.666Z
Link: https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616

Ajeya from OpenPhil wrote a very understandable and quite compelling summary of Paul's views on AI-Alignment.

(@Paul & @Ajeya: I would love to crosspost the whole thing here if you are open to that)
Murphy’s Quest Ch 3: Murphyjitsu
by alkjash — 2018-03-05T02:40:01.260Z

Having learned the secret to Sleep Refresh, the rest of Training progresses smoothly. Throughout the day, we stream in and out of the barracks to get a minute of shut-eye between Kobold spawns.

12 ears an hour, 10 hours a day, it only takes 8 more mind-numbing days to finish this fetch quest. To pass the time, Plun and I share stories and songs on the grassy knoll.

Plun’s family was slaughtered by orcs in the war front and he was raised in a local orphanage. He gets teary-eyed telling the story.

Emotional stuff makes me nervous. I think about how Mom comforts me, and reach a hand out to pat his shoulder.

I wish I could say we shared a poignant moment of silent understanding between two human beings and that was somehow enough. I wish I could say that being present for a friend in his moment of need inextricably linked together the strands of our fates.

The truth is, my hand brushes awkwardly against his (rather bulky) bicep, and he shoots me a weird look.

“No Homo!” I shout instinctively.

“What?”

“Uh, no comment, no comment. Continue your story please!”

…

“When the slave trade ended several centuries ago, the Belstine peerage decided to draft their orphans into the war effort.” He gives me a long history lesson but I’m too busy cringing inside.

After Plun’s lecture, he teaches me a melancholy song about war veterans fighting to their dying breath for Beltine. It’s altogether mediocre even to my tone-deaf ears. I absentmindedly pick a modern song to teach him in exchange. What came next, I should have predicted.

Even in this world, Taylor Swift is an instant hit.

—

I’ve been dreaming every night of the old world.

“Mommy, why is my name Murphy?”

“Well, dear, I wanted to name you something normal like John or Allen, but I changed my mind after meeting you. You were just too special!”

She never did tell me what happened. Dad says she first went to the hospital to give birth to me on March 11, 1997.

My birthday is April 4.

It was her decision to name me Murphy, after Murphy’s Law:

Everything that can go wrong will go wrong.

Mom used to say, “Murphy, sometimes things just go wrong and it’s nobody’s fault.”

Her words ring in my ears at this very moment as I try to block out the chorus of Back to December echoing around camp for the fourth day in a row. Plun set it to a local military theme. Even the LVL 10 guards are singing it.

Will T Swift be my lasting contribution to t...