trentmkelly/LessWrong-43k huggingface dataset:
How safe "safe" AI development?
by Gordon Seidoh Worley — 2018-02-28T23:21:50.307Z

There are now at least two AI development companies purporting to research safe AI and both have existed for a couple years now, so I think it's worth taking another look at how safe "safe" AI development is.

One of those companies is GoodAI. I don't know a lot about them beyond what's on their website. They seem clearly to be aware of AI safety concerns and the need for alignment but are also pursuing capabilities research. OpenAI is the other and is similarly pursuing capabilities research but, at least based on what I know about them publicly, only go so far as to say they want safe AI, although OpenAI does employ at least one person known to be actively working on AI safety. There may be other companies in the safe AI development space but to the best of my knowledge other efforts do not make explicit statements about safety and are only focused on capabilities (although AI for self-driving cars, for example, has certain mundane safety concerns different from those of AI safety research).

When OpenAI was started there was some discussion about whether it was a good idea. Ben Hoffman said no, Nate Soares had a positive reaction, and others had mixed responses. Some quick searching hasn't turned up any explicit opinions on GoodAI, so presumably people would feel similarly about them as they do about OpenAI. A quick rehash on some of the arguments:

No AI is safe AI until we are sure we can align it.
Capabilities research is in conflict with safety research so no capabilities effort can be trusted to be fully committed to safety.
If you are going to research AI anyway, better to try to be safe than not at all.
More people working on safety is always better, and the associated capabilities research these companies are doing would have happened anyway.

We're a couple years on now, though, and given that AI seems to be on a strong upward swing, does it make sense to encourage more companies to be like OpenAI and GoodAI and target safe AI, perhaps via a self-regulatory organization, or is encouraging safety as an explicit goal in capabilities research unlikely to have much effect?

I'm inclined to suspect that some attention to safety is better than none because it gives a wedge with which to push for more safety later, so I'm especially curious about arguments that it wouldn't help.
Beyond algorithmic equivalence: self-modelling
by Stuart_Armstrong — 2018-02-28T16:55:55.161Z

In the previous post, I discussed ways that the internal structure of an algorithm might, given the right normative assumption, allow us to distinguish bias from reward.

Here I'll be pushing the modelling a bit further.

Self-modelling

Consider the same toy anchoring biased problem as the previous post, with the human algorithm H, some object X, a random integer 0≤n≤99, and an anchoring bias given by

H(X,n)=34V(X)+14n,

for V some valuation function that is independent of n.

On these inputs, the internal structure of H is:

However, H is capable of self-modelling, to allow it to make long term decisions. At time t, H models itself at time t+1 as:

Note that H is in error here: it doesn't take into account the influence of n on its own behaviour.

In this situation, it could be justifiable to say that H's self model is the correct model of its own values. And, in that case, the anchoring bias can safely be dismissed as a bias.

Self-model and preparation

Let's make the previous setup a bit more complicated, and consider that, sometimes, the agent H is aware of the effect of n, and sometimes they aren't.

At time t, they also have an extra action choice: either n , which will block its future self from seeing n, or ∅, which will proceed as normal. Suppose further that whenever H is aware of the effect of n, they take action n:

And when H isn't aware of the effect of n, they don't take any action/takes ∅:

Then it seems very justifiable to see H as opposing the anchoring effect in themselves, and thus classifying it as a bias rather than a value/preference/reward.

The philosophical position

The examples in this post seem stronger than in the previous one, in terms of justifying "the anchoring bias is actually a bias".

More importantly, there is a philosophical justification, not just an ad hoc one. We are assuming that H has a self model of their own values - they have a model of what is a value and what is a bias in their own behaviour.

Then we can define the reward of H, as the reward that H models itself as having.

In subsequent posts, I'll explore whether this definition is justified, how to access these self-models, and what can be done about errors and contradictions in self-models.
Beyond algorithmic equivalence: algorithmic noise
by Stuart_Armstrong — 2018-02-28T16:55:36.036Z

There is a 'no-free-lunch' theorem in value learning; without assuming anything about an agent's rationality, you can't deduce anything about its reward, and vice versa.

Here I'll investigate whether you can deduce more if you start looking into the structure of the algorithm.

Algorithm (in)equivalence

To do this, we'll be violating the principle of algorithmic equivalence: that two algorithms with the same input-output maps should be considered the same algorithm. Here we'll instead be looking inside the algorithm, imagining that we have either the code, a box diagram, an FMRI scan of a brain, or something analogous.

To illustrate the idea, I'll consider a very simple model of the anchoring bias. An agent H (the "Human") is given an object X (in the original experiment, this could be wine, book, chocolates, keyboard, or trackball), an random integer 0≤n≤99, and is asked to output how much they would pay for it.

They will output H(n,X)=34V(X)+14n, for some valuation subroutine V that is independent of n. This gives a quarter weight to the anchor n.

Assume that H tracks three facts about X: the person's need for X, the emotional valence the person feels at seeing it, and a comparison with objects with similar features. Call these three subroutines Need, Emo, and Sim. For simplicity, we'll assume each subroutine outputs a single number, that then gets averaged.

Now consider four models of H as follows, with arrows showing the input-output flows:

I'd argue that a) and b) imply that the anchoring bias is a bias, c) is neutral, and d) implies (at least weakly) that the anchoring bias is not a bias.

How so? In a) and b), n maps straight into Sim and Need. Since n is random, it has no bearing on how much X is needed, and on how valuable similar objects are. Therefore, it makes sense to see its contribution as noise or error.

In d), on the other hand, it is superficially plausible that a recently heard random input could have some emotional effect (if n was not a number but a scream, we'd expect it to have an emotional impact). So if we wanted to argue that, actually, the anchoring bias is not a bias but that people actually derive pleasure from outputting numbers that are close to numbers they heard recently, then n going into Emo would be the right place for it to go. Setup c) is not informative either wa... ,,,,,,,,,,,,,,,,,,,,,,
Using the universal prior for logical uncertainty (retracted)
by cousin_it — 2018-02-28T13:07:23.644Z

(I'm not sure about any of this)

(Yup! Vanessa Kowalski pointed out a crucial error, see her comment below. The idea about the universal prior picking up all computable regularities is unproved. I'm leaving the post as it was, but please read it as a historical document only.)

Abram Demski had an old paper describing a naive way to use the universal prior for logical uncertainty. He pointed out some problems with it, but I'm not sure the problems are real. I think the approach will work surprisingly well.

The idea is that, since the universal prior is just a probability distribution over bit sequences, we don't have to use it to predict bits one by one. We can also update it on facts like "the seventh bit is 1" without mentioning the previous six bits. To choose which bits to update on, we can mechanically search through all possible proofs in Peano arithmetic (PA) by increasing length. Whenever we prove a sentence with Gödel number N, we tell the prior that the Nth bit is 1; whenever we disprove one, we tell the prior that the bit is 0. After each update, the prior's opinions about all other bits (and thus all other sentences in PA) will shift, hopefully in a meaningful way.

It's easy to see that the probability of any provable sentence will converge to 1, because at some point it becomes 1 by fiat, and disprovable sentences will likewise go to 0. How fast is the convergence? I think it will be extremely fast, far outpacing our slow proof search and subsuming any possible human or computer reasoning about PA - even though the prior knows nothing about PA and only receives opaque bits.

To see why, recall that the universal prior can pick up any computable regularities. For example, if you feed it a sequence where all even-numbered bits are 1 and odd-numbered bits are hard to predict, it will quickly become very certain that all future even-numbered bits will be 1. Similarly, if there's some decidable set of Gödel numbers that for some reason correspond to provable sentences (no matter how long their proofs are), the universal prior will become very certain of that after receiving enough data. The same goes for regularities in relationships between bits, like "for all A and B, if A is provable and A→B is provable, then B is provable as well". All such shortcuts will be learned in full generality as quickly as possible.

This approach to logical u...
2/27/08 Update – Frontpage 3.0
by Raemon — 2018-02-28T06:26:23.255Z

We finally got around to a long-postponed update to the frontpage, where logged in users see content that is hopefully more relevant to them.

Some notes:

Non logged in users should have a mostly unchanged experience
Logged in users now see a list of recommended sequences. The list will change over time. We aren't sure about the details, but we want to highlight old content that might be worth re-reading, and new sequences that seem high caliber. For now we expect to change the sequences about once a week.
We added back a formal Curated Post section, mostly identical to the old one. Since longtime users often spend most time in the Community filter, Curated posts didn't stand out as strong as they used to. We had gotten some feedback that this made getting into Curated feel less special than it used to, and wanted to fix that.
Logged in users now always see all options for the "Curated/Frontpage/Community/Meta/Daily" menu
TDT for Humans
by alkjash — 2018-02-28T05:40:00.450Z

This is part 19 of 30 of Hammertime. Click here for the intro.

As is Hammertime tradition, I’m making a slight change of plans right around the scheduled time for Planning. My excuse this time:

Several commenters pointed out serious gaps in my knowledge of Focusing. I will postpone Internal Double Crux, an advanced form of Focusing, to the next cycle. Instead, we will have two more posts on making and executing long-term plans.

Day 19: TDT for Humans

Previously on planning: Day 8, Day 9, Day 10.

Today I’d like to describe two orders of approximation to a working decision theory for humans.

TDT 101

Background reading: How I Lost 100 Pounds Using TDT.

Choose as though controlling the logical output of the abstract computation you implement, including the output of all other instantiations and simulations of that computation.

~ Eliezer

In other words, every time you make a decision, pre-commit to making the same decision in all conceptually similar situations in the future.

The striking value of TDT is: make each decision as if you would immediately reap the long-term rewards of making that same decision repeatedly. And if it turns out you’re an updateless agent, this actually works! You actually lose 100 pounds by making one decision.

I encourage readers who have not tried to live by TDT to stop here and try it out for a week.

TDT 201

There are a number of serious differences between timeless agents and human beings, so applying TDT as stated above requires an unacceptable (to me) level of self-deception. My second order of approximation is to offer a practical and weak version of TDT based on the Solitaire Principle and Magic Brain Juice.

Three objections to applying TDT in real life:

Spirits

A human is about halfway between “one monolithic codebase” and “a loose confederation of spirits running a random serial dictatorship.” Roughly speaking, each spirit is the piece of you built to satisfy one primordial need: hunger, friendship, curiosity, justice. At any given time, only one or two of these spirits is present and making decisions. As such, even if each individual spirit as updateless and deterministic, you don’t get to make decisions for all the spirits currently inactive. You don’t have as much control over the other spirits as you would like.

Different spirits have access to different data and beliefs. I’ve mentioned, for example, that I have different personalities spea...
Set Up for Success: Insights from 'Naïve Set Theory'
by TurnTrout — 2018-02-28T02:01:43.790Z

Foreword

This book has been reviewed pretty thoroughly already. Rather than restate each chapter, I'll be sharing insights: some book-specific, some general.

I am quite proud of my time-to-completion for this book - just over a week, working around a very strenuous courseload. I went from having to focus really hard to pick up new concepts to reading notation nearly as fluently as the English surrounding it. The chapters started making sense - it felt less random, and more like a coherent story wherein the protagonist slowly adds Powers to their arsenal.

Naïve Set Theory

Functions

Functions f:X→Y are just static sets of ordered pairs {(x,f(x)):x∈X} . They are not dynamic indexing functions, they do not perform efficient lookup, please do not waste an hour of your life trying to figure out how you could do such a thing within the simple machinery afforded by set theory up to that point.

This is one of those things that Nate talked about - how skipping over just one word a few chapters prior can cause you to waste hours. During my confusion, I knew this was probably the case, but I still couldn't manage to immediately overcome my intuitions of what a function should be. This is one reason I'm open to working through the MIRI Research Guide with others.

Families

Families are, ironically enough, just a special kind of function; don't let your intuition fool you - they aren't "groups of functions". A family belonging to ∏i∈IXi maps each element i of the index set I to an element xi∈Xi. For example, a family from {1,2} to X1:={cat,dog},X2:={tent,fire} could be {(1,cat),(2,fire)} (thanks to Dacyn for helping me clarify my writing).

Zorn's Lemma

I spent three hours staring at this proof. I understood what ZL meant. I grasped the relevant concepts. I read other versions of the proof. I still spent three long hours on this damn proof, and then I went to my classes. I don't know why I ended up figuring it out, but I suspect it was a combination of two factors: my brain worked through some things during the day, and I really wanted it. On the bus home, I mentally snapped and decided I was going to understand the proof. And I did.

I'm pleased to share my detailed proof outline of Zorn's Lemma, the product of many hours of ambient exasperation, rewritten in my own notation. Looking back, the proof in the book was pretty bad; it ... ,,,,,,,,,
Intuition should be applied at the lowest possible level
by Rafael Harth — 2018-02-27T22:58:42.000Z

Earlier today I lost a match at Prismata, a turn-based strategy game without RNG. When I analyzed the game, I discovered that changing one particular decision I had made on one turn from A to B caused me to win comfortably. A and B had seemed very close to me at the time, and even after knowing for a fact that B was far superior, it wasn't intuitive why.

Then I listed the main results from A and B, valued those by intuition, and immediately B looked way better.

One can model these problems on a bunch of different levels, where going from level n to n+1 means hiding the details of level n and approximating their results in a cruder way. On level 1, one would compare the two subtrees whose roots are decisions A and B (this should work just like in chess). Level 2 would be looking at exact resource and attack numbers in subsequent turns. Level 3 would be categorizing the main differences of A and B and giving them intuitive values, and level 4 deciding between A and B directly. What my mistake showcases is that, even in a context where I am quite skilled and which has limited complexity, applying intuition at level 4 instead of 3 lead to a catastrophic error.

If you can't go lower, fine. But there are countless cases of people using intuition on a level that's unnecessarily high. Hence if it's worth doing, it's worth doing with made-up numbers. That is just one example of where applying intuition one level further down: "what quantity of damage arises from this" rather than "how bad is it" can make a big difference. On questions of medium importance, briefly asking yourself "is there any point where I apply intuition on a level that's higher than necessary" seems like a worthy exercise.

Meta: I write this in the spirit of valuing obvious advice, and the suspicion that this error is still made fairly often.
The sad state of Rationality Zürich - Effective Altruism Zürich included
by roland — 2018-02-27T14:51:05.881Z

Epistemic status: Personal experience.

This is another post in the spirit of:

The Craft is Not The Community
What Is Rationalist Berkley’s Community Culture?
The Craft & The Community - A Post-Mortem & Resurrection

Edit: I know that not everyone in the communities is aware of what is going on or involved with it, this was another reason for me to post this publicly.

I used to be a member of Rationality and EA Zürich before I was excluded by leading figures. Actually I was the organizer of the first Zürich meetup.

I will name people because they are all public figures in the sense that they are leaders and organizers of said events and I think it is time to name people and no longer allow them to hide behind anonymity. As mentioned in the 3rd posting above a major cause of problems in the community is the fear of being confrontational, etc...

The people involved in this are(names obfuscated with ROT-13):

Marko Thiel, organizer of Rationality Zürich

Michal Pokorný organizer of Rationlity Zürich and EA Zürich, CFAR alumnus

Daniel Wünn, organizer of EA Zürich.

There is also a closed facebook group named "Rationality Zürich" of which I was a member.

In Dec 2017 I posted a question in this group asking members for experiences with Circling because I was interested and I heard members talking about it so I thought that maybe someone would share their experiences. To my surprise I didn't get much feedback. Only later I managed to connect the dots with the following events.

A few days later I got a message from Michal Pokorný and Daniel Wünn saying that I was no longer welcome in the Rationality/EA Zürich meetups. The provided reason was that people felt uncomfortable about me and one example would be me talking about dating and sex. We had a group discussion about the latter topic in one EA meeting. Several people where engaged in that discussion some very vehemently, so it wasn't as if I was forcing the topic, but anyways I'm getting ahead of myself.

I was surprised and taken aback by this, especially since this happened over fb messages and not in a personal conversation. Nevertheless I said if talking about sex and dating was the problem I wouldn't mind not talking about that. I got the answer "This is not enough and really we don't have any concrete poin...
The worst trolley problem in the world 
by CronoDAS — 2018-02-27T03:56:26.159Z

We all know the standard trolley problem by now. A runaway trolley with five people trapped on it is on a track that's going to send it off a cliff, killing everyone on board. There's a lever that can send it onto a side track, though, that will result in the trolley stopping safely, but there is a single person tied to the track who will be run over by the trolley and die.

Let's make it even more horrible. The person tied to the side track is the only one who can move the lever. If the person chooses to die to save the others, most moral systems would generally consider this a Good and Noble Thing. But sacrificing oneself is difficult; most people want very much not to die, and except in unusual circumstances it goes against biological survival instincts. If someone doesn't sacrifice themselves, is this a morally permissible act, given human frailty? We do demand that soldiers, even conscripted ones, be willing to perform acts of self-sacrificial bravery, and call those who refuse "cowards" and "deserters". Does the initial position of the lever matter? A person who does not throw a lever to divert the trolley onto himself might be a coward, but a person who throws a lever to divert the trolley away from himself and over a cliff might be a murderer.
Categories of Sacredness
by Zvi — 2018-02-27T02:00:00.403Z

Previously: Eternal, and Hearthstone Economy versus Magic Economy, Out to Get You

On Lesser Wrong, JenniferRM gave a reply that is worth quoting in full:

If I understand correctly, the cognitive process/bias/heuristic/whatever of “sacredness” is relevant here.

Neither nails nor dollars are sacred so you’re free to trade dollars for nails.

A kidney is sacred, so you can’t trade that for dollars, but you can trade it for another kidney (although such trades still feel a bit weird).

Sacred things are often poorly managed in practice, and sacredness is easy to make fun of, but a decent defense of sacredness might be that it is one of the few widely installed psychological mechanisms in real life for managing the downsides of having markets in things. Thus, properly deployed sacredness might let you have “trade” in one area without ending up with “totalalizing trade”?

In the smaller and hopefully lower stakes world of video games, I think the suggestion would be to have card classes with different trading characteristics.

The lowest class of very non-sacred things could be swapped with extremely low transaction costs within the class and also be tradeable directly for money.

Higher sacredness things would have a separate market, perhaps with transaction costs like needing a purchaseable delivery mechanism or imposing delays so that objects go into limbo after the trade is finalized while “being delivered”. The most sacred things would be “inalienable” so they can’t be traded or given away or perhaps not even be destroyed.

Exactly where sacredness should be deployed in order to maximize fun seems like a deep and relatively unstudied problem.

One place in real life where the inalienability of something has large and substantive differences from jurisdiction to jurisdiction is the question of the rights of artistic creators to their artwork. In some jurisdictions, an artist cannot legally sell their right to veto the use of their artwork if deployed in artistically compromising ways (like use in advertising or political campaigns) after mere copyrights have been sold.

In the US artistic moral rights are not treated as very sacred, and the lack of sacredness in art production is probably part of the US’s cultural dominance a la Hollywood, but it has arguably also had large effects in the lives of artists, visibly so with people like Bill Waterson and Prince.

Here, Benquo offers ...