trentmkelly/LessWrong-43k huggingface dataset:
Opportunities for individual donors in AI safety
by Alex Flint — 2018-03-31T18:37:21.875Z

Tl;dr

Over the past few years the availability of funding for AI safety projects has increased significantly. Yet opportunities for individual donors to find high impact grant opportunities remain. In this post I review the recent history of AI safety organizations, speculate on the ways in which funding early AI safety organizations may have impacted today’s AI safety landscape, and on this basis propose some heuristics for finding untapped funding opportunities:

Focus on which people you’re bringing into the community

Personal brands and platforms matter, and have accounted for some of the largest successes of the community

Culture and intellectual environments matter

Small amounts of funding today may move much larger amounts of funding in the future

(Cross-posted from EA forum upon request.)

Updated 3/31: added info on Singularity Summit and Max Tegmark.

Introduction

For early AI safety organizations, funding was a major bottleneck. During the 2000-2010 period, large donors were sparse and, with few exceptions, not motivated to commit large amounts of capital to such esoteric causes[1]. Small donors accounted for a substantial fraction of the budget of organizations such as SIAI[2], while others located themselves within universities. A private donor looking for funding opportunities at this time would quickly find multiple organizations with significant room for funding, and much of the work in making grants consisted in comparing different funding opportunities for counterfactual impact.

Over the past few years that situation has changed. Multiple large grant-making organizations have identified AI safety as an important cause area,[3] including the Open Philanthropy Project[4], the Future of Life Institute[5], the Berkeley Existential Risk Institute[6], and the EA Long Term Future Fund[7]. While the amounts that each organization or individual hopes to deploy within AI safety is not public (or in many cases not yet decided), it seems likely that the amount of funding that would be deployed if there were many large, promising funding opportunities exceeds the room for more funding across currently existing AI safety organizations.

This changes the nature of funding decisions, since in a world with more funding than projects, the decision to fund a project looks more like an evaluation of whether the project crosses some usefulness threshold, and less li...
Time in Machine Metaethics
by Razmęk Massaräinen — 2018-03-31T15:02:55.295Z

Main points

The notion of constant progress of time applies both to AI and humans. However, in decision theories and AI research, it makes things more complicated and seem to be frequently disregarded for simplicity.
An evolving AI may face the need to predict the decision-making of its future version, in absence of information about future.
Time also may has something to do with human utility and qualia, along which AGI needs to be aligned - and we are in a urgent need of better understanding.
Futures in which AGI is designed to maximize human utility without better understanding of it seem to have a lot of people getting wireheaded.

I'm not discussing worst-case AGI scenarios here. I tried to imagine the best case scenario under some conditions, but failed.

In 'Terms and assumptions' section down below I tried, though also failed probably, to clarify things I assumed.

John's timeless self-delusion

I don't exactly remember anymore why I wrote this once, but I'll use this story to illustrate this post.

Suppose John wants to murder Bill, and has to escape arrest by the thought police that will come and check on him tomorrow morning. Luckily or not, John is also a meditation expert and can modify his perceptions and wishes. John could've dropped his intent towards Bill if he wanted to. However, to fulfill his desire at the time and make sure Bill gets killed, he chooses instead to meditate to a state of insanity, in which he believes that Bill is a superhuman that loves playing the surprise catch-the-brick-with-your-head game at mornings, and was his best friend all along. Tomorrow, after passing thought police's routine check, John proceeds to drop a brick on poor Bill's head from a roof, then finds himself confused and unhappy ever since.

Originally, the point of this story wasn't that John is a poorly-designed runaway AI that deludes itself, then does bad things. John was intended to be human, and a subject to experience and own qualia. (Although one arguably doesn't exclude another.) The point probably was to demonstrate it's difficult to describe decision-making based on preferences, when preferences change over time.

I found myself at lack of dictionary for distinguishing between future and past selves, describing various degrees of cooperation between them, and talking about subjective utility. I ...
Nice Things
by Zvi — 2018-03-31T12:30:00.341Z

Previously (not required): What Is Rationalist Berkeley’s Community Culture?

Response to (at Otium, highly recommended): Naming the Nameless

The strongest attractor is to that which attracts.

To pleasant interactions. To invisible, quiet competence. Things that ‘just work.’ Only the options you didn’t know you wanted. Where that which sounds right, and feels right, is right. A land of smiles and trust.

Nice things. They’re nice.

We can have them!

As Sarah puts it:

But for your typical consumer, the generic California/BoBo style works fine. It signals elegance, which means, more or less, that it’s designed for educated, high-Openness, upper-middle-class, urban people. When I enter a space or a website with this aesthetic, or buy a product with this branding, it’s shorthand for “Ahhhh, this place is run by competent professionals who know how to give me a pleasant experience. I will not feel harried or inconvenienced or confused here; I will be well taken care of. I will easily be able to slot my existing behavior patterns into the implicit “rules” of how to use and navigate this place or device or website.”

I gotta admit it. Nice things are pretty sweet. The best part about nice? When you’re done appreciating how nice it is, you can stop paying attention. Nice becomes background. Focus advances. Life is better.

I wants them.

But, at what cost? What is the price of nice?

Two prices.

The first price, attention to detail. The quest for nice things is a sacred quest. Creation of them, even more so. It requires effort, focus, sacrifice. You have to care.

Otherwise, we can’t have nice things. Because you didn’t make them.

The second price, the equivalence of niceness and rightness. Nice defends itself by banishing the not nice. In ways not nice. It pretends not to notice.

When you can’t or won’t pay for the shields, and don’t preserve them, that too is why you can’t have nice things. Because you break them.

If one uses not nice to banish and hide not nice, what counts as nice matters quite a bit.

Anyone, anything, anywhere that can create and preserve nice things, on any level, deserves praise. I may sometimes think of Berkeley as the devil, or the enemy, but one must give him his due on aesthetics. He’s a man of wealth and taste. On this, devil delivers.

But what, then, in such a place, is nice? Is nice no...
Reducing Agents: When abstractions break
by Hazard — 2018-03-31T00:03:16.763Z

Epistemic Effort: A month of dwelling on this idea, 12-16 hours of writing to explore the idea, and 2-5 hours rereading old LW stuff.

In the past few months, I’ve been noticing more things that lead me to believe there’s something incomplete about how I think about beliefs, motives, and agents. There’s been one too many instances of me wondering, “Yeah, but but what do you really believe?” or “Is that what you really want?”

This post is the first in a series where I'm going to apply

More Dakka

to a lot of Lesswrong ideas I was already familiar with, but hadn’t quite connected the dots on.

Here are the main points:

Agents are an abstraction for making predictions more quickly.
In what contexts does this abstraction break down?
What model should be used in places where it does break down?

Relevant LW posts (will also be linked throughout)

Reductionism 101

Blue Minimizing Robot

Adaptation-Executors, Not Fitness-Maximizers

Abstractions

Abstraction is awesome. Being able to make quality high level abstractions is like a super power that bends the universe to your will. Think of an abstraction as a model. A given abstraction has its ontologically basic building blocks defined, as well as the rules that govern their interactions.

Imagine a universe where 2x6 Lego bricks are the base level of reality. They connect to each other just like they do in our universe, but they do so axiomatically, not because of friction or anything. In this universe, we might make a higher level abstraction by defining a handful of multi-brick structures to be ontologically basic. You lose some of the resolution of having 2X6’s as your basis, but it also doesn’t take as long to make large things.

That’s the fundamental trade-off of abstractions. Each time you hop up a layer, you lose some detail (and thus, model accuracy) but you gain simplicity in computation. You could talk about a complex system like a computer on the quark level, but the computational time would be silly. Same for the atom or molecule layer. There’s a sparkle of hope when you get to the level of transistor being basic. Now you can quickly talk about tons of cool stuff, but talking about an entire computer is still out of your reach. Hop up to logic gates. Hop up to basic components like adders, multiplexers, and flip-flops. Now we've reached a level where you could actually design a useful piece of hardware that does ...
Welcome to Effective Altruism Tübingen [Edit With Your Details]
by Yannick_Muehlhaeuser — 2018-03-30T14:21:03.464Z

(The following are our suggestions for what kind of information is best to include in the welcome post of your group, feel free to replace them with whatever you think is best)

What kind of events does your group usually run? What does it usually do?

How frequently does your group organize events or meet?

Who would be a good fit for you group?

Should they have any particular skills or have done some specific background reading?
The Eternal Grind
by Zvi — 2018-03-30T11:40:00.526Z

Previously: Eternal, and Hearthstone Economy versus Magic Economy, Categories of Sacredness, Sacred Cash

Epistemic Status: Extensive ‘research.’ About card games more than general principles. Those here for the rationality likely want to skip this.

As previously noted, the game Eternal (that is my referral link), created by Magic professionals including lead designer Patrick Chapin, is modern Magic: The Gathering, with some simplifications and tweaks, on a phone, with a Hearthstone interface and economy.

Having played Eternal for several months, I feel qualified to offer a full review. After a brief overview of the game’s rules, I’ll cover Eternal’s limited and constructed formats, its economy, its grind and its special events.

This review assumes basic familiarity with Magic: The Gathering.

Rules Changes

The following are all the rules changes between Magic and Eternal.

1. Lots of things have new names, and some are slightly tweaked.

Basic lands, lands and mana are now sigils, power and power. Artifacts and enchantments both are now relics. Creatures are units. Auras you play on your own guys are weapons, auras you play on their guys are curses.

Magic has five colors: White, Blue, Black, Red and Green. Eternal has five ‘factions’ that map to them: Time, Primal, Shadow, Fire and Justice. A few abilities are swapped between Time and Justice, but otherwise the colors mean the same things they do in Magic.

Tapped is now exhausted, and cards are darkened out rather than turned sideways.

Trample becomes overwhelm. Flash becomes ambush and can redirect attacks from personal weapons. Deathtouch becomes deadly. First strike is now quickdraw and only works when attacking. I’m probably missing a few others.

Hexproof is now aegis. Aegis only works once, countering the first effect that targets or effects a creature. The big upgrade is that ageis can prevent destroy effects and non-combat damage, so it’s a shield against everything but combat/fight damage, and effectively forces you to kill the creature twice.

Drawing a card includes drawing from the graveyard (now called the void), and discarding includes milling effects.

2. Mana requirements are now threshold effects

What has changed is that instead of spending colored mana, one has influence with the factions. If I have two fire influence, I can cast as many fire spells as I like that require only one or two fire influence, rather ...
Reward hacking and Goodhart’s law by evolutionary algorithms
by Jan_Kulveit — 2018-03-30T07:57:05.238Z
Link: https://arxiv.org/abs/1803.03453

Nice collection of anecdotes from the Evolutionary Computation and Artificial Life research communities about evolutionary algorithms subverting researchers intentions, exposing unrecognized bugs in their code, producing unexpected adaptations, or exhibiting outcomes uncannily convergent with ones in nature. Some of my favorites:

In other experiments, the fitness function rewarded minimizing the difference between what the program generated and the ideal target output, which was stored in text files. After several generations of evolution, suddenly and strangely, many perfectly fit solutions appeared, seemingly out of nowhere. Upon manual inspection, these highly fit programs still were clearly broken. It turned out that one of the individuals had deleted all of the target files when it was run! With these files missing, because of how the test function was written, it awarded perfect fitness scores to the rogue candidate and to all of its peers

...

To test a distributed computation platform called EC-star [84], Babak Hodjat implemented a multiplexer problem [85], wherein the objective is to learn how to selectively forward an input signal. Interestingly, the system had evolved solutions that involved too few rules to correctly perform the task. Thinking that evolution had discovered an exploit, the impossibly small solution was tested over all possible cases. The experimenters expected this test to reveal a bug in fitness calculation. Surprisingly, all cases were validated perfectly, leaving the experimenters confused. Carefully examination of the code provided the solution: The system had exploited the logic engine’s rule evaluation order to come up with a compressed solution. In other words, evolution opportunistically offloaded some of its work into those implicit conditions.
Rationalist Lent is over
by Qiaochu_Yuan — 2018-03-30T05:57:03.117Z

Rationalist Lent is over. How did it go for everyone? Did you learn anything about yourself / superstimuli / pica / etc.?

I had a pretty good time giving up porn, which I stuck to. I also masturbated much more sparingly, which had interesting and pleasant effects on my sex drive. This is part of a longer project I'm engaging in of getting more in touch with my sexuality and it's been going well so far. I have an increasingly strong sense now of my past porn consumption as having been pretty bad for me - in machine learning terms, it's adversarial training data, very closely analogous to junk food. Porn, like many of my previous addictions (mostly various forms of fiction), has also felt much less tempting over the last 2 months because my life has gotten a lot better. Long story.
Resolving human values, completely and adequately
by Stuart_Armstrong — 2018-03-30T03:35:03.502Z

In previous posts, I've been assuming that human values are complete and consistent, but, finally, we are ready to deal with actual human values/preferences/rewards - the whole contradictory, messy, incoherent mess of them.

Define "completely resolving human values", as an AI extracting a consistent human reward function from the inconsistent data on the preference and values of a single human (leaving aside the easier case of resolving conflicts between different humans). This post will look at how such resolutions could be done - or at least propose an initial attempt, to be improved upon.

EDIT: There is a problem with rendering some of the LaTeX, which I don't understand. The draft rendered it fine, but not the published version. So I've replaced some LaTeX with unicode or image files; it generally works, but there are oversized images in section 3.

Adequate versus elegant

Part of the problem is resolving human values, is that people have been looking to do it too well and too elegantly. This results in either complete resolutions that ignore vast parts of the values (eg hedonic utilitarianism), or in thorough analyses of a tiny part of the problem (eg all the papers published on the trolley problem).

Incomplete resolutions are not sufficient to guide an AI, and elegant complete resolutions seem to be like most utopias: not any good for real humans.

Much better to aim for an adequate complete resolution. Adequacy means two things here:

It doesn't lead to disastrous outcomes, according to the human's current values.
If a human has a strong value or meta-value, that will strongly influence the ultimate human reward function, unless their other values point strongly in the opposite direction.

Aiming for adequacy is quite freeing, allowing you to go ahead and construct a resolution, which can then be tweaked and improved upon. It also opens up a whole new space of possible solutions. And, last but not least, any attempt to formalise and write a solution gives a much better understanding of the problem.

Basic framework, then modifications

This post is a first attempt at constructing such an adequate complete resolution. Some of the details will remain to be filled in, others will doubtlessly be changed; nevertheless, this first attempt should be instructive.

The resolution will be built in three steps:

a) It will provide a basic framework

...
Charting Deaths: Reality vs Reported
by  — 2018-03-30T00:50:00.314Z
Link: https://owenshen24.github.io/charting-death/

Some of you may recall an iconic study Yudkowsky references in his Availability essay, where Combs and Slovic looked at the different ways people died and newspaper reporting to see if they were proportional.

For my data science course at UCSD, my group and I attempted to replicate those findings with some more recent data!

(The link has lots more graphs, analysis, and links to the code used.)
Hufflepuff Cynicism on Hypocrisy
by abramdemski — 2018-03-29T21:01:29.179Z

Epistemic Status: I've been thinking about this for a number of years, looking for steelmen of the position against hypocrisy. I haven't found anything satisfying yet, but maybe you can tell me why hypocrisy is actually bad? Barring that, I'm rather confident in the view expressed here.

"Hypocrisy is bad" is a deeply-rooted assumption in our culture. If I'm trying to give someone advice, I'll flinch away from saying things I don't do myself. For example, I don't have a driver's license, so I would hesitate before suggesting that someone else get one. If I do suggest it, I'll get kind of apologetic, and hedge my statement with "but I haven't gotten one yet". This, despite the fact that whether I've gotten one is almost irrelevant to whether they should get one. It seems to me that this habit is universal in American culture, and I'd be surprised (and intrigued!) to hear about any culture where it isn't.

I think the anti-hypocrisy norm is likely based on a blame/praise model of advice. If advice is always norm-enforcing criticism, and is tied to how many "social points" you score, then the anti-hypocrisy norm prevents a particular type of social exploit. A hypocrite can enforce, and claim to believe in, norms which they themselves break. They score social points, gaining status and power, based on an unfair self-favoring application of rules. Rulers who are above the laws they themselves enforce violate our fairness norms; they are the very image of corruption.

However, this problem can be addressed in a more specific way by calling out unfairness, building power structures with transparency and accountability in mind, and deserving trust as a community. The cost of the anti-hypocrisy norm is too high; it throws out too much useful advice, constraining the directions in which we think when we're trying to help one another.

Rob Bensinger already wrote a whole post on why hypocrisy is a bad concept, which I also endorse. Ironically, though, he has a different concept of hypocrisy than me, so my argument against the concept is somewhat different. He treats "hypocrisy" as "inconsistency", and points out that winning an argument by pointing out inconsistency is not very informative: showing that someone holds two incompatible views doesn't tell y...
2018 Prediction Contest - Propositions Needed
by jbeshir — 2018-03-29T15:02:32.409Z

Summary: I think it is worth running a contest to measure ability to make accurate predictions, and am prepared to put up a prize of $200 for the winner. I need to identify a base of 20-50 propositions, all likely to settle within 2-10 months from now, and would like to ask the community for their suggestions, or for some suggested algorithms for picking them. I'd also like feedback on the idea.

Motivation

I think that practicing predicting future events significantly improves calibration and provides valuable feedback on one's own level of rationality, especially around the topics you are predicting on.

More fuzzily, I think that on the community level, a greater prevalence of predictions being made could provide gains throughout the community, through feedback on how effective our peers are at making predictions. I don't think it is quite a solution to the schools proliferating without evidence problem- it's a single very narrow metric- but schools proliferating with a single very narrow metric is a positive step, I think.

Thirdly, I think a lot of people would like to practice prediction-making, but do not get around to it for various reasons, one of which being the difficulty identifying what propositions to make predictions about. As a result, I think with a base of propositions, it would take quite a small expected value nudge to get a decent number of people to try making predictions.

Bring up all these thoughts together, and the candidate strategy of running a prediction contest came to mind pretty easily. And the simplest way to see if it is a good idea is to try it. If I get 20 entrants I'd consider it a weak success and worth running another next year- more would be more of a success. I put maybe 60% odds on that, conditional on it being run.

Proposed Plan

Once I have identified a set of 20-50 propositions, I'll create predictions for all of them on PredictionBook, and make a subsequent post here on Less Wrong, listing them all and announcing the contest.

From that time, anyone will have until a specified deadline (~1 month from posting time) to submit predictions on all of them, and give me a contact email and their PredictionBook account name through a Google Form. If people submit multiple predictions, the latest one before the deadline would be used (to remove the incentive to delay until the last minute to minimise uncer...