trentmkelly/LessWrong-43k huggingface dataset:
The Clockmaker's Argument (But not Really)
by GregorDeVillain — 2019-02-17T21:20:17.451Z

First things first, the intention of this post is not to promote theism, deism or religion in any way, shape or form. I am rather gnostic on the atheism spectrum so that's really not something I intend.

Having said that, I am right up to here with weak arguments supporting God due to religious inability to make concessions so I decided that if I am to properly be rational about this, I might as well make the best argument I ever could for the existence of God if I were so inclined. Given the nature of religion when it comes to blows with science, it might well be something we'll have to face when the church faces its last leg and making last ditch efforts.

So, here goes:

There is this EXCEPTIONAL human being, we'll call him Bob.

Bob is an expert. To mention all the subjects he's an expert at would take us all day but suffice to say he's at least one of the best, if not THE best, in the whole entire fields (yes, every single subgroup of them, please suspend your disbelief) of biology, physics, chemistry, architecture, engineering (again, all the fields of engineering) and everything in between. He's by all accounts the most brilliant and knowledgeable mind in the history of mankind.

He's also at the peak of human physical condition. Gotten many gold Olympic metals, and he surely would again if he competed, has genes to develop his muscles even when he isn't working out, is a short-sleeper, and he has such a low probability of genetic diseases, no one bothers counting the 0s. Indeed, he may well break all the records for longevity in the coming years by a fair margin.

Now Bob is quite fascinsted by the thought of time. He is so interested in fact that, like all minds so far above the median, decides to choose it as his life destroying obsession. He studies it in all its ways, shapes and forms and each time he considers how inaccurate even the best time pieces in the world. A whole yoctosecond lost per second? Preposterous. Surely he could do better.

So he decides to create a digital clock based on quantum computing, using thousands of optical single ion built in one singular hardware with a software so advanced to calibrate and autocorrect and solve possible problems as they arise it gives rise to its own intelligence. It's not much of an intelligence, indeed though it has the processing power, its flexibility is quite limited in relevance to a regular politicized hu

...
Can We Place Trust in Post-AGI Forecasting Evaluations?
by ozziegooen — 2019-02-17T19:20:41.446Z

TLDR

Think "A prediction market, where most questions are evaluated shortly after an AGI is developed." We could probably answer hard questions more easily post-AGI, so delaying them would have significant benefits.

Motivation

Imagine that select pre-AGI legal contracts stay valid post-AGI. Then a lot of things are possible.

There are definitely a few different scenarios out there for economic and political consistency post-AGI, but I believe there is at least a legitimate chance (>20%) that legal contracts will exist for what seems like a significant time (>2 human-experiential years.)

If these contracts stay valid, then we could have contracts set up to ensure that prediction evaluations and prizes happen.

This could be quite interesting because post-AGI evaluations could be a whole lot better than pre-AGI evaluations. They should be less expensive and possibly far more accurate.

One of the primary expenses now with forecasting setups is the evaluation specification and execution. If these could be pushed off while keeping relevance, that could be really useful.

Idea

What this could look like is something like a Prediction Tournament or Prediction Market where many of the questions will be evaluated post-AGI. Perhaps there would be a condition that the questions would only be evaluated if AGI happens within 30 years, and in those cases, the evaluations would happen once a specific threshold is met.

If we expect a post-AGI world to allow for incredible reasoning and simulation abilities, we could assume that it could make incredibly impressive evaluations.

Some example questions:

To what degree is each currently-known philosophical system accurate?

What was the expected value of Effective Altruist activity Y, based on the information available at the time to a specific set of humans?

How much value has each Academic field created, according to a specific philosophical system?

What would the GDP of the U.S. have been in 2030, conditional on them doing policy X in 2022?

What were the chances of AGI going well, based on the information available at the time to a specific set of humans?

Downsides

My guess is that many people would find this quite counterintuitive. Forecasting systems are already weird enough.

There's a lot of uncertainty around the value systems and epistemic l states of authoritative agencies, post-AGI. Perhaps they would be so incredibly dif...
Extraordinary ethics require extraordinary arguments
by aaq — 2019-02-17T14:59:17.156Z

Previous post: Fighting the allure of depressive realism

My blog entries are about a personal battle against depression and anxiety, from the point of view of someone who has been immersed in rationalist/LW ideas and culture for a few years now.

I want to illustrate a particular, recurring battle I have with scrupulosity. (I'm not the best at dialogues, so bear with me for a moment.)

Me: Alright, it's time to be productive and get to my homework.

???: Hold on! How can you possibly justify that if you haven't solved ethics yet?

Me: What? Who are you?

SD: Allow me to introduce myself, I'm your Skeptic Demon. I whisper ethical concerns in your ear to make sure you're always doing the right thing.

Me: That sounds more like a daemon than a demon to me.

SD: Demon. Trust me.

Me: Solving ethics can't possibly be expected of a single person, demon.

SD: Right you are! But you've looked around the world enough to know that everything you do could have ripple effects that might be disastrous. So how can you possibly feel good about working on your homework without accounting for that?

Me: What? It's just homework.

SD: Oh, no it isn't. Doing well on homework means sending a better signal to employers means more people want to hire you down the line, including for unscrupulous activities. And you've done not-great things before, so we can't be sure you'll resist. In fact the existence of first-, second-, third-, and nth-order effects implies you might not even realize when you're being offered such.

Me: Erm... Well, it's true that things have unintended consequences, but--

SD: No "buts"! You want to be a good person, right? So we gotta reason this out.

Me: I guess you have a point...

SD: Alright. So let's get started.

(hours pass)

SD: Okay. You're on shaky ice with some of these considerations. I'm not totally convinced you won't be tempted by the money to go and do something net harmful yourself, but I will give you a one-time pass. You may proceed to start your assignment .

Me: I'm exhausted and I just want to go to sleep now.

SD: Then my work is done here. *disappears in a puff of shaky logic*

This kind of conversation happens to me all the time. Why?

On one level, it's easy to see what the skeptic demon is doing. He's trolling. He's keeping me from doing the actual produc...
Limiting an AGI's Context Temporally
by EulersApprentice — 2019-02-17T03:29:26.634Z

Okay, so I have a proposal for how to advance AI safety efforts significantly.

Humans experience time as exponential decay of utility. One dollar now is worth two dollars some time in the future, which is worth eight dollars even further in the future, and so forth. This is the principle behind compound interest. Most likely, any AI entities we create will have a comparable relationship with time.
So: What if we configured an AI's half-life of utility to be much shorter than ours?

Imagine, if you will, this principle applied to a paperclip maximizer. "Yeah, if I wanted to, I could make a ten-minute phone call to kick-start my diabolical scheme to take over the world and make octillions of paperclips. But that would take like half a year to come to fruition, and I assign so little weight to what happens in six months that I can't be bothered to plan that far ahead, even though I could arrange to get octillions of paperclips then if I did. So screw that, I'll make paperclips the old-fashioned way."
This approach may prove to be a game-changer in that it allows us to safely make a "prototype" AGI for testing purposes without endangering the entire world. It improves AGI testing in two essential ways:

Decreases the scope of the AI's actions, so that if disaster happens it might just be confined to the region around the AGI rather than killing the entire world. Makes safety testing much safer on a fundamental level.
Makes the fruits of the AI more obvious more quickly, so that iteration time is shortened drastically. If an AI doesn't care about any future day, it will take no more than 24 hours to come to a conclusion as to whether it's dangerous in its current state.

Naturally, finalized AGIs ought to be set so that their half-life of utility resembles ours. But I see no reason why we can't gradually lengthen it over time as we grow more confident that we've taught the AI to not kill us.

(Note: There are 6x10^27 grams of matter on Earth. Throw in a couple orders of magnitude for the benefit of being undisturbed and this suggests that taking over the world represents a utility bonus of roughly 10^30. This is pretty close to 2^100, which suggests that an AGI will not take over the world if its fastest possible takeover scheme would take more than 100 half-lives. Of course, this is just Fermi estimation here, but ...
Major Donation: Long Term Future Fund Application Extended 1 Week
by habryka — 2019-02-16T23:30:11.243Z

(Crossposted to the EA Forum)

The Long-Term Future EA Funds application round is extended for one more week (until Midnight (PST) Sunday 24th of February).

The LTF Fund received a major donation such that the amount of money available for granting has more than doubled, to a total of ~$1.2 million, making our available funds greater than all money that the LTF fund has ever given away historically.

This means we can consider:

Grants of much larger ambition and scope.
Grants with longer time-horizons (who will need funding in future years)
Grants to larger teams
Seed funding for new organisations
Many more grants to smaller projects and individuals

I think this means applying this round is a particularly good idea, as well as forwarding the application to anyone else you know who might want to apply.

Apply here.

More info on the LTF Fund can be found here, you can see our last round of grants here.

If you’re uncertain about whether we’re interested in funding a project, you can ask me in a comment, or email me and the other fund managers at ealongtermfuture@gmail.com.
Is there a way to hire academics hourly?
by Ixiel — 2019-02-16T14:21:18.679Z

Seems there should be but I'm not finding it. Craigslist and Upwork don't seem to, but I was (am actually; any takers?) looking for someone with expertise in geography and global mythology and it seems pretty clear to me the value to me is probably higher than they'd charge with those specialties, so I assume it's just my very weak googling skills getting in the way.

Thanks in advance, and if it doesn't exist and you want to build it I expressly release any claim to a cut of the profits,

Charlie
Graceful Shutdown
by Martin Sustrik — 2019-02-16T11:30:00.927Z

Within a structured concurrency world, graceful shutdown is a little understood, little addressed and confusing topic.

I've been fighting with it myself for almost a year (longer than with any other problem I can recall) and it haven't been a pleasant experience. If felt like running in circles. Too many contradictory requirements, to much complexity and too often a solution I've came up with turned out to be one of the solution I've already rejected in disguise.

Nathaniel Smith on structured concurrency forum helpfully pointed me to the discussion about graceful shutdowns in Trio and oh boy, does that discussion sound familiar

Anyway, finally I've came up with a solution that's simple and works for all use cases. But it turns out it's one of those ideas that seem not to make sense until something clicks in one's mind. From there on though, they seem perfectly obvious, up to the point where one wonders how the others can possibly not get it.

In this post I'll try to explain.

The use case

The classic use case for graceful shutdown is a web server. It's handling many connections from the clients. When it's being shut down, you want it to happen in a controlled manner so that clients don't end up with half-loaded pages and such. Specifically, you want to stop the accept loop to make sure that no new connections are being made. Then you want to wait for say 10 seconds to give existing connections time to finish their work. Finally, when the time is up, you want to cancel any remaining connections and exit the process.

Of course, the nature of the workload may vary. Web server needs to shut down connections. CRUD application needs to shut down its connection to database. Yet different application may want to save data to disk before exiting.

What's graceful shutdown, anyway?

Graceful shutdown intuitively seems to be some kind of cancellation. Here, I would like to compare it to the classic hard cancellation and argue that graceful shutdown isn't anything like it.

Let's have a look at hard coroutine cancellation first. The code of the coroutine looks like this

if(rc == ECANCELED) return;
rc = recv(s, &msg);
if(rc == ECANCELED) return;
rc = sleep(60); i
f(rc == EANCELED) return;

When parent coroutine hard-cancels a child coroutine, a blocking call within the coroutine will immediately return an error (o...
Why didn't Agoric Computing become popular?
by Wei Dai — 2019-02-16T06:19:56.121Z

I remember being quite excited when I first read about Agoric Computing. From the authors' website:

Like all systems involving goals, resources, and actions, computation can be viewed in economic terms. This paper examines markets as a model for computation and proposes a framework--agoric systems--for applying the power of market mechanisms to the software domain. It then explores the consequences of this model and outlines initial market strategies.

Until today when Robin Hanson's blog post reminded me, I had forgotten that one of the authors of Agoric Computing is Eric Drexler, who also authored Comprehensive AI Services as General Intelligence, which has stirred a lot of recent discussions in the AI safety community. (One reason for my excitement was that I was going through a market-maximalist phase, due to influences from Vernor Vinge's anarcho-captalism, Tim May's crypto-anarchy, as well as a teacher who was a libertarian and a big fan of the Austrian school of economics.)

Here's a concrete way that Agoric Computing might work:

For concreteness, let us briefly consider one possible form of market-based system. In this system, machine resources-storage space, processor time, and so forth-have owners, and the owners charge other objects for use of these resources. Objects, in turn, pass these costs on to the objects they serve, or to an object representing the external user; they may add royalty charges, and thus earn a profit. The ultimate user thus pays for all the costs directly or indirectly incurred. If the ultimate user also owns the machine resources (and any objects charging royalties), then currency simply circulates inside the system, incurring computational overhead and (one hopes) providing information that helps coordinate computational activities.

When later it appeared as if Agoric Computing wasn't going to take over the world, I tried to figure out why, and eventually settled upon the answer that markets often don't align incentives correctly for maximum computing efficiency. For example, consider an object whose purpose is to hold onto some valuable data in the form of a lookup table and perform lookup services. For efficiency you might have only one copy of this object in a system, but that makes it a monopolist, so if the object is profit maximizing (e.g., running some algorithm that automatically adjusts prices so as to maximize p

...
Pedagogy as Struggle
by  — 2019-02-16T02:12:03.665Z

[See also: On Giving Advice and Recognizing vs Generating]

So this quarter, I’ve been tutoring for an undergraduate computer science course, and one interesting thing that I’ve revised is how I think about teaching/learning.

Let me try to illustrate. Here’s a conversation that happened between some staff and the professor one day; it’s about what advice to give students when testing their code.

Tutor A: “I think that we should remind students to be very thorough about how they approach testing their code. Like, so far, I’ve been telling them that every line of code they write should have a test.”

Tutor B: “Hmmm, that might backfire. After all, lots of those lines of code are in functions, and we really just want them to make sure that their functions are working as intended. It’s not really feasible to test lines of code inside a function.”

Professor: “Actually…that seems like a fine outcome. We want students to be thinking about testing, and I’d actually be very excited if someone came up to me and asked how to test for what goes on inside a function…”

This was surprising to me because most discussions I’ve previously had about student learning had focused on how to reduce confusion for the students. But in this scenario, the professor was fine with it happening; if anything, they seemed pleased that the concepts, when taken to their extremes, incited more questions.

And this general concept, of giving students something that's Not The Answer, in an effort to move them closer to The Answer seems to show up in several other areas.

For example, from my shallow understanding of how kōans work in Zen Buddhism, there’s a similar mechanic going on. The point of a kōan isn’t to develop a fully satisfactory answer to the question it asks, but to wrestle with the strangeness / paradoxical nature of the kōan. The auxiliary things that happen along the way, en route to the answer is really what the kōan is about.

To be clear, the thing I’m trying to point at isn’t just giving people practice problems, the way that we already do for math or physics.

Rather, I’m imagining things like people purposefully writing incorrect / confusing material, such that it prompts students to ask more questions. I guess there’s already a good amount of people in the rationality space who write abstrusely, but I wonder how many are doing so for pedagogical reasons? Seeing as noticing confusi...
How the MtG Color Wheel Explains AI Safety
by Scott Garrabrant — 2019-02-15T23:42:59.637Z

Duncan Sabien has a post titled How the ‘Magic: The Gathering’ Color Wheel Explains Humanity. Without the context of that post, or other experience with the MtG color wheel, this post will probably not make sense. This post may not make sense anyway. I will use a type of analysis that is sometimes used to talk about humans (and often criticized even when used for humans), but rarely used in any technical subjects. I will abstract so far that everything will start to look like (almost) everything else. I will use wrong categories and stretch facts to make them look like they fit into my ontology.

I will describe 5 clusters of ideas in AI and AI safety, which correspond to the 5 Magic the Gathering colors. Each color will also come along with a failure mode. For each failure mode, the two opposing colors (on the opposite side of the pentagon) form a collection of tools and properties that might be useful for fighting that failure mode.

Mutation and Selection

So I want to make an AI that can accomplish some difficult task without trying to kill me (or at least without succeeding in killing me). Let's consider the toy task of designing a rocket. First, I need a good metric of what it means to be a good rocket design. Then, I need to search over all the space of potential rocket designs, and find one that scores well according to my metric. I claim that search is made of two pieces: Mutation and Selection, Exploration and Optimization, or Babble and Prune.

Mutation and Selection are often thought of as components of the process of evolution. Genes spin off slightly modified copies over time through mutation, and selection repeatedly throws out the genes that score badly according to a fitness metric (that is itself changing over time). The result is that you find genes that are very fit for survival.

However, I claim that mutation and selection are much more general than evolution. Gradient descent is very close to (a speed up of) the following process. Take an initial point called your current best point. Sample a large number of points within an epsilon ball of the current best point. Select the best of the sampled points according to some metric. Call the selected point the new current best point, and repeat.

I am not trying to claim that machine learning is simple because it is mostly just mutation and selection. Rather, I am trying to claim that many of the com...
Some disjunctive reasons for urgency on AI risk
by Wei Dai — 2019-02-15T20:43:17.340Z

(This has been sitting in my drafts folder since August 2017. Robin Hanson's recent How Lumpy AI Services? made me think of it again. I'm not sure why I didn't post it back then. I may have wanted to add more reasons, details and/or citations, but at this point it seems better to just post it as is. Apologies to those who may have come up with some of these arguments earlier.)

Robin Hanson recently wrote, "Recently AI risk has become something of an industry, with far more going on than I can keep track of. Many call working on it one of the most effectively altruistic things one can possibly do. But I’ve searched a bit and as far as I can tell that foom scenario is still the main reason for society to be concerned about AI risk now." (By "foom scenario" he means a local intelligence explosion where a single AI takes over the world.) In response, I list the following additional reasons to work urgently on AI alignment.

Property rights are likely to not hold up in the face of large capability differentials between humans and AIs, so even if the intelligence explosion is likely global as opposed to local, that doesn't much reduce the urgency of working on AI alignment.

Making sure an AI has aligned values and strong controls against value drift is an extra constraint on the AI design process. This constraint appears likely to be very costly at both design and run time, so if the first human level AIs deployed aren't value aligned, it seems very difficult for aligned AIs to catch up and become competitive.

AIs' control of the economy will grow over time. This may happen slowly in their time frame but quickly in ours, leaving little time to solve value alignment problems before human values are left with a very small share of the universe, even if property rights hold up.

Once we have human-level AIs and it's really obvious that value alignment is difficult, superintelligent AIs may not be far behind. Superintelligent AIs can probably find ways to bend people's beliefs and values to their benefit (e.g., create highly effective forms of propaganda, cults, philosophical arguments, and the like). Without an equally capable, value-aligned AI to protect me, even if my property rights are technically secure, I don't know how I would secure my mind.