trentmkelly/LessWrong-43k huggingface dataset:
Should I do it?
by MrLight — 2020-11-19T01:08:10.506Z

Shall we bite the apple?

Hello community, nice to meet you here.

For a long time I have been tormented by a question that I can't seem to answer alone.

Let's talk hypothetically. Let's assume that I have developed a method that can be mapped in a small program. This method, once started, will develop an AGI by itself.

The program works independently, evolves and gets better from minute to minute. Not just better with one problem, but with any number of problems. The algorithm is able to rewrite and adapt itself. The algorithm follows this path of continuous improvement until it ends in an AGI.

This algorithm consists of several individual components, which together give this algorithm the capabilities described above. Imagine that I wrote and tested each of these individual components separately from each other in an alpha release (error driven design). And further imagine, each of these parts delivered the expected result.

The only thing left is to write these components in a faster language like C ++ or Assembler, bring them together and press 'Start'.

My dilemma is that I cannot ensure that the process can be stopped once it has started. I'm curious.

So I sit here and think. Should I do it and provide evidence? Or should I just wait, do nothing and watch someone else come up with the idea how to do it?

Or should I disclose the process without a program? Just publish it so that everyone can read it? A gift to humanity. The question is justified when you consider the risks.

But if you want to test it in a safe environment, then it's expensive and costs a lot, a lot of money. I am getting impatient.

... and if you keep walking on the paths of imagination, imagine that you have turned to your government about your dilemma, and to other governments as well, as well as to the economy, and nobody answers you ...

So I have two options.

A.) Either my aborted mathematics studies and 34 years of experience as a programmer led me to a dead end. I need a prove for myself.

B.) I'm right and nobody believes me. Either because I've never published anything or because they think it's technically impossible. Or they consider the risk to be insignificant, or the profit to be too small. I need a prove for them.

For (A) the solution is simple. Publish proceedings and someone will be found to prove that I am wrong. So I have the proof and I can do something else.

For (B) the...
2020 Election: Prediction Markets versus Polling/Modeling Assessment and Postmortem
by Zvi — 2020-11-18T23:00:01.166Z

Moderation/Commenting: I tried to minimize it, but this post necessarily involves some politics. See note at end of post for comment norms. I hope for this to be the last post I make that has to say anything about the election.

Previously: Evaluating Predictions in Hindsight, PredictIt: Presidential Market is Increasingly Wrong

There have been several posts evaluating FiveThirtyEight’s model and comparing its predictions to those of prediction markets. I believe that those evaluations so far have left out a lot of key information and considerations, and thus been highly overly generous to prediction markets and overly critical of the polls and models. General Twitter consensus and other reactions seem to be doing a similar thing. This post gives my perspective.

The summary is:

The market’s movements over time still look crazy before the election, they don’t look great during the election, and they look really, really terrible after the election.
The market’s predictions for safe states were very underconfident, and it looks quite stupid there, to the extent that one cares about that.
According to the Easy Mode evaluation methods, Nate’s model defeats the market.
The market’s predictions on states relative to each other were mostly copied from the polls slash Nate Silver’s model, which was very good at predicting the order, and the odds of an electoral college versus popular vote split at various margins of the popular vote.
The market’s claim to being better is that they gave Trump a higher chance of winning, implying similar to what Nate predicted if the final popular vote was Biden +4, which is about where it will land.
My model of the market says that it was not doing the thing we want to give it credit for, but rather representing the clash between two different reality tunnels, and the prices over time only make sense in that context.
The market priced in a small but sane chance of a stolen election before the election, but seems to have overestimated that chance after the election and not adjusted it over time.

We will go over the methods given in Evaluating Predictions in Hindsight for using Easy Mode, then ask remaining questions in Hard Mode.

First, we need to mention the big thing that most analysis is ignoring. Estimates were not made once at midnight before election day. Estimates were continuous by all parties until then, and then continued to be continuous...
The ethics of AI for the Routledge Encyclopedia of Philosophy
by Stuart_Armstrong — 2020-11-18T17:55:49.952Z

I've been tasked by the Routledge Encyclopedia of Philosophy to write their entry on the ethics of AI.

I'll be starting the literature reviews and similar in the coming weeks. Could you draw my attention to any aspect of AI ethics (including the history of AI ethics) that you think is important and deserves to be covered?

Cheers!
The Pointers Problem: Human Values Are A Function Of Humans' Latent Variables
by johnswentworth — 2020-11-18T17:47:40.929Z

An AI actively trying to figure out what I want might show me snapshots of different possible worlds and ask me to rank them. Of course, I do not have the processing power to examine entire worlds; all I can really do is look at some pictures or video or descriptions. The AI might show me a bunch of pictures from one world in which a genocide is quietly taking place in some obscure third-world nation, and another in which no such genocide takes place. Unless the AI already considers that distinction important enough to draw my attention to it, I probably won’t notice it from the pictures, and I’ll rank those worlds similarly - even though I’d prefer the one without the genocide. Even if the AI does happen to show me some mass graves (probably secondhand, e.g. in pictures of news broadcasts), and I rank them low, it may just learn that I prefer my genocides under-the-radar.

The obvious point of such an example is that an AI should optimize for the real-world things I value, not just my estimates of those things. I don't just want to think my values are satisfied, I want them to actually be satisfied. Unfortunately, this poses a conceptual difficulty: what if I value the happiness of ghosts? I don't just want to think ghosts are happy, I want ghosts to actually be happy. What, then, should the AI do if there are no ghosts?

Human "values" are defined within the context of humans' world-models, and don't necessarily make any sense at all outside of the model (i.e. in the real world). Trying to talk about my values "actually being satisfied" is a type error.

Some points to emphasize here:

My values are not just a function of my sense data, they are a function of the state of the whole world, including parts I can't see - e.g. I value the happiness of people I will never meet.
I cannot actually figure out or process the state of the whole world
… therefore, my values are a function of things I do not know and will not ever know - e.g. whether someone I will never encounter is happy right now
This isn’t just a limited processing problem; I do not have enough data to figure out all these things I value, even in principle.
This isn’t just a problem of not enough data, it’s a problem of what kind of data. My values depend on what’s going on “inside” of things which look the same - e.g. whether a smiling face is actually a rictus grin
This isn’t just a problem of needing suffic

...
Normativity
by abramdemski — 2020-11-18T16:52:00.371Z

Now that I've written Learning Normativity, I have some more clarity around the concept of "normativity" I was trying to get at, and want to write about it more directly. Whereas that post was more oriented toward the machine learning side of things, this post is more oriented toward the philosophical side. However, it is still relevant to the research direction, and I'll mention some issues relevant to value learning and other alignment approaches.

How can we talk about what you "should" do?

A Highly Dependent Concept

Now, obviously, what you should do depends on your goals. We can (at least as a rough first model) encode this as a utility function (but see my objection).

What you should do also depends on what's the case. Or, really, it depends on what you believe is the case, since that's what you have to go on.

Since we also have uncertainty about values (and we're interested in building machines which should have value uncertainty as well, in order to do value learning), we have to talk about beliefs-about-goals, too. (Or beliefs about utility functions, or however it ends up getting formalized.) This includes moral uncertainty.

Even worse, we have a lot of uncertainty about decision theory -- that is, we have uncertainty about how to take all of this uncertainty we have, and make it into decisions. Now, ideally, decision theory is not something the normatively correct thing depends on, like all the previous points, but rather is a framework for finding the normatively correct thing given all of those things. However, as long as we're uncertain about decision theory, we have to take that uncertainty as input too -- so, if decision theory is to give advice to realistic agents who are themselves uncertain about decision theory, decision theory also takes decision-theoretic uncertainty as an input. (In the best case, this makes bad decision theories capable of self-improvement.)

Clearly, we can be uncertain about how that is supposed to work.

By now you might get the idea. "Should" depends on some necessary information (let's call them the "givens"). But for each set of givens you claim is complete, there can be reasonable doubt about how to use those givens to determine the output. So we can create meta-level givens about how to use those givens.

Rather than stopping at some finite level, such as learning the human utility function, I'm claiming that we should lea...
What would a world of widespread statistical numeracy look like?
by Pontor — 2020-11-18T16:23:41.613Z

I was just reading the thread comparing covid and tobacco, and it made me start wondering about the effect of statistical numeracy in general.
Personally, I have a lot of room for improvement when it comes to these skills (but at least I am aware of this). I do regularly notice the difference in my impression when someone talks about a 3x increase vs a 300% increase, or 1/1000 vs 0.1%, etc.; and I often make a quick conversion in my head when it's convenient. I also know a few mortality stats by heart which I can use to very roughly benchmark certain claims I hear about risk and safety.

Frequently, when I practice this minimal numeracy, it is accompanied by a sense of futility. When the stakes mostly involve policy-making or group action, my own statistical literacy may be inactionable--it may make basically no difference to my life or the world. What matters instead is what sorts of political messages resonate with voters, or what sorts of heuristics will catch on, etc.

So to sharpen my question, suppose you went back in time 20 years, magically caused the whole world to be much more numerate, and then just lived normally for the next 20 years. What about this world, if anything, would be drastically different from our own world?
(For those who want to get serious about the hypothetical: Let's say that in this alternate world, a typical high school graduate in the US has been trained in the habits of mind outlined in the second paragraph. Let's also say that numeracy and literacy track closely--so in any given country, you would be just as surprised to witness base-rate neglect as you would be to witness an inability to read road signs. Feel free to ask for more details or to tweak the hypothetical yourself.)
Should we postpone AGI until we reach safety?
by otto.barten — 2020-11-18T15:43:51.846Z

Should we postpone AGI, until its risks have fallen below a certain level, thereby applying the precautionary principle? And if so, would setting up policy be a promising way to achieve this?

As I argued here in the comments, I think calling for precautionary principle-policy, notably towards political decision makers, would be a good idea. I've had a great LW and telephone discussion with Daniel Kokotajlo about this, who disagrees, with the arguments below. I think it is valuable to make our lines of reasoning explicit and sharpen them by discussion, which is why I'm promoting them to a post.

Assuming AGI in a relevant time span, there are two ways in which humanity can decrease x-risk to acceptable levels: 1) AI alignment, consisting both of technical AI safety and reasonable values alignment, and 2) AGI postponement until 1 can be achieved with sufficient existential safety (this may be anywhere between soon and never).

Since it is unsure whether we can achieve 1 at all, and also whether we can achieve it in time assuming we can achieve it in principle, we should aim to achieve 2 as well. The main reason is that this could lead to a significant reduction of total existential risk if successful, and we don't know much about how hard it is, so it could well be worthwhile. Companies and academics are both not incentivized to postpone limited AGI development in accordance with the precautionary principle. Therefore, I think we need a different body calling for this, and I think states make sense. As a positive side effect, pressure from possible postponement on companies and academia would incentivize them to invest significantly more in alignment, thereby again reducing existential risk.

Daniel disagreed with me mostly because of three reasons. I obtained two more counterarguments from the discussions below and elsewhere. The following so far appears to be a reasonably complete list of counterarguments.

Calling for AGI postponement until safety has been achieved, he thinks, would alienate the AI community from the AI safety community and that would hurt the chances of achieving AI safety.
It would be rather difficult (or extremely difficult, or impossible) to get useful regulations passed, because influencing governments is generally hard and influencing them in the right ways is even harder.
Restricting AGI research while allowing computer hardware progress, other

...
Sunzi's《Methods of War》- Introduction
by lsusr — 2020-11-18T08:23:06.394Z

This is a translation of the Chapter 1 of The Art of War by Sunzi. No English sources were used. The original text and many of the interpretations herein come from 古诗文网.

孙子曰：兵者，国之大事，死生之地，存亡之道，不可不察也。

War determines life and death of troops, existence and destruction of a country. It cannot be ignored.

故经之以五事，校之以计，而索其情：一曰道，二曰天，三曰地，四曰将、五曰法。

Five aspects are of paramount important:

Dao

Heaven

Earth

Generalship

Method

道者，令民与上同意也，故可以与之死，可以与之生，而不畏危。

"Dao" concerns alignment. Your side must be unified. By dying together, living together, you shall be unafraid.

天者，阴阳，寒暑、时制也。

"Heaven" concerns timing, yin and yang, winter and summer.

地者，远近、险易、广狭、死生也。

"Earth" concerns the near and far, impassable and passable, open fields and choke points, death and life.

将者，智、信、仁、勇、严也。

"Generalship" is a matter of wisdom, fidelity, benevolence, bravery and severity.

法者，曲制、官道、主用也。

"Method" concerns tactics, doctrine and organization.

凡此五者，将莫不闻，知之者胜，不知者不胜。

A commander must not ignore these five aspects. Understanding them brings victory. Lack of understanding does not bring victory.

故校之以计，而索其情，曰：主孰有道？将孰有能？天地孰得？法令孰行？兵众孰强？士卒孰练？赏罚孰明？

Ask yourself: Are ruler and subjects aligned? Is the general capable? Heaven (climate) and Earth (geography) in your favor? Methods effective? Troops strong? Trained? Enlightenedly punished?

吾以此知胜负矣。将听吾计，用之必胜，留之；将不听吾计，用之必败，去之。计利以听，乃为之势，以佐其外。势者，因利而制权也。

These things determine victory and defeat.

兵者，诡道也。故能而示之不能，用而示之不用，近而示之远，远而示之近；利而诱之，乱而取之，实而备之，强而避之，怒而挠之，卑而骄之，佚而劳之，亲而离之。攻其无备，出其不意。此兵家之胜，不可先传也。

The art of war depends on local conditions. The near informs you about the far. The far informs you about the near.

If the enemy is clever then tempt.

If the enemy is disordered then raid.

If the enemy is capable then prepare.

If the enemy is mighty then run.

If the enemy is angry then provoke.

If the enemy is inferior then threaten.

If the enemy is dissolute then persevere.

Attack where the enemy is unprepared. Do what is least expected. But do not forget the five aspects. They are of primary importance.

夫未战而庙算胜者，得算多也；未战而庙算不胜者，得算少也。多算胜，少算不胜，而况于无算乎？吾以此观之，胜负见矣。

A war cannot be won without lots of equipment. This facet of war too must be examined.
Comparative Advantage Intuition
by arxhy — 2020-11-18T01:23:36.549Z

On an intuitive level, how much should comparative advantage be taken into account when deciding on a life path?

It seems unrealistic to analyze every factor involved to maximize utility, but without doing that, there is a wide range of extents to which comparative advantage can be favored or disfavored compared to focusing on more important issues.

There are, I'm sure, other factors that don't instantly spring to mind that affect this significantly, like how easily a given person develops new skills. I am also asking because I don't know what all of those factors are.
Thoughts on Voting Methods
by abramdemski — 2020-11-17T20:23:07.255Z

I've been nerd-sniped by voting theory recently. This post is a fairly disorganized set of thoughts.

Condorcet Isn't Utilitarian

The condorcet criterion doesn't make very much sense to me. My impression is that a good chuck of hard-core theorists think of this as one of the most important criteria for a voting method to satisfy. (I'm not really sure if that's true.)

What the condorcet criterion says is: if a candidate would win pairwise elections against each other candidate, they should win the whole election.

Here's my counterexample.

Consider an election with four candidates, and three major parties. The three major parties are at each other's throats. If one of them wins, they will enact laws which plunder the living daylights out of the losing parties, transferring wealth to their supporters.

The fourth candidate will plunder everyone and keep all the wealth. However, the fourth candidate is slightly worse at plundering than the other three.

We can model this scenario with just three voters for simplicity. Here are the voter utilities for the different candidates:

Candidates
Voters	A	B	C	D
1	100	0	0	1
2	0	100	0	1
3	0	0	100	1

D would beat everyone in a head-to-head election. But D is the worst option from a utilitarian standpoint!! Furthermore, I think I endorse the utilitarian judgement here. This is an election with only terrible options, but out of those terrible options, D is the worst.

VSE Isn't Everything

VSE is a way of basically calculating a utilitarian score for an election method, based on simulating a large number of elections. This is great! I think we should basically look at VSE first, as a way of evaluating proposed systems, and secondarily evaluate formal properties (such as the condorcet criterion, or preferably, others that make more sense) as a way of determining how robust the system is to crazy scenarios.

But I'm also somewhat dissatisfied with VSE; I think there might be better ways of calculating statistical scores for voting methods.

Candidate Options Matter

As we saw in the example for Condorcet, an election can't give very good results if all the candidates are awful, no matter how good the voting method.

Voting Methods Influence Candidate Selection

Some voting methods, specifically plurality (aka first-past-the-post) and instant runoff voting, are known to create incentive dynamics which encourage two-party systems to eventually emerge.

In order...
On the Shoulders of Giants
by meanderingmoose — 2020-11-17T18:10:53.819Z

A couple hundred thousand years ago, our species reached a point where we could pass knowledge directly (through language and demonstrated action), rather than relying on adaptations to be passed on through genes. With this shift, culture was born – and it allowed us to very rapidly acquire a great deal of knowledge about our world (with the rate of progress escalating over the last several hundred years). However, due to the cultural nature of this knowledge, an individual human separated from these shared learnings still cannot get very far. Our best examples of this phenomenon come in the form of feral children (of which there are thankfully few); in these cases, growing up separated from society has resulted in a level of intelligence comparable to our Great Ape cousins (with limited ability to learn further). Though our knowledge sits outside our genes, our genetic makeup does have an influence, as we’ve evolved to be good at learning from others. Babies can recognize (and pay attention to) faces essentially from birth, and throughout childhood (and beyond) we rely heavily on observing and listening to others to learn about the world.

Our accumulated cultural knowledge has allowed us to climb to the top of Earth’s biological hierarchy, and to learn far deeper truths about our world and ourselves than any other species has uncovered. We’ve discovered atoms and quarks, gravity and electromagnetism, our place in the universe, and even some basic principles of how our brains work to do these things. Each discovery we make extends the shared knowledge base and helps propel it further forward, as we’re able to share and pass on the insights. Newton famously wrote, “If I have seen further, it is by standing upon the shoulders of giants.” For him, these giants were the scientists who came before him, but in reality we have our whole lineage of giants to thank – from the first users of tools, to the first speakers of language, to the first formers of settlements. These early discoveries laid the groundwork from which all future cultural knowledge has grown.

Our advanced ability to engage with the world is dependent on absorbing this cultural knowledge; as noted before, without it, we’re no different than other apes (for additional detail on this topic, check out The Secret of Our Success, by Joseph Henrich). However, to say we simply “absorb” the...
Welcome to EA Brussels [Edit With Your Details]
by masasin — 2020-11-17T17:40:39.591Z

(The following are our suggestions for what kind of information is best to include in the welcome post of your group, feel free to replace them with whatever you think is best)

What kind of events does your group usually run? What does it usually do?

How frequently does your group organize events or meet?

Who would be a good fit for you group?

Should they have any particular skills or have done some specific background reading?