trentmkelly/LessWrong-43k huggingface dataset:
Industrial literacy
by jasoncrawford — 2020-09-30T16:39:06.520Z
Link: https://rootsofprogress.org/industrial-literacy

I’ve said before that understanding where our modern standard of living comes from, at a basic level, is a responsibility of every citizen in an industrial civilization. Let’s call it “industrial literacy.”

Industrial literacy is understanding…

That the food you eat is grown using synthetic fertilizers, and that this is needed for agricultural productivity, because all soil loses its fertility naturally over time if it is not deliberately replenished. That before we had modern agriculture, more than half the workforce had to labor on farms, just to feed the other half. That if synthetic fertilizer was suddenly lost, a mass famine would ensue and billions would starve.
That those same crops would not be able to feed us if they were not also protected from pests, who will ravage entire fields if given a chance. That whole regions used to see seasons where they would lose large swaths of their produce to swarms of insects, such as boll weevils attacking cotton plants in the American South, or the phylloxera devouring grapes in the vineyards of France. That before synthetic pesticides, farmers were forced to rely on much more toxic substances, such as compounds of arsenic.
That before we had electricity and clean natural gas, people burned unrefined solid fuels in their homes—wood, coal, even dung (!)—to cook their food and to keep from freezing in winter. That these primitive fuels, dirty with contaminants, created toxic smoke: indoor air pollution. That indoor air pollution remains a problem today for 40% of the world population, who still rely on pre-industrial fuels.
That before twentieth-century appliances, housework was a full-time job, which invariably fell on women. That each household would spend almost 60 hours a week on manual labor: hauling water from the well for drinking and cooking, and then carrying the dirty water outside again; sewing clothes by hand, since store-bought ones were too expensive for most families; laundering clothes in a basin, scrubbing laboriously by hand, then hanging them up to dry; cooking every meal from scratch. That the washing machine, clothes dryer, dishwasher, vacuum cleaner, and microwave are the equivalent of a full-time mechanical servant for every household.
That plastics are produced in enormous quantities because, for so many purposes—from food containers to electrical wire coatings to children’s toys—we need a materia

...
Jason Crawford on the non-linear model of innovation: SSC Online Meetup
by JoshuaFox — 2020-09-30T10:13:13.853Z

Jason Crawford on the non-linear model of innovation.

Please signup here by an hour before the event, and we will send you an invitation

https://forms.gle/XXk3hPNkcEJJbHFC9

IMPORTANT: As Daylight Savings Time approaches an end, note the times: 18:30 UTC, 20:30 IST, 11:30 PDT.

Innovation is often described with a “linear” model from discovery to invention to distribution. There is an element of truth in this, but a naive interpretation of the model does not match the reality of science and invention. In this talk, I'll show the feedback mechanisms between discovery and invention and how they are intertwined, using examples including the transistor at Bell Labs and the career of Louis Pasteur.

Jason Crawford is the author of The Roots of Progress (https://rootsofprogress.org/), where he writes about the history of technology and the philosophy of progress. Previously, he spent 18 years as a software engineer, engineering manager, and startup founder.
Holy Grails of Chemistry
by chemslug — 2020-09-30T02:03:07.289Z

This is a linkpost-slash-suggestion to check out https://www.chemistryworld.com/holy-grails/.

In 1995, the journal Accounts of Chemical Research published a series of articles called Holy Grails of Chemistry. This series of articles represents an attempt by a group of chemists to make a partial list of Hamming questions for chemistry. They chose eight research goals that they thought were worthy of the moniker of "Holy Grail", and asked key researchers in those fields to write essays about why they were important and the potential gains if the goal were realized. Although not everyone was impressed with their choices, I think their selections have stood the test of time fairly well. The selected topics were Manipulation of Matter at the Atomic and Molecular Levels, Room Temperature Superconductors, Unnatural Selection in Chemical Systems, Direct Observation of the Transition State, Controlling the Future of Matter, Artificial Photosynthesis, Biomimetic Chemistry and Artificial Enzymes, and Selective Intermolecular Carbon-Hydrogen Bond Activation.

In September 2020, Chemistry World, the member magazine of the Royal Society for Chemistry, took a look at these "Holy Grails" 25 years later. Most have seen significant progress and we seem to be on the cusp of realizing the original aspirations of a couple.

I think the LessWrong audience might enjoy both the original articles as well as the follow-ups because of the community's focus on intellectual progress and choosing important problems. Plus, there's just some incredibly cool science summarized in the follow-up pieces. For instance, did you know that some hydride materials are superconductive up to 260 K if you subject them to crazy high pressures? Or that you can use directed evolution to expand the kinds of chemistry enzymes can catalyze? Or that we're getting pretty good at functionalizing particular C-H bonds in complex organic molecules?

You can find the Chemistry World series starting from the splash page here. I would have come up with a somewhat different list of "most important topics in chemistry" than the original authors did, but that's no surprise in a field as big and varied as chemistry is. I think the Chemistry World team did a good job of creating accessible summaries of the chemistry community's ongoing quests for these "Holy...
“Unsupervised” translation as an (intent) alignment problem
by paulfchristiano — 2020-09-30T00:50:06.077Z

Suppose that we want to translate between English and an alien language (Klingon). We have plenty of Klingon text, and separately we have plenty of English text, but it’s not matched up and there are no bilingual speakers.

We train GPT on a mix of English and Klingon text and find that it becomes fluent in both. In some sense this model “knows” quite a lot about both Klingon and English, and so it should be able to read a sentence in one language, understand it, and then express the same idea in the other language. But it’s not clear how we could train a translation model.

Of course some concepts won’t have translations, and the model will often be uncertain about the translation of a term. But we can still ask for a model to explain the meaning of a Klingon expression as best as it can to an English-speaking user. For example, it could say “This is an idiomatic expression that’s often used to express great uncertainty” or “This is a small animal that is familiar to most Klingon speakers, I think it’s kind of like a frog but am not really sure” rather than translating a sentence directly.

How can we construct an objective that incentivizes the model to “try its best” at this translation task?

Translation-specific approaches

There are many published heuristics for unsupervised translation (e.g. Lample et al). I don’t think those techniques should completely satisfy us:

Existing methods can’t lead to a model that appropriately describes its uncertainty or talks the user through a hard-to-translate expression. (At least as far as I’m aware.)
We have no real reason to think existing methods fully utilize the model’s understanding, or to expect those methods to scale well. (In practice, I think they are impressive but still lag behind the quality of our models’ understanding.)
These heuristics are specific to translation, whereas we’d like to find general methods that can scale up to harder problems.

Existing alignment techniques

If we try to apply RL from human feedback to translation, we immediately run into a problem: how am I supposed to judge which of two English explanations of a Klingon sentence is better, given that I don’t know Klingon?

Debate doesn’t easily address this difficulty either — if one model claims that “qapla” means “great success” and the other claims it means “minor success,” I can’t easily decompose that disagreement into simpler sub-questions that ...
Examples of self-governance to reduce technology risk?
by Jia — 2020-09-29T19:31:41.405Z

Cross-posted from the EA Forum

I'm searching for examples of self-governance efforts to reduce technology risk. Do people have cases to suggest?

The more similar to AI development the better. That is, efforts by companies or academic communities to address risks that affect third parties, with minimal involvement from governments beyond basic law and order.

Examples from academia:

Leo Szilard and other physicists coordinating to prevent Germany from obtaining atomic bomb data, 1939-1940

Various efforts in biotechnology:

Asilomar conference on recombinant DNA, 1975

Mutations Database Initiative, 1999-2001

Synthetic biology conferences SB1.0 and SB2.0, 2006

Biology journals discussing publication restrictions, 2001-2011

Examples from the commercial sector:

DNA synthesis companies screening orders and buyers, 2004-2012

Efforts by the nanotechnology companies in the US, UK and Europe, 2004-2007
AGI safety from first principles: Goals and Agency
by Richard_Ngo — 2020-09-29T19:06:30.352Z

The fundamental concern motivating the second species argument is that AIs will gain too much power over humans, and then use that power in ways we don’t endorse. Why might they end up with that power? I’ll distinguish three possibilities:

AIs pursue power for the sake of achieving other goals; i.e. power is an instrumental goal for them.

AIs pursue power for its own sake; i.e. power is a final goal for them.

AIs gain power without aiming towards it; e.g. because humans gave it to them.

The first possibility has been the focus of most debate so far, and I’ll spend most of this section discussing it. The second hasn’t been explored in much depth, but in my opinion is still important; I’ll cover it briefly in this section and the next. Following Christiano, I’ll call agents which fall into either of these first two categories influence-seeking. The third possibility is largely outside the scope of this document, which focuses on dangers from the intentional behaviour of advanced AIs, although I’ll briefly touch on it here and in the last section.

The key idea behind the first possibility is Bostrom’s instrumental convergence thesis, which states that there are some instrumental goals whose attainment would increase the chances of an agent’s final goals being realised for a wide range of final goals and a wide range of situations. Examples of such instrumentally convergent goals include self-preservation, resource acquisition, technological development, and self-improvement, which are all useful for executing further large-scale plans. I think these examples provide a good characterisation of the type of power I’m talking about, which will serve in place of a more explicit definition.

However, the link from instrumentally convergent goals to dangerous influence-seeking is only applicable to agents which have final goals large-scale enough to benefit from these instrumental goals, and which identify and pursue those instrumental goals even when it leads to extreme outcomes (a set of traits which I’ll call goal-directed agency). It’s not yet clear that AGIs will be this type of agent, or have this type of goals. It seems very intuitive that they will because we all have experience of pursuing instrumentally convergent goals, for example by earning and saving money, and can imagine how much better we’d be at them if we were more intelligent. Yet since evolution ha...
Seek Upside Risk
by Neel Nanda — 2020-09-29T16:47:14.033Z
Link: https://www.neelnanda.io/blog/33-upside-risk

When I meet somebody new, there are two interesting things that could happen. We could get on super well, become great friends, and be part of each other’s lives for year to come. Or we don’t. Maybe we like each other, maybe we don’t, but we likely just go our separate ways. But the upside in the first case massively outweighs the downside in the second case. Yet, we all spend far less time meeting new people than we could. Why?

Upside Risk

I call this general phenomena of a small chance of a massive win, vs a reasonable chance of a tiny loss, upside risk. And when I think about some of the really awesome things in my life, a lot of it came from sources of upside risk:

I have a lot of amazing friends, most of whom I went out of my way to meet and keep in touch with
Hobbies! I tried blogging, it might have been a bit of a waste of time, but turned out to be fucking awesome. I could have lost a bit of time, but I now have a great new creative outlet!
Asking for advice and favours - I’m going to be spending the next 3 months doing a research project, and this would never have happened if I hadn’t been asking loads of people for advice and suggestions!

In general, I think there are a lot of sources of upside risk in life, and that they’re extremely positive in expectation. And if you do enough of them, this pays out well. So I think it’s extremely valuable to actively seek out upside risk, and to shape your life to welcome it.

There’s even a fairly solid mathematical justification for this! This is known as the explore/exploit problem in reinforcement learning - do I exploit and do what I currently think is the best idea, or do I explore and try something new, something that might give me information that I can use to inform future actions. The problem is essentially pricing the value of information.

And a pretty decent (and easy-to-compute) strategy is to take the Upper Confidence Bound. We have uncertainty over each strategy, and pick the strategy with the highest 95% confidence bound. Essentially, it pays to be an optimist, to always take the action with the best chance of being awesome.

This post is going to be about why we don’t seek upside risk as much as we could, and how we can change this.

Becoming emotionally OK with it

So, this is a pretty simple argument. Yet, I do this far less than I could. Why...
Doing discourse better: Stuff I wish I knew
by dynomight — 2020-09-29T14:34:55.913Z
Link: https://dyno-might.github.io/2020/09/29/doing-discourse-better-stuff-i-wish-i-knew/

Nick and Maria want to talk about sugar. Does it reduce lifespan? They disagree, but are honest and principled. They want to find the truth and agree on it.
David Friedman on Legal Systems Very Different from Ours: SlateStarCodex Online Meetup
by JoshuaFox — 2020-09-29T11:18:07.901Z

David Friedman on Legal Systems Very Different from Ours: A brief survey of a range of legal system, past and present, from Imperial China and Periclean Athens to modern Amish and Romany.

The event will be Oct 11, 2020, at 17:30 UTC, 20:30 Israel Daylight Time, 10:30 Pacific Daylight Time.

Sign up here, up to an hour before the event, and we'll send you an invitation to the online meetup

David Friedman is an academic economist with a doctorate in physics recently retired from spending the previous twenty-three years teaching in a law school. His first book, The Machinery of Freedom: Guide to a Radical Capitalism, was published in 1973 and includes a description of how a society with property rights and without government might function. There as elsewhere, he offers a consequentialist defense of libertarianism.

His most recent non-fiction book is Legal Systems Very Different from Ours, covering systems from Periclean Athens through modern Amish and Romany. He is also the author of three novels, one commercially published and two self-published, and, with his wife, a self-published medieval and renaissance cookbook and a larger self-published book related to their hobby of historical recreation.

Much of his published work, including journal articles, essays, drafts of forthcoming work and the full text of several books, can be read on his web page: daviddfriedman.com
AGI safety from first principles: Superintelligence
by Richard_Ngo — 2020-09-28T19:53:40.888Z

In order to understand superintelligence, we should first characterise what we mean by intelligence. We can start with Legg’s well-known definition, which identifies intelligence as the ability to do well on a broad range of cognitive tasks.[1] The key distinction I’ll draw in this section is between agents that understand how to do well at many tasks because they have been specifically optimised for each task (which I’ll call the task-based approach to AI), versus agents which can understand new tasks with little or no task-specific training, by generalising from previous experience (the generalisation-based approach).

Narrow and general intelligence

The task-based approach is analogous to how humans harnessed electricity: while electricity is a powerful and general technology, we still need to design specific ways to apply it to each task. Similarly, computers are powerful and flexible tools - but even though they can process arbitrarily many different inputs, detailed instructions for how to do that processing needs to be individually written to build each piece of software. Meanwhile our current reinforcement learning algorithms, although powerful, produce agents that are only able to perform well on specific tasks at which they have a lot of experience - Starcraft, DOTA, Go, and so on. In Reframing Superintelligence, Drexler argues that our current task-based approach will scale up to allow superhuman performance on a range of complex tasks (although I’m skeptical of this claim).

An example of the generalisation-based approach can be found in large language models like GPT-2 and GPT-3. GPT-2 was first trained on the task of predicting the next word in a corpus, and then achieved state of the art results on many other language tasks, without any task-specific fine-tuning! This was a clear change from previous approaches to natural language processing, which only scored well when trained to do specific tasks on specific datasets. Its successor, GPT-3, has displayed a range of even more impressive behaviour. I think this provides a good example of how an AI could develop cognitive skills (in this case, an understanding of the syntax and semantics of language) which generalise to a range of novel tasks. The field of meta-learning aims towards a similar goal.

We can also see the potential of the generalisation-based approach by looking at how humans developed....
AGI safety from first principles: Introduction
by Richard_Ngo — 2020-09-28T19:53:22.849Z

This is the first part of a six-part report called AGI safety from first principles, in which I've attempted to put together the most complete and compelling case I can for why the development of AGI might pose an existential threat. The report stems from my dissatisfaction with existing arguments about the potential risks from AGI. Early work tends to be less relevant in the context of modern machine learning; more recent work is scattered and brief. I originally intended to just summarise other people's arguments, but as this report has grown, it's become more representative of my own views and less representative of anyone else's. So while it covers the standard ideas, I also think that it provides a new perspective on how to think about AGI - one which doesn't take any previous claims for granted, but attempts to work them out from first principles.

Having said that, the breadth of the topic I'm attempting to cover means that I've included many arguments which are only hastily sketched out, and undoubtedly a number of mistakes. I hope to continue polishing this report, and I welcome feedback and help in doing so. I'm also grateful to many people who have given feedback and encouragement so far. I plan to cross-post some of the most useful comments I've received to the Alignment Forum once I've had a chance to ask permission. I've posted the report itself in six sections; the first and last are shorter framing sections, while the middle four correspond to the four premises of the argument laid out below.

AGI safety from first principles

The key concern motivating technical AGI safety research is that we might build autonomous artificially intelligent agents which are much more intelligent than humans, and which pursue goals that conflict with our own. Human intelligence allows us to coordinate complex societies and deploy advanced technology, and thereby control the world to a greater extent than any other species. But AIs will eventually become more capable than us at the types of tasks by which we maintain and exert that control. If they don’t want to obey us, then humanity might become only Earth's second most powerful "species", and lose the ability to create a valuable and worthwhile future.

I’ll call this the “second species” argument; I think it’s a plausible argument which we should take very seriously.[1] However, the version stated above relies on...
is scope insensitivity really a brain error?
by Kaarlo Tuomi — 2020-09-28T18:37:37.715Z

I am reading a post called The Martial Art of Rationality by Eliezer Yudkowsky, in which he makes the following claim:

"If you’re a fast learner, you might learn faster—but the art of rationality isn’t about that; it’s about training brain machinery we all have in common. And where there are systematic errors human brains tend to make—like an insensitivity to scope—rationality is about fixing those mistakes, or finding work-arounds."

this post follows one in which he explained the concept of scope insensitivity by discussing a study that found that persons asked to contribute to dealing with the consequences of fuel spill on wildlife habitat did not contribute more money to save more birds. the contributors were deemed to be insensitive to the scope of the suffering.

the point I want to discuss is whether it is entirely fair to describe scope insensitivity, as defined in this way, as a "systematic human brain error"?

it seems to me that this is bordering on saying that persons who made a different choice to yours are therefore not just wrong, but suffering from something, their brain is not working properly and they need to be taught how to make better choices. where "better" obviously means, more in line with the choice you would make.

scope insensitivity would only be irrational if saving birds were the only criteria in play. to save more birds, give more money. but this is almost never the case, people are more complex than this and they need to consider more criteria than this and each person may consider different criteria and weight them differently. to label those differences as "systematic human brain error" seems to be a very one-dimensional response.

I think we need to bear in mind that the original study did not allow for the possibility that folk did not pay more because they were unable to afford more, or because they would prefer to allocate their charitable spending to alleviate human suffering rather than animal suffering. in fact, the study explicitly said that they were unable to account for the lack of sensitivity to scope. it seems wrong, in fact plain anti-intellectual, for Yudkowsky to claim that their scope insensitivity is a "systematic human brain error"?

please discuss.