trentmkelly/LessWrong-43k huggingface dataset:
Novelty Generation - The Art of Good Ideas
by Shoshannah Tekofsky — 2022-08-20T00:36:06.479Z

Recently I set out to understand how new ideas are formed in the mind. My process consisted of poking at my own mind, poking at other people's minds, and then organizing group sessions of poking at each other's minds.

The Scientific Process™

The following is a list of insights from introspection and outrospection (a word I made up to refer to probing other people on their introspection), my guesses on how this relates to scientific progress, especially in a pre-paradigmatic field, and an addendum on the experimental exercises we tried. Once I told people about my ideas, I discovered my insights overlap heavily with the Babble & Prune sequence as well the Bus Ticket Theory of Genius. I suspect there might be more written about this, but here's the specific, shiny new wheel I reinvented!

A Model of Good Ideas

But first, what's the target?

My exploration started with attempts to generate the most novel ideas. How are they formed? How can we generate them faster, more efficiently, just ... more-newer? How do we even generate ideas that are farther off the norm?

In response, someone suggested using LSD to maximize novelty generation. That made me pause and reflect. "That's not the thing I mean", I realized. LSD creates a huge perturbation in your experiences and idea formation, but most of it is ... not useful? Which made me realize, most novelty is noise. Random recombinations are very new and (mostly) very useless. There must be a second factor. That second factor is coherence.

If you consider Novelty on the Y-axis and Coherence on the X-axis, you have a 2-dimensional space of idea quality. AI alignment research needs new ideas... but like ... good, new ideas.

A useful analogy is aliens. People who think of aliens and generate high novelty and low coherence ideas are conspiracy theorists and UFO catchers. People who are high on coherence but low on novelty dismiss the concept of aliens. But people who are high on both coherence and novelty might generate something like the Fermi paradox. Of course, once an idea has been generated, and it's a good idea, it will be adopted by all high coherence people, no matter their skills at novelty generation. But creating the theory in the first place takes both.

This explains the uniqueness of "geniuses". Einstein or Da Vinci were not the smartest people, but they were extremely smart (coheren...
How to do theoretical research, a personal perspective
by Mark Xu — 2022-08-19T19:41:21.562Z

"Where do new [algorithms] come from? I keep reading about someone who invented [an algorithm] to do something-or-other but there's no mention of how."

A shrug of robed shoulders. "Where do new books come from, Mr. Potter? Those who read many books sometimes become able to write them in turn. How? No one knows."

"There are books on how to write -"

"Reading them will not make you a famous playwright. After all such advice is accounted for, what remains is mystery. The invention of new [algorithms] is a similar mystery of purer form."

- Harry Potter and the Methods of Rationality

Most of the content in this document came out of extensive conversations with Paul Christiano.

(This document describes one way of thinking about how to do one particular type of research. There are other ways to productively do this kind of research, and other productive kinds of research. Consider this document peppered with phrases like “from my perspective”, “I think”, “my sense is”, etc.)

A lot of people have a vague mental picture of what empirical research looks like, which consists of exploring data, articulating hypotheses about the data, and running experiments that potentially falsify the hypothesis. I think people lack a similarly mechanistic picture of what theoretical research looks like, which results in them not knowing how to do theory, being skeptical of the possibility of theoretical progress, etc. I do think the difficulty of getting high-quality real-world feedback makes theoretical research more difficult than empirical research, but I think it’s possible to get enough real-world feedback when doing theory that you can still expect to make steady progress.

Currently, I think many people think of theory as “someone sits in a room and has a brilliant insight to solve the problem.” Instead, I think a more accurate picture is very similar to the picture one has for empirical research: the theorist explores some data, gradually builds an intuitive sense of what’s going on, articulates hypotheses that capture their intuition, and falsifies their hypotheses by testing them against data, all the while iteratively building up their understanding.

The key difference between the empirical researcher and the theoretical researcher is while the empirical researcher can build intuition and falsify hypothesis by considering real-world data, the theorist, although ultimately be ground...
Can You Upload Your Mind & Live Forever? From Kurzgesagt - In a Nutshell
by Noosphere89 — 2022-08-19T19:32:12.434Z
Link: https://www.youtube.com/watch?v=4b33NTAuF5E

An interesting video on mind uploading from almost 2 years ago, focusing on explaining the concept to the wider world. It's fairly simplified, as expected of a video that has to cross large inferential distances, and the Kurzgesagt channel is basically a simplified (though still mostly accurate) channel for communicating complicated topics and simplifying them.

In a sense, this would be a post that fits in right with the old LW 1.0 culture years ago with a transhumanist and mind uploading culture, before AI safety was the new goal of LW.
Guided Consumption Theory: A Virtuous Dance between Altruistic Agents, Economic Discriminators, and Opportunistic Helpers
by Brad West  — 2022-08-19T17:49:40.990Z

Reposting from the EA Forum

Previous posts have set forth the concept of Guided Consumption and provided research supporting the viability of businesses with charities in the equity position (“Guiding Producers”). In this post, I evaluate the agents involved in Guided Consumption, their motives, and why creating the infrastructure enabling Guided Consumption is an extremely high value project.

TLDR: Creating the infrastructure and public awareness enabling Guided Consumption – consumers choosing to direct their purchases through charity-owned businesses- is extremely high value because it accords with the motivations and cost-tolerances of the sets of agents involved. The general public, motivated to do good in low, no, or negative cost situations, is empowered to do good by shifting its purchases: economic discrimination. Other economic actors interacting with Guiding Producers are motivated -either by their own good will or good publicity- to treat Guiding Producers more favorably than normal companies. Altruistic Agents, motivated to maximize impact attained by their resource expenditure, are empowered to do so by creating the infrastructure and public awareness of Guided Consumption. This project is tractable due to the malleability of public awareness and the discoverability of the best contexts for Guiding Producers.

The Main Two Agent Groups (and a Third Minor, but Significant Group)

Altruistic Agents as Builders of the Infrastructure (“AAs”): This set of agents consist of individuals and organizations with a primary motivation of maximizing positive impact. AAs are fine with bearing heavy costs, if those costs correspond with higher utility. Examples of AAs would be philanthropists or those who forgo career paths with higher salaries in favor of career paths that enable the highest expected impact (80,000 Hours is an organization advising individuals on how to plan for a career with the highest expected impact for one’s capabilities). Sometimes an AA is domain specific. Many within the Effective Altruism community consider themselves generally AAs. AAs would play the function of providing resources to create the infrastructure and further the Guided Consumption Movement, if they can be persuaded that this is the best use of their resources.

Ordinary Members of the Public as Economic Discriminators (“EDs"): This is an...
Epistemic Artefacts of (conceptual) AI alignment research
by Nora_Ammann — 2022-08-19T17:18:47.941Z

The fact that this post is seeing the light of day now rather than in some undefined number of weeks is, not in small part, due to participating in the second Refine blog post day. Thank you, fake-but-useful containers, and thank you, Adam.

Tl;dr

In this post, I describe four types of insights - what I will call Epistemic Artefacts - that we may hope to acquire through (conceptual) AI alignment research. I provide examples and briefly discuss how they relate to each other and what role they play on the path to solving the AI alignment problem. The hope is to add some useful vocabulary and reflective clarity when thinking about what it may look like to contribute to solving AI alignment.

Four Types of Epistemic Artefacts

Insofar as we expect conceptual AI alignment research to be helpful, what sorts of insights (here: “epistemic artefacts”) do we hope to gain?

In short, I suggest the following taxonomy of potential epistemic artefacts:

Map-making (de-confusion, gears-level models, etc.)
Characterising risk scenarios
Characterising target behaviour
Developing alignment proposals

(1) Map-making (i.e. conceptual de-confusion, gears-level understanding of relevant phenomena, etc.)

First, research can aim to develop a

gears-level understanding

of phenomena that appear critical for properly understanding the problem as well as for formulating solutions to AI alignment (e.g. intelligence, agency, values/preferences/intents, self-awareness, power-seeking, etc.). Turns out, it’s hard to think clearly about AI alignment without having a good understanding of and “good vocabulary” for phenomena that lie at the heart of the problem. In other words, the goal of "map-making" is to dissolve conceptual bottlenecks holding back progress in AI alignment research at a the moment.

Figuratively speaking, this is where we are trying to draw more accurate maps that help us better navigate the territory.

Some examples of work on this type of epistemic artefact include

Agency: What it is and why it matters

,

Embedded Agency

,

What is bounded rationality?

,

The ground of optimization

,

Game Theory

,

Mathematical Theory of Communication

,

Functional Decision Theory

and

Infra-Bayesianism

—among many others.

(2) Identifying and specifying risk scenarios

We can further seek to identify (new) civilizational risk scenarios brou...
Introducing the Existential Risks Introductory Course (ERIC)
by CallumMcDougall — 2022-08-19T15:54:34.641Z

Crossposted to the EA Forum.

Introduction

We (the Cambridge Existential Risks Initiative) ran an Existential Risks Introductory Course (ERIC) in the first quarter of 2022, aiming to introduce the field of existential risks, without being explicitly associated with any particular philosophy. We expect the programme to be most useful to people who are new to this field, and we hypothesised that we may be able to reach a different target audience by not explicitly branding it as EA.

The full curriculum we used for the programme, along with exercises and organisation spotlights, can be found

here

. This was primarily designed by Callum McDougall, with some inputs from the rest of the CERI team.

If you are interested in joining the next iteration of the course in Winter 2022 (either as a participant or as a facilitator),

please fill out this interest form

.

This post contains an overview of the course, which is followed by an abbreviated version of the syllabus for the ease of gathering feedback. The weekly summaries may also be helpful for community builders looking for summaries of any of the core readings from our syllabus.

We welcome any feedback on the content, exercises or anything else pertaining to the course, either here publicly on the Forum, or you can also reach out to us privately if you prefer that.

Course overview

The course consists of 8 weeks of reading (split into core and applied). Some weeks also include exercises, which participants are encouraged to complete and discuss in the session. Each week, participants will meet for 1.5 hour sessions where they will discuss the material and exercises with a facilitator.

The topics for each week are as follows:

Week 1: Introduction to Existential Risks
Provides an introduction to x-risks, why they might be both highly important and neglected, and introduces some important terminology.

Week 2: Natural & Anthropogenic Risks
Discusses natural risks, and risks from nuclear war and climate change.

Week 3: Biosecurity, And How To Think About Future Risks
Discusses risks from engineered pandemics, as well as a broader look at future risks in general and how we can reason about them and prepare for them.

Week 4: Unaligned Artificial Intelligence
Discusses risks from unaligned AI, and provides a brief overview of the different approaches that are being taken to try and

...
Statistics for vague concepts and "Colors" of places
by Q Home — 2022-08-19T10:33:53.540Z

I'm posting this prematurely because in today's world you just don't know what happens tomorrow. Or today.

Could you help me to formulate statistics with the properties I'm going to describe?

I want to share my way of seeing the world, analyzing information and experiencing other people. (But it's easier to talk about fantastical places, so I'm going to give examples with fantastical places.)

I think my ideas could help to formalize (a little bit, not entirely) thinking about vague concepts and the process of generating hypotheses. It's something Bayesianism doesn't do (it tries to deal with atomic outcomes), if I understand correctly. By "vague concepts" I mean concepts that have specific enough meaning in context, but no definite meaning outside of a context. For example, the concept of a "game" is easy to understand in a specific context, but potentially outright impossible to understand/define outside of any context. The same with the concept of "being healthy" (can you list all conditions of "being healthy" outside of any context?). The same with human values, such as "freedom". The same with most of the words in the human language. All those context-dependent objects create clusters of things with family resemblances.

I think my ideas could help to describe how vague concepts obtain specific meaning in context. And how new contexts are created. I feel that it's related to hypotheses generation because some general (scientific) ideas/paradigms don't have any meaning outside of context (maybe Thomas S. Kuhn wrote about this in The Structure of Scientific Revolutions). But when they compete with each other in the real world, they obtain specific meaning.

I think it all relates even to human personalities ("personality" is a vague concept too): if you took all your behaviors in the course of your life, would there be a single thread that connects them all and differentiates you from most other people? And if "no", does it mean that you're just a combination of random behaviors? Your entire personality is a "context-dependent object", it makes sense only in a specific context, contrasted against something else.

If you could formally analyze (at least to some degree) concepts that are connected only by loose connections of "family resemblance" or "sorites paradox" types of connections, it would be an extremely powerful thinking tool, and a needed one. You could an...
Are language models close to the superhuman level in philosophy?
by Roman Leventov — 2022-08-19T04:43:07.504Z

An alternative way to ask the question in the title of this post: what capacities do philosophers use that language models lack?

Reading OpenAI’s “Self-critiquing models for assisting human evaluators” and Google’s “Language Model Cascades” has led me to the idea that most philosophical methods can be programmed via combining language models in a certain cascade, fine-tuning them on existing philosophical literature, and applying the right kind of reinforcement pressures to the system. And I seriously wonder if using the current SoTA language models (e. g., PaLM) in this way would already produce superhuman-level philosophical writing, i. e. philosophical writing that human philosophers would have a harder time critiquing than any existing philosophy done by humans, on a given topic.

Good human philosophers should know a lot of existing arguments on various questions. Language models obviously have an edge over humans in this: they can easily “read” all the philosophical writing existing in the world.

Philosophers use creative arguments, examples, and thought experiments to test and compare their theories. Perhaps, AI philosophers can search for such arguments or examples using random sampling, then self-criticising an argument or self-assessing the quality of an example for a particular purpose, and then modifying or refining the argument or the example. Perhaps, even with the current SoTA models, this search will be closer to brute force than the intuitive inquiry of a human philosopher, and can potentially take a long time, since the models probably can’t yet understand the structure of philosophical theories well (if I don’t underestimate them!).

The only philosophical theories and arguments that still seem firmly out of reach of language models are those based on scientific theories, e. g. some theories in the philosophy of language or the philosophy of mind and consciousness based on certain theories in neuroscience or psychology or anthropology, philosophy of consciousness, probability, agency based on certain theories in physics, etc. However, philosophical writings of this kind should probably be treated as interpretations of the respective scientific theories rather than philosophical theories in their own right. In other words, these topics are actually the areas where sciences chip away from philosophy, as it has gradually happened with all the exist...
What does moral progress consist of?
by jasoncrawford — 2022-08-19T00:22:11.756Z
Link: https://forum.effectivealtruism.org/posts/EBjYudmBqJxdDmcbM/what-does-moral-progress-consist-of

There’s something that strikes me as odd about the way I hear “moral progress” discussed in the EA/longtermist community in general and in What We Owe the Future in particular.

The issues that are generally discussed under this heading are things like: animal welfare, existential risk, “coordination” mechanisms, and so forth.

However, when I look out at the world, I see a lot of problems rarely discussed among EAs. Most of the world is still deeply religious; in some communities and even whole nations, this pervades the culture and can even create oppression. Much of the world still lives under authoritarian or even totalitarian governments. Much of the world lives in countries that lack the institutional capacity to support even rudimentary economic development, leaving their people without electricity, clean water, etc.

And the conversation around the world’s problems leaves even more to be desired. A lot of discussion of environmental issues is lacking in scientific or technical understanding, treating nature like a god and industry like a sin—there’s even now a literal “degrowth” movement. Many young people in the US are now attracted to socialism, an ideology that should have been left behind in the 20th century. Others support Trump, an authoritarian demagogue with no qualifications to leadership. Both sides seem eager to tear down the institutions of liberalism, a “burn it all down” mentality. And all around me I see people more concerned with tribal affiliations than with truth-seeking.

So when I think about what moral progress the world needs, I mostly think it needs a defense of Enlightenment ideas such as reason and liberalism, so that these ideas can become the foundation for addressing the real problems and threats we face.

I think this might be highly relevant even to someone solely concerned with existential risk. For instance, if we want to make sure that an extinction-level weapon doesn’t get in the hands of a radical terrorist group, it would be good if there were fewer fundamentalist ideologies in the world, and no nation-states that sponsor them. More prosaically, if we want to have a good response to pandemics, it would be good to have competent leadership instead of the opposite (my understanding is that the US covid response could have been much better if we had just followed the pre-existing pandemic response plan). If we want to make sure...
Alignment's phlogiston 
by Eleni Angelou — 2022-08-18T22:27:31.093Z

Crossposted from the EA Forum: https://forum.effectivealtruism.org/posts/DCtqgsywCRakLvHn6/alignment-s-phlogiston

Epistemic status: quick and dirty reaction to the claim that alignment research is like archaeology. I wouldn't trust anyone suggesting that scientific and technological revolutions can be simply broken down into a series of discoveries. But I like the phlogiston metaphor, nevertheless.

In this post, John Wentworth makes the case that alignment research is at a pre-paradigmatic stage. As he states, experts in the field share a "fundamental confusion" and there is no explicit consensus on the nature of the subject matter and the best ways to approach it. What is also characteristic of an immature science is the disunity [1]of frameworks that concerns the concepts, theories, agendas, practices, methodological tools, and other criteria for what qualifies as having high explanatory force. Such disunity seems to describe the current state of AI safety.

In this post, Adam Shimi compares alignment/AI safety to historical sciences such as archaeology. This can only mean that either alignment is not at a pre-paradigmatic state or that archaeology is not a mature science. However, archaeology doesn't suffer from the "fundamental confusion" of alignment; it might not be possible to employ the same observational tools researchers do in physics or chemistry, but archaeologists do have a shared view of how to study their subject. I very much doubt that the average archaeologist would go ahead and tell you that they're fundamentally confused about their field and how they approach the most important questions of their research agenda.

Looking back at the history of science, the field of AI safety seems to have more similarities with alchemy. The alchemists were people deeply confused about their methods and how likely they are to succeed. They all, however, shared a common aim summarized in this threefold: to find the Stone of Knowledge (The Philosophers' Stone), to discover the medium of Eternal Youth and Health, and to discover the transmutation of metals. Their "science" had the shape of a pre-paradigmatic field that would eventually transform into the natural science of chemistry. Their agenda ceased to be grounded upon mystical investigations as the science began to mature.

The claim here is not that alignment has in an...