trentmkelly/LessWrong-43k huggingface dataset:
Is there some reason LLMs haven't seen broader use?
by tailcalled — 2022-11-16T20:04:48.473Z

When GPT-3 first came out, I expected that people would use it as a sort of "common-sense reasoning module". That is, if you want to process or generate information in some way, then you can give GPT-3 a relevant prompt, and repeatedly apply it to a bunch of different inputs to generate corresponding outputs. After GPT-3 came out, I had expected that people would end up constructing a whole bunch of such modules and wire them together to create big advanced reasoning machines. However, this doesn't seem to have panned out; you don't see much discussion about building LLM-based apps.

Why not? I assume that there must be something that goes wrong along the way, but what exactly goes wrong? Seems like it has the potential to teach us a lot about LLMs.
When should we be surprised that an invention took “so long”?
by jasoncrawford — 2022-11-16T20:04:47.295Z
Link: https://rootsofprogress.org/when-to-be-surprised-an-invention-took-so-long

My first highly popular essay was “Why did we wait so long for the bicycle?” I’ve asked the same question of the cotton gin and the threshing machine. Others have asked it of the steam engine and the wheel. Recently Brian Potter asked it about wind power and Anton Howes about semaphore signaling systems. See more examples here.

When asking these questions, we should think about when the question even needs an answer. That is, “why did it take so long” is only interesting if it took an abnormally long amount of time.

Here’s my model for this.

First, an invention is not going to happen at all if (1) it’s not technically possible or (2) there’s no market for it.

Gate (1), technical possibility, could include, for example:

Scientific foundations. No light bulb before electromagnetism, no antibiotics before the germ theory.
Components. Airplanes were not possible before the internal combustion engine.
Materials. Skyscrapers could only be built once cheap steel girders were available.
Manufacturing techniques. Precision machining was necessary to make the gears, sprockets, chains, bearings, and other parts for a wide variety of inventions, probably including the threshing machine and the bicycle.

Gate (2), the market, is whether it can be done commercially at a price that anyone will pay. If someone does make an invention there is no market for, it doesn’t go anywhere, and we might not even hear about it, because it is unlikely to make the history books. In any case, it wouldn’t affect the world, because it wouldn’t get distribution, and so it wouldn’t be historically relevant for our purposes. You see examples of this from time to time, such as the Korean movable-type printing press that predated Gutenberg.

Note that the bar inventions have to meet is not just a proof of concept: they have to be sufficiently powerful, efficient, and reliable to be of practical use. Early computing machines were too slow; early threshing machines broke down too frequently; early light bulbs burned out too quickly. These are technically interesting prototypes, but not true inventions. An invention does not merely demonstrate a concept, it solves a problem—the whole problem, not just a part of it, even if it is the most visible or obvious part. An invention has to be practical. (See more discussion of this here.)

Once something is technically possible and economically viable, then the clock st...
Questions about Value Lock-in, Paternalism, and Empowerment
by Sam F. Brown — 2022-11-16T15:33:46.715Z
Link: https://sambrown.eu/writing/paternalism-questions

Pulling a child out of the path of a fast car is the right thing to do, whether or not the child agrees, understands, is grateful, or even is hurt during the rescue. Paternalistic acts like this, when we might argue that a person's straightforward consent ought to be overridden, add uncomfortable complications to liberal ethical frameworks.

As artificial intelligence continues to improve, there are an increasing number of domains in which its judgement is superior to that of humans. At what point is the relationship between AI and humans similar to that between an adult and a child? And to what extent should we demand to understand, agree or consent to decisions made by AI?

I see Paternalism and intentional Value Lock-in as intimately related. Considering that we may, by influencing the creation of AI, have a significant impact on the freedoms of future people: how should we responsibly discharge this power?

In writing this essay I’m hoping to reach people who’d like to discuss these things in more depth, and people who might want to work together.

I'd be very grateful for any criticism, suggestions, pointers, or discussion :)

Structure of this writing

I start with brief overviews of what I mean by ‘Value Lock-in’ and ‘Paternalism’, and the connection I see between them.

Then I discuss a core intuition behind my recent research into a plausible minimally-paternalistic approach to alignment: the idea of ‘un-asking’ human values. I include a brief discussion of technical approaches, which I plan to expand on more fully in later writing.

The core of this essay is a list of questions for which I currently have no answer:

Metaethics: Is all moral change contingent?
Paternalism: How much epistemic humility is appropriate or desired? What is the minimal amount required as a condition of logical consistency?
Empowerment: How does choice maximisation behave in the context of Social Choice Theory?
Practicality/deployment: Is the alignment tax of something like “maximise Empowerment-of-Others” too high?
Empowerment: Is there a way to generalise choice-maximisation approaches beyond toy models?
Hybridisation: Is there a way to balance goal-inference and choice-maximisation in a way that is robust to both reward-hacking and value lock-in?
Paternalism: Is there a way to generalise debate between LLM agents to comprehensively and meaningfully explore non-binary/open questions?

Finally, I ...
If Professional Investors Missed This...
by jefftk — 2022-11-16T15:00:01.093Z

One of the largest cryptocurrency exchanges,

FTX, recently
imploded after

apparently
transferring customer funds to cover losses at their affiliated hedge
fund. Matt Levine has

good
coverage, especially his recent post on

their
balance sheet. Normally a crypto exchange going bust isn't
something I'd pay that much attention to, aside from sympathy for its
customers, but its

Future
Fund was one of the largest funders in effective altruism (EA).

One reaction I've seen in several places, mostly outside EA, is
something like, "this was obviously a fraud from the start, look at
all the red flags, how could EAs have been so credulous?" I think
this is mostly wrong: the red flags they cite (size of FTX's claimed
profits, located in the Bahamas, involved in crypto, relatively young
founders, etc.) are not actually strong indicators here. Cause for
scrutiny, sure, but short of anything obviously wrong.

The opposite reaction, which I've also seen in several places, mostly
within EA, is more like, "how could we have caught this when serious
insitutional investors with hundreds of millions of dollars on the
line missed it?" FTX had raised about
$2B in external funding, including ~$200M from Sequoia, ~$100M from SoftBank, and ~$100M from the Ontario Teacher's Pension Plan. I
think this argument does have some truth in it: this is part of why
I'm ok dismissing the "obvious fraud" view of the previous paragraph.
But I also think this lets EA off too easily.

The issue is, we had a lot more on the line than their investors did.

Their worst case was that their investments would go to zero and they
would have mild public embarrassment at having funded something that
turned out so poorly. A strategy of making a lot of risky bets can do
well, especially if spending more time investigating each opportunity
trades off against making more investments or means that they
sometimes lose the best opportunities to competitor funds. Half of
their investments could fail and they could still come out ahead if the other
half did well enough. Sequoia wrote
after, "We are in the business of taking risk. Some investments
will surprise to the upside, and some will surprise to the downside."

This was not our situation:

The money FTX planned to donate represented a far greater portion
of the EA "portfolio" than FTX did for these institutional investors,
T

...
Disagreement with bio anchors that lead to shorter timelines
by Marius Hobbhahn — 2022-11-16T14:40:16.734Z

This would have been a submission to the

FTX AI worldview prize

. I’d like to thank Daniel Kokotajlo, Ege Erdil, Tamay Besiroglu, Jaime Sevilla, Anson Ho, Keith Wynroe, Pablo Villalobos and Simon Grimm for feedback and discussions. Criticism and feedback are welcome. This post represents my personal views.

Update Dec 2023: I now think timelines faster than my aggressive prediction in this post are accurate. My median is now 2030 or earlier.

The causal story for this post was: I first collected my disagreements with the bio anchors report and adapted the model. This then led to shorter timelines. I did NOT only collect disagreements that lead to shorter timelines. If my disagreements would have led to longer timelines, this post would argue for longer timelines.

I think the bio anchors report (the one from 2020, not Ajeya’s personal updates) puts too little weight on short timelines. I also think that there are a lot of plausible arguments for short timelines that are not well-documented or at least not part of a public model. The bio anchors approach is obviously only one possible way to think about timelines but it is currently the canonical model that many people refer to. I, therefore, think of the following post as “if bio anchors influence your timelines, then you should really consider these arguments and, as a consequence, put more weight on short timelines if you agree with them”. I think there are important considerations that are hard to model with bio anchors and therefore also added my personal timelines in the table below for reference.

My best guess bio anchors adaption suggests a median estimate for the availability of compute to train TAI of 2036 (10th percentile: 2025, 75th percentile: 2052). Note that this is not the same as predicting the widespread deployment of AI. Furthermore, I think that the time “when AI has the potential to be dangerous” is earlier than my estimate of TAI because I think that this poses a lower requirement than the potential to be economically transformative (so even though the median estimate for TAI is 2036, I wouldn’t be that surprised if, let’s say 2033 AIs, could deal some severe societal harm, e.g. > $100B in economic damage).

You can find all material related to this piece including the colab notebook, the spreadsheets and the long version in

this google folder

.

Executive summary

I ...
Current themes in mechanistic interpretability research
by Lee Sharkey — 2022-11-16T14:14:02.030Z

This post gives an overview of discussions - from the perspective and understanding of the interpretability team at Conjecture - between mechanistic interpretability researchers from various organizations including Conjecture, Anthropic, Redwood Research, OpenAI, and DeepMind as well as some independent researchers. It is not a review of past work, nor a research agenda. We're thankful for comments and contributions from Neel Nanda, Tristan Hume, Chris Olah, Ryan Greenblatt, William Saunders, and other anonymous contributors to this post, which greatly improved its quality. While the post is a summary of discussions with many researchers and received comments and contributions from several, it may nevertheless not accurately represent their views.

The last two to three years have seen a surge in interest in mechanistic interpretability as a potential path to AGI safety. Now there are no fewer than five organizations working on the topic (Anthropic, Conjecture, DeepMind, OpenAI, Redwood Research) in addition to numerous academic and independent researchers.

In discussions about mechanistic interpretability between a subset of researchers, several themes emerged. By summarizing these themes here, we hope to facilitate research in the field more broadly.

We identify groups of themes that concern:

Object-level research topics in mechanistic interpretability
Research practices and tools in mechanistic interpretability
Field building and research coordination in mechanistic interpretability
Theories of impact for mechanistic interpretability

Object-level research topics in mechanistic interpretability

Solving superposition

Anthropic’s recent

article

on Toy Model of Superposition laid out a compelling case that superposition is a real phenomenon in neural networks. Superposition appears to be one of the reasons that polysemanticity happens, which makes mechanistic interpretability very difficult because it prevents us from telling simple stories about how features in one layer are constructed from features in previous layers.

A solution to superposition will look like the ability to enumerate all the features that a network represents, even if they’re represented in superposition. If we can do that, then we should be able to make statements like “For all features in the neural network, none violate rule X” (and more ambitiously, for&nbs...
Unpacking "Shard Theory" as Hunch, Question, Theory, and Insight
by Jacy Reese Anthis — 2022-11-16T13:54:15.850Z

I read several shard theory posts and found the details interesting, but I couldn't quite see the big picture. I'm used to hearing "theory" refer to a falsifiable generalization of data. It is typically stated as a sentence, paragraph, list, mathematical expression, or diagram early in a scientific paper. Theories range from extremely precise like Newton's three laws of motion to extremely broad like Frankish's consciousness illusionism (i.e., consciousness is an illusion).

I have also been generally confused about what it means to "solve alignment" before AGI arrives given that there is not (yet) consensus around any pre-AGI[1] formalization of the alignment problem itself: Wouldn't any proposed solution still have a significant number of people (say, >10% of Alignment Forum users) who think that it doesn't even pose the problem the right way? What should our "theories" even be aiming for?

With help from Nathan Helm-Burger, I think I now better understand what's referred to as shard theory and want to share my understanding as an exercise in problem and solution formulation in alignment. I think "shard theory" refers to four sequential components: a Shard Hunch that motivates a two-part Shard Question, the first part of which is currently being answered by the gradual development of an actual Shard Theory of human values, which hopefully provides answers to the second part with Shard Insight that can be implemented in AI systems to facilitate alignment.[2] Namely:

Shard Hunch: A human brain is a general intelligences, and its intentions and behavior are reasonably aligned with its many shards of values (i.e., little bits of "contextual influences on decision-making"[3]). Maybe something like that alignment can work for AI too![4]
Shard Question: How does the human brain ensure alignment with its values, and how can we use that information to ensure the alignment of an AI with its designers' values?
Shard Theory: The brain ensures alignment with its values by doing A, B, C, etc.
Shard Insight: We can ensure the alignment of an AI with its designers' values by doing X, Y, Z, etc. mapped from shard theory.

This is exciting! Now when I read shard theory research, I feel like I properly understand it as gradually filling in A, B, C, X, Y, Z, etc. For example, Assumptions 1, 2, and 3 in "The Shard Theory of Human Values" are examples of A, B, and C, and the...
Miracles and why not to believe them
by mruwnik — 2022-11-16T12:07:40.781Z

Wonders and signs

For a while I've wanted to explain to people why the testimony of the apostles isn't sufficient for me to believe them. I intuitively find that the gospels aren't enough for be to believe them. Even if they're better preserved than any other ancient text etc. I usually go with extraordinary claims require extraordinary evidence, which is a good heuristic and is convincing to me, but I didn't have a good mathematical explanation of why it's so. I could come up with various reasons of why it's a good idea, but not a *proof*. It turns out that this is just yet another result of Bayes' rule...

Telepathy

Dr Samuel Soal was a parapsychologist who did a number of ESP experiments between 1941-1943 to discover whether it was possible to read someones mind. The basic setup was an agent with a set of 5 animal cards which were shuffled before each test. The agent would pick one at random, and the subject would try to guess what card was picked. Naively, one would expect the subject to be right around once in 5 trials, i.e. p = 0.2. This is not what Soal found. A Mrs Steward, who was the test subject, managed to correctly guess the card 9410 times out of 37100 trials (I'm impressed by her patience), which is a success rate of f = 0.2536. This is 25.8 standard deviations away from the expected success rate, which is a big deal. So we have:

- P(H0|X) - the null hypothesis that she should have random results with a success rate of 0.2
- P(Hf|X) - an alternative hypothesis that she is an ESPer if the success rate is 0.2535

Plugging the values into the appropriate equations (check page 121 of the book), it turns out that the probability of getting this data with the null hypothesis is some 3.15×10−139. So it appears that the only thing left is to accept that ESP is real, given the following equation:

P(Hf|DX)=P(Hf|X)P(D|HfX)P(D|X), where P(D|X)=P(DH0|X)+P(DHf|X)

Since P(D|H0X) is absurdly small, this can be approximated as P(Hf|DX)=P(Hf|X)P(D|HfX)P(Hf|X)P(D|HfX)≈1, as P(H0|DX)≈0. Not very helpful. This pretty much says that if your prior of P(Hf|X) isn't too small, you're going to end up with Objective Scientific Evidence that ESP is real.

Getting real

The obvious problem with the above calculations is that those aren't the only two possible hypotheses. An alternative would be that they cheated. Or that the c...
How do people do remote research collaborations effectively?
by Krieger — 2022-11-16T11:51:46.571Z

I think remote research collaboration has a valley-of-inefficiency, where at mid-low levels of collaboration-competence it is much better to just do all research on your own rather than trying to split things up between multiple people (probably due to coordination/communication overhead and all the usual stuff).

While there's been a lot of posts here regarding advice for independent researchers, I haven't seen much for research collaborations (especially at small-scale, without a legible central leadership). People seem to do it pretty frequently/reliably though, e.g., all the forum posts with multiple authors.

Are there any advice/resources/procedures for effective remote research collaborations on the scale of 2-5 people?
Method of statements: an alternative to taboo 
by Q Home — 2022-11-16T10:57:49.937Z

[Status: this post grew 11 times bigger than I intended. Initially, I just wanted to write about about AI risk and Logical Decision Theory. I encourage you to read the first 3-4 parts of the post and decide if my method makes sense or not. If the method doesn't "click" with you, then this post is in category of vague musings. If the method "clicks", then the content of the post is more similar to strict theorems. The method is related to many rational techniques, I think it could be a powerful research tool. If you disagree with my method, maybe you can still use it as a fake framework. The secondary goal of the post is to gradually introduce my theory of perception.]

...

I want to share a way of dissolving disagreements. It's also a style of thinking. I call it "the method of statements", here's the description:

Take an idea, theory or argument. Split it into statements of a certain type. (Or multiple types.)

Evaluate the properties of the statements.

Try to extract as much true/interesting information as possible from those statements.

One rule:

A statement counts as existing even if it can't be formalized or expressed in a particular epistemology.

I can't define what a "statement" is. It's the most basic concept. Sometimes "statements" are facts, but not always. "Statements" may be non-verbal or even describe logical impossibilities. A set of statements can be defined in any way possible. I will give a couple of less controversial examples of applying my method. Then more controversial examples. And share my research ideas in the context of the method.

There's three ways to look at the method:

It's a little technique (like "taboo") which can be interesting to think about.

It's similar to the research method Eliezer used in the Sequences: think about properties of the things you already know, prove interesting statements about the things you already know.

The method is a counterpart of LW rationality. It can deal with all kinds of weird/meta- truths, such as metaethical truths. We could try to use it for writing Friendly AI.

Analytic philosophy (when it deconstructs ideas into logical propositions) is an inspiration. But "statements" in my method are something more than logical propositions.

Taboo

We all know the technique called "taboo". Imagine a disagreement about this question: ("Taboo Your Words")

If a tree falls in a forest and no one is ar

...
The two conceptions of Active Inference: an intelligence architecture and a theory of agency
by Roman Leventov — 2022-11-16T09:30:47.484Z

I think much of the confusion about Active Inference arises because the term is used to refer to two related, yet distinct concepts (one can also call them abstractions, theories, or ontics): an intelligence architecture and a theory of agency. In the literature on Active Inference, little or no attention is given to this distinction, which leads to misinterpretations, such as when the writer used the term to refer to one concept, and the reader understood it as referring to another, or, even more likely, not realising the distinction in the first place.

In Active Inference, Parr, Pezzulo, and Friston have called these two conceptions the “Low Road” and the “High Road to Active Inference”, and described them in chapters 2 and 3 of the book, respectively.

Active Inference as an architecture for intelligent agents

First, Active Inference can be seen as an architecture for autonomous intelligent agents. The main pillars of this architecture are variational Bayesian inference, planning-as-inference and action control-as-inference, and specification of preferences as a distribution over future observations (or hidden world states). This architecture can be implemented as an AI agent.

Parr, Pezzulo, and Friston see the process model of human intelligence on the neurobiological level as implementing this architecture. I think this position should be thought of as a subjective (or if you want, philosophical) view, rather than a falsifiable scientific claim.

Seth (2021) also classifies his "controlled and controlling hallucination" process theory of the brain as a species of Active Inference.

This conception of Active Inference is directly comparable to Reinforcement Learning, LeCun’s vision for the architecture of autonomous machine intelligence, the architecture of Gato, ReduNets, and most cognitive architectures.

Normative engineering science tells us that architectures should be compared with respect to a set of desirable properties (characteristics) of the engineered physical objects that have these architectures, i. e., with respect to a concrete engineering task. In the realm of biological evolution, genotypes and memotypes of some species of living organisms can be evolved through evolution, and the biological "architectures" they entail should be compared to each other based on their fitness to a particular ecological niche or based on some other evolutionary metri...