trentmkelly/LessWrong-43k huggingface dataset:
Knowledge Base 2: The structure and the method of building
by iwis — 2023-10-09T11:53:14.689Z

This is the second post of a series of posts that propose to build a crowdsourced knowledge base and use it to increase intelligence of people and computers, including AI. This post describes the structure of the knowledge database and the method of building it.

Question and answer websites

Do you know Quora.com, StackExchange.com, or StackOverflow.com? On each of these websites you can ask questions and get answers. Figure 1 schematically shows the interface of these websites. Questions can be tagged, both questions and answers can be commented. The main advantage compared to internet forums is the ability to vote for or against an answer. Thanks to this, answers can be sorted from the best to the worst [1]. Users earn points for their activity.

Figure 1

Fine-grained analogy

Let’s try to use an analogous interface to build a knowledge base. By a knowledge base I mean a database with reasonably credible information that is easily accessible to both people and computers. To do this, we change a format of a question to this pair: object/item name [> object/item name] > feature name, and the text of an answer to the value of this feature. The square brackets indicate that we can optionally specify what part of the object/item we have in mind. So, for many questions instead of asking a question like What is the resolution of HP ProBook 430 G1 notebook screen? we can just write: HP ProBook 430 G1 > Screen > Resolution. Likewise, instead of adding the answer HP ProBook 430 G1 notebooks have a screen with 1366×768 resolution, we can just add the answer 1366×768.

A “question” can have several mutually nonconflicting answers – for example, the aforementioned notebook model has variants that differ in the screen resolution – see figure 2a. Users can vote for or against correctness of a particular answer. Based on these votes, the system estimates probability of its correctness. Answers are also called information because each answer is a piece of information.

Figure 2 (click here for a navigable version of the slides)

Generally, the interface is quite similar to Wikidata - the system used to collect data displayed in Wikipedia infoboxes.

Quantifiers

A user can add one of the following quantifiers to an answer specifying the scope of its validity:

all - the answer applies to all notebooks of this model

some - the answer applies to some notebooks of this model

...
We don't understand what happened with culture enough
by Jan_Kulveit — 2023-10-09T09:54:20.096Z

This is a quick response to

Evolution Provides No Evidence For the Sharp Left Turn

, due to it winning first prize in

The Open Philanthropy Worldviews contest

. I think part of the post is sufficiently misleading about evolutionary history and the OP first prize gives it enough visibility, that it makes sense to write a post-long response.
Central evolutionary biology related claim of the original post is this:

The animals of the generation learn throughout their lifetimes, collectively performing many billions of steps of learning.
The generation dies, and all of the accumulated products of within lifetime learning are lost.
Differential reproductive success slightly changes the balance of traits across the species.

…
The only way to transmit information from one generation to the next is through evolution changing genomic traits, because death wipes out the within lifetime learning of each generation.
…
However, this sharp left turn does not occur because the inner learning processes suddenly become much better / more foomy / more general in a handful of outer optimization steps. It happens because you devoted billions of times more optimization power to the inner learning processes, but then deleted each inner learner shortly thereafter. Once the inner learning processes become able to pass non-trivial amounts of knowledge along to their successors, you get what looks like a sharp left turn. But that sharp left turn only happens because the inner learners have found a kludgy workaround past the crippling flaw where they all get deleted shortly after initialization.

In my view, this interpretation of evolutionary history is something between "speculative" and "wrong".
Transmitting some of the data gathered during the lifetime of the animal to next generation by some other means is so obviously useful that is it highly

convergent

. Non-genetic communication channels to the next generation include epigenetics, parental teaching / imitation learning, vertical transmission of symbionts, parameters of prenatal environment, hormonal and chemical signaling,

bio-electric signals

, and transmission of environmental resources or modifications created by previous generations, which can shape the conditions experienced by future generations (e.g. beaver dams).
Given the fact overcoming the genetic bottleneck is so highly convergent, it seems a bit surprisin...
Paper: Identifying the Risks of LM Agents with an LM-Emulated Sandbox - University of Toronto 2023 - Benchmark consisting of 36 high-stakes tools and 144 test cases!
by Singularian2501 — 2023-10-09T00:00:19.859Z

Paper: https://arxiv.org/abs/2309.15817

Github: https://github.com/ryoungj/toolemu

Website: https://toolemu.com/

Abstract:

Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks - such as leaking private data or causing financial losses. Identifying these risks is labor-intensive, necessitating implementing the tools, manually setting up the environment for each test scenario, and finding risky cases. As tools and agents become more complex, the high cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks. To address these challenges, we introduce ToolEmu: a framework that uses an LM to emulate tool execution and enables the testing of LM agents against a diverse range of tools and scenarios, without manual instantiation. Alongside the emulator, we develop an LM-based automatic safety evaluator that examines agent failures and quantifies associated risks. We test both the tool emulator and evaluator through human evaluation and find that 68.8% of failures identified with ToolEmu would be valid real-world agent failures. Using our curated initial benchmark consisting of 36 high-stakes tools and 144 test cases, we provide a quantitative risk analysis of current LM agents and identify numerous failures with potentially severe outcomes. Notably, even the safest LM agent exhibits such failures 23.9% of the time according to our evaluator, underscoring the need to develop safer LM agents for real-world deployment.
AI Alignment Breakthroughs this week (10/08/23)
by Logan Zoellner — 2023-10-08T23:30:54.924Z

[Cross-posted from https://midwitalignment.substack.com/p/ai-alignment-breakthroughs-this-week-a6e ]
Note: I change to a new (much longer) format based off of feedback. Please let me know if you find this helpful (or not)

AI Alignment Breakthroughs this week (10/08/23)

Hi,

Based off of some feedback, I’m going to try and change the format a little bit. For each entry I will try to give:

A brief description

Why this is a breakthrough

How it helps with alignment

I’m still going to try things somewhat short in order to save my (and hopefully your) time.

That being said, there there were two major breakthroughs this week that I want to highlight (both in Mechanistic Interpretability), AI Lie Detector and Interpreting Neuron Superposition.

AI Lie Detector

( https://twitter.com/OwainEvans_UK/status/1707451418339377361 )

There’s a good writeup of this here, but the most interesting thing is that it was trained to detect lies on one model (GPT-3.5), but actually works on a number of others models. The fact that it generalizes to other models gives us strong hope that it is the kind of thing we can robustly use when scaling up AI (as long as we are using similar architectures).

Obviously this might stop working if we switch to a radically different architecture. And it also might stop working if we apply too much optimization pressure (say we start using gradient descent to find plans the lie-detector says are truthful). Still, used appropriately, it seems like a valuable tool to for an aligner to use to double-check the output of an LLM.

Interpreting neurons in superposition

This one is a little harder to explain than the AI lie detector, but it’s arguably a much bigger breakthrough.

( https://twitter.com/ch402/status/1709998674087227859)

Anthropic claims to have a method that allows them to understand neurons in a neural network that are in superposition. In this case, there’s no spooky quantum weirdness going on, rather superposition simply means that a neuron can encode more than one concept. For example, if a neuron held two bits of state, one of them could mean “black vs white” and one of them could mean “cat vs dog”. In this case, the four possible values would be:

00: “black cat”
01: “white cat”
10: “black dog”
11: “white dog”
Obviously in a real neural network the superposition could be much more complicated than this, and fin...
"The Heart of Gaming is the Power Fantasy", and Cohabitive Games
by Raemon — 2023-10-08T21:02:33.526Z
Link: https://bottomfeeder.substack.com/p/the-heart-of-gaming-is-the-power

I quite liked @mako yass' Peacewagers/Cohabitive Games so Far. But I found myself wanting to link to this essay by Jeff Vogel, an indie game developer and blogger. I think it says something important about the constraints of how to develop Cohabitive games.

Then I thought "actually, Jeff Vogel's post here is just a great post, I wanna make a top-level linkpost."

Vogel is an indie-game developer I've been following since the 90s, and he's got a great blog that's focused in large part on how to be commercially successful as a game developer. You can have all kinds of beautiful artsy ideas as an game designer, but if you can't make something that makes enough money to pay your bills, well, then there's a limit to how many games you can make, how many people you can hire to help make them, etc.

I was excited by mako's vision of "cohabitive games from a perspective of improving society." i.e. helping people practice openended negotation, and giving coordination theorists an empirical sandbox. But I think succeeding at this requires games that are commercially successful. (i.e. if you've only made a niche product for a few atypical gamers, you probably haven't succeeded at the more ambitious goals in the Peacewagers post)

With that context, here's some quotes from Jeff Vogel's post:

"Power Fantasy" is usually used as a pejorative term. The phrase often has the word "adolescent" tacked on before it for a fun bit of ad hominem.

Yet. What makes video games different from other artistic media? It is activity. Video games aren't about consuming words or sounds or images. They are about action, doing things.

In almost every game, what are you doing? You are making changes in your environment. You are giving gifts in a dating sim. Designing roadways in Sim City. Shooting bad guys in, well, just about everything else.

When you change your environment, you are exercising Power.

You are doing this inside a game, a space that does not exist. Every video game takes place in a mental construct that is imagined. In other words, all video games take place in Fantasy. (In the dictionary sense, not the "casting fireball at an orc" sense.) Yet, in your brain, the Power FEELS real, and that is what counts.

Video games are about using power to make changes in a fantasy space, for pleasure. They are power fantasies.

There is nothing shameful about this. The only error is denying it. This w

...
FAQ: What the heck is goal agnosticism?
by porby — 2023-10-08T19:11:50.269Z

Thanks to George Wang, Liron Shapira, Eliezer[1], and probably dozens of other people at Manifest and earlier conferences that I can't immediately recall the names of for listening to me attempt to explain this in different ways and for related chit chat.

I've gotten better at pitching these ideas over the last few conferences, but I think some of the explanations I've provided didn't land nearly as well as they could have. It's sometimes hard to come up with great explanations on the spot, so this FAQ/dialogue is an attempt to both communicate the ideas better and to prepare a bunch of explanatory contingencies.

The content and tone of these questions shouldn't be taken to represent any real person I talked to.[2]

This post is very roughly organized into four sections:

A series of questions introducing what I mean by goal agnosticism,
Addressing general doubts about goal agnosticism and diving into what it is and isn't in more detail,
Addressing concerns about goal agnosticism as we reach extreme optimization pressures and capabilities, and
A conclusion of miscellaneous questions/notes and a discussion of objections that I currently lack nonfuzzy answers for.

The questions and answers flow into each other, but it should be safe to jump around.

Introductory Questions

Q1: Why should anyone care about goal agnosticism?

Goal agnostic systems enable a promising path for incrementally extracting extreme capability from AI in a way that doesn't take a dependency on successfully pointing a strong optimizer towards human-compatible values on the first try. An intuitive form of corrigibility seems accessible.

I view goal agnostic systems as a more forgiving foundation from which we can easily build other agents. The capabilities of the system, particularly in predictive models as are used in LLMs, can directly help us aim those same capabilities. The problem is transformed into an iterative, bottom-up process, rather than a one shot top-down process.

If we can verify, maintain, and make productive use of goal agnostic systems, we have a decent shot at avoiding the convergent pitfalls implied by strong goal-directed optimizers.

As a bonus, there are some reasons to think that goal agnosticism and its productive use is an attractor in design space, at least for a notable subset of the tech tree.

Q2: What do you mean by goal agnosticism?

I have been using the phrase "goal agnostic...
Time is homogeneous sequentially-composable determination
by TsviBT — 2023-10-08T14:58:15.913Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/06/time-is-homogeneous-sequentially.html. First completed June 30, 2023.]

Time is the character of courses of events in which determinations——the ways that one event determines the next——are uniform across events and across composing determinations.

Thanks to Sam Eisenstat, Scott Garrabrant, Brady Pelkey, and Kyle Scott for related conversations.

Reasons to talk about time

Why talk about time? Here are some key concepts that are bound up with time, and so make demands on the understanding of time:

Stability. Suppose a human understands another mind well enough to know that the other mind will have effects the human desires. That understanding (arguably) has to describe something in some sense fixed about the other mind. What the understanding describes about the mind is, in other words, stable in the mind. Stability is an in some sense dynamical concept, since it says: "This could change over time, but it doesn't.".

Learning, growth. A mind comes into being by gaining new understanding over the course of a voyage of novelty——information is gained, ideas are created, elements become explicit. This process is fraught, in part because it's dynamic.

Counterfactuals, agency. The notion of affecting something is, pretheoretically, temporal. "If I do this, then what will happen afterward as a consequence of that decision?"

Values coming to be. The fundamental question "What determines a mind's effects?" asks about the mind's telotect——what brings about that the mind pushes the world in some direction, and what selects the telopheme, which says what direction the mind pushes the world. Across ontological crises, and across the process of becoming more coherent starting from nothing, and across the process of unfolding reference through novelty, a mind has to somehow arrive at how it will affect [the world as understood by the new understanding] (for example, by FIAT).

Change, tendency. In general, there may be mental clines——elements that by their nature tend to change in some direction. An element is provisional when it hasn't yet followed all its clines. Clines follow their tendencies over time.

Dynamics. In general, dynamical concepts depend on time. E.g.: attraction basins, in/stability, dis/equilibrium, identity, forces, path dependence. These concepts beg questions about the temporal context like: {st

...
Linkpost: Are Emergent Abilities in Large Language Models just In-Context Learning?
by Erich_Grunewald — 2023-10-08T12:14:46.825Z
Link: https://arxiv.org/abs/2309.01809

A new preprint, Lu et al., published last month, has important implications for AI risk if true. It essentially suggests that emergent capabilities of current LLMs are (with some exceptions) mediated by a model's ability to do in-context learning. If so, emergent capabilities may be less unexpected, given that improvements to in-context learning (e.g., via instruction tuning) are relatively predictable.

Wei et al. defines "emergent ability" like so: "An ability is emergent if it is not present in smaller models but is present in larger models." So you can have sudden "jumps" in performance on various tasks as you scale models up, e.g. (from Wei et al.):

Lu et al. is saying, yes, but that is not because the larger models learned to do those specific tasks better, or learned to become better generic reasoners. It is because the larger models learned to follow instructions better, partly from scaling up, and partly being trained on data sets of instructions and instruction-following ("instruction tuning").

The methodology: "We conduct rigorous tests on a set of 18 models, encompassing a parameter range from 60 million to 175 billion parameters, across a comprehensive set of 22 tasks. Through an extensive series of over 1,000 experiments, we provide compelling evidence that emergent abilities can primarily be ascribed to in-context learning."

The paper finds that, when accounting for the "biasing factors", only 2 of 14 previously "emergent" tasks actually show emergence. The blue and yellow lines are instruction-tuned models evaluated via few-shot prompting, showing emergence. The purple and green lines are non-instruction-tuned models evaluated via zero-shot prompting, generally showing no emergence.

They write:

Ultimately, this absence of evidence for emergence represents a significant step towards instilling trust in language models and leveraging their abilities with confidence, as it is indicative of the complete lack of latent hazardous abilities in LLMs, in addition to being controllable by the user. [...]

The distinction between the ability to follow instructions and the inherent ability to solve a problem is a subtle but important one. Simple following of instructions without applying reasoning abilities produces output that is consistent with the instructions, but might not make sense on a logical or common sense basis. This is reflected in the well-known phe

...
Bird-eye view visualization of LLM activations
by Sergii — 2023-10-08T12:12:25.593Z
Link: https://grgv.xyz/blog/llama/

I’m starting to learn about mechanistic interpretability, and I’m seeing lots of great visualizations of transformer internals, but somehow I’ve never seen the whole large model’s internal state shown at once, on one image.

So I made this visualization, for Llama-2-7B. Attention matrices are on the left, in 32 rows for 32 blocks, top to bottom. To the right, there are 64 rows: residual stream (odd rows) and internal MLP activations (even rows). Finally, output MLP and unembedding layer are on the bottom.

Activation maps are downscaled horizontally, with maxpooling, to fit into 1000px wide image.

Example for a prompt “2+2=”:

And an example for the prompt: "William Shakespeare was born in the year”:

And for the prompt "blue pencils fly over moonlit toasters”:

Probably not especially useful for interpretability, but at least it looks pretty )
Perspective Based Reasoning Could Absolve CDT
by dadadarren — 2023-10-08T11:22:49.458Z

I have been advocating perspective-based reasoning (PBR) as the solution to anthropic paradoxes for the past few years. Here I would like to explore its implication on the decision theory side, most notably how it could absolve the shortcomings of CDT.

The concept of PBR is simple: treat perspective and accompanying indexicals as a fundamental part of basic logic, something that cannot be explained otherwise. e.g. you know this physical person is you, not because this person is logically significant or physically unique in some way, but because you can inherently tell that is the self. You simply know this since the subjective feeling is immediately linked to it. There's no other rationale.

Anthropics

The defining feature of anthropic paradoxes is the use of perspective-based indexicals in question formulation while not attaching them to specific physical observers (or observer-moments). In The Presumptuous Philosopher: the evidence of "my existence", in The Doomsday Argument: "the birth rank of me/our generation" and in The Sleeping Beauty Problem: "the awakening today" are all examples of this.

People tend to solve these questions with a physicalist view from nowhere, thus disregarding perspectives in reasoning. Not able to treat the indexicals as primitive, they are arbitrarily transcoded to mean something else, e.g. treat them as random samples. That is what leads to paradoxes.

In contrast, these questions should be answered from the given perspective, the indexicals treated as primitively understood. So self-existence is presupposed prior knowledge, and the probability distribution of indexicals among some assumed reference class is ill-defined. PBR therefore would reject both Doomsday Argument and Presumptuous Philosopher.

A Common Decision-Based Argument

It goes like this: if every observer in my situation makes the same decision as I do then the group would be better off overall. Therefore the probability that would lead me to such a decision is the right one. (what is considered "better", e.g. by measuring the average or sum of rewards, differs from camp to camp) It is often used to argue for the validity of the probability distribution of self among a supposed reference class, e.g. "the probability that I am the first 90% of human beings born into the world".

The supposed probability concerns the indexical I specifically, the ...