trentmkelly/LessWrong-43k huggingface dataset:
QNR prospects are important for AI alignment research
by Eric Drexler — 2022-02-03T15:20:53.826Z

Attention conservation notice: This discussion is intended for readers with an interest in prospects for knowledge-rich intelligent systems and potential applications of improved knowledge representations to AI capabilities and alignment. It contains no theorems.

Abstract

Future AI systems will likely use quasilinguistic neural representations (QNRs) to store, share, and apply large bodies of knowledge that include descriptions of the world and human values. Prospects include scalable stores of “ML-native” knowledge that share properties of linguistic and cognitive representations, with implications for AI alignment concerns that include interpretability, value learning, and corrigibility. If QNR-enabled AI systems are indeed likely, then studies of AI alignment should consider the challenges and opportunities they may present.

1. Background

Previous generations of AI typically relied on structured, interpretable, symbolic representations of knowledge; neural ML systems typically rely on opaque, unstructured neural representations. The concept described here differs from both and falls in the broad category of structured neural representations. It is neither fully novel nor widely familiar and well explored.

The term “quasilinguistic neural representations” (QNRs) will be used to denote vector-attributed graphs with quasilinguistic semantics of kinds that (sometimes) make natural language a useful point of reference; a “QNR-enabled system” employs QNRs as a central mechanism for structuring, accumulating, and applying knowledge. QNRs can be language-like in the sense of organizing (generalizations of) NL words through (generalizations of) NL syntax, yet are strictly more expressive, upgrading words to embeddings[1a] (Figure 1) and syntax trees to general graphs (Figure 2). In prospective applications, QNRs would be products of machine learning, shaped by training, not human design. QNRs are not sharply distinguished from constructs already in use, a point in favor of their relevance to real-world prospects.[1b]

Motivations for considering QNR-enabled systems have both descriptive and normative aspects — both what we should expect (contributions to AI capabilities in general) and what we might want (contributions to AI alignment in particular).[1c] These are discussed in (respectively) Sections 2 and 3.

[1a] For example, embeddings can represent images in ways that

...
On Bounded Distrust
by Zvi — 2022-02-03T14:50:00.883Z

Response To (Scott Alexander): Bounded Distrust

Would that it were that simple.

There is a true and important core idea at the center of Bounded Distrust.

You can (and if you are wise, you often do) have an individual, institution or other information source that you absolutely do not trust to reliably tell the truth or follow its own explicit rules. Yet by knowing the implicit rules, and knowing the incentives in place and what the consequences would be if the source was lying to various extents, you can still extract much useful information.

Knowing what information you can and can’t extract, and what claims you can trust from what sources in what contexts to what extent, is a vital life skill.

It is also a difficult and often an anti-inductive skill. Where there is trust, there is the temptation to abuse that trust. Each person has a unique set of personal experiences in the world and samples a different set of information from various sources, which then evolves one’s physical world models and one’s estimates of trustworthiness in path dependent ways. Making real efforts from your unique epistemic perspective, will result in a unique set of heuristics for who you can and cannot trust.

Perspectives can become less unique when people decide to merge perspectives, either because they trust each other and can trade information on that basis, or because people are conforming and/or responding to social pressure. In extreme cases large groups adopt an authority’s stated heuristics wholesale, which that authority may or may not also share.

Scott’s model and my own have much in common here, but also clearly have strong disagreements on how to decide what can and cannot be trusted. A lot of what my weekly Covid posts are about is figuring out how much trust we can place where, and how to react when trust has been lost.

This all seems worth exploring more explicitly than usual.

I’ll start with the parts of the model where we agree, in list form.

None of our institutions can be trusted to always tell the truth.
However, there are still rules and associated lines of behavior.
Different rules have different costs associated with breaking them.
These costs vary depending on the details, and who is breaking what rule.
This cost can, in some cases, be so high as to be existential.
In some situations, this cost is low enough that statements cannot be

...
Why Do I Think I Have Values?
by J Bostock — 2022-02-03T13:35:07.656Z

Epistemic status: I just thought of this today, and it made a huge amount of my confusion on this topic disappear.

Values are this weird concept. Humans aren't utility maximizers, but we think of ourselves as them. There's a paradox in the values-beliefs framework: humans can identify the parts of themselves which correspond to values and beliefs, but this should actually be impossible? Here I propose the following answer:

Humans model themselves with an algorithm which has evolved to model other humans as value-belief based systems.

So why would this be the case? I can think of a few reasons.

Minimaxing

The minimax algorithm as applied to chess (or other finite-option, total-information games) means assuming your opponent will play the best move for them on every turn. This is the optimal[1] way to play in the infinite compute limit, and is also pretty close to optimal in most other situations (actual algorithms sometimes attempt to compute the probability of their opponent playing each given move).

So what do you do if you're a hominid playing politics against other hominids? It seems likely that the most successful strategy would be to (by default) assume your opponent is as powerful as possible. This you'll take the fewest risks, and that you'll be the most accurate when it comes to the most dangerous opponents.

Since coherent decisions imply consistent utility, the most powerful opponents are expected utility maximizers, especially those who want things which cause them to compete with you. If your political opponent isn't well approximated by a utility maximizer, they won't be a threat. Even more subtly, if your opponent takes a mixture of actions, the most "important" actions will look like them acting as an expected utility maximizer. All the other actions will be lost as noise.

Low Compute

Modelling another human as an expected utility maximizer is also probably quite efficient. Starting with a prior that they want the exact same things as a typical human, and then updating away from that, is pretty cheap. Starting with a prior that they will act towards their desires based on all the information they can see is also pretty cheap. Combined they do a decent job of guessing another human's actions. Especially the set of actions which are most relevant to you.

This only requires modelling another's senses, keeping track of their knowledge, any d...
Trying harder works, sometimes
by Kaveh-Sedghi — 2022-02-03T05:04:16.903Z

It seems to be commonly accepted that pressing the "Try Harder" button does not work. While I agree with this in general, I would argue that there are edge cases where trying harder is a valid option.

To illustrate this, is a story: of science, and how I used it to change a particularly pernicious habit.

A Story of Habit Change

I used to have the bad habit of lying on my bed and using my computer, rather than utilizing my desk. This was bad for several reasons:

1. Reduced productivity, as doing work on paper becomes practically impossible.

2. Net-negative impact on long-term health due to bad posture.

Naturally, I wanted to get rid of this habit, and I turned to one of the most effective tools I know of - the Scientific Method. I hypothesized that the reason I was lying on my bed rather than using my desk was due to comfort - the bed was the more appealing option.

I resolved, then, to reduce the comfort appeal of my bed by relocating my blanket, along with my pillows, to an unused room. I reasoned that this would make the friction of attempting to get back the comfort high, helping me break the habit.

For a time, this strategy worked. My productivity increased, and my posture became qualitatively better. But after about 12 days, my strategy failed.

The strategy was sound, and the hypothesis seemed to correspond to observations (while being intuitively correct as well). Yet, it did not work in the long run.

As a rationalist, and as a knower, I should have recognized that perhaps my hypothesis was flawed, and tried a different approach. Instead, I opted to try the same strategy again - I repeated the experiment. I did not attempt to change anything, and simply did what I had done earlier. But this time, I tried harder, and resolved to be more committed.

It has been 3 months, and so far, this seems to be working.

Reflections

My experience reminded me of something I had read, in Philosophy of Science: A Very Short Introduction. Okasha, the author, points out that there are problems associated with Popper's demarcation of science from pseudo-science. Popper criticized ideas such as Freudian psychoanalysis because he thought that they could explain any observation - essentially, he thought that they couldn't be falsified.

However, there are instances where sticking to a theory and explaining away failures and new observations is helpful. The example given in the book is of...
Observed patterns around major technological advancements
by Richard Korzekwa  — 2022-02-03T00:30:06.306Z

Cross-posted from the AI Impacts blog

Summary

In this post I outline apparent regularities in how major new technological capabilities and methods come about. I have not rigorously checked to see how broadly they hold, but it seems likely to me that they generalize enough to be useful. For each pattern, I give examples, then some relatively speculative explanations for why they happen. I finish the post with an outline of how AI progress will look if it sticks to these patterns. The patterns are:

The first version of a new technology is almost always of very low practical value

After the first version comes out, things improve very quickly

Many big, famous, impressive, and important advances are preceded by lesser-known, but still notable advances. These lesser-known advances often follow a long period of stagnation

Major advances tend to emerge while relevant experts are still in broad disagreement about which things are likely to work

Introduction

While investigating major technological advancements, I have noticed patterns among the technologies that have been brought to our attention at AI Impacts. I have not rigorously investigated these patterns to see if they generalize well, but I think it is worthwhile to write about them for a few reasons. First, I hope that others can share examples, counterexamples, or intuitions that will help make it clearer whether these patterns are “real” and whether they generalize. Second, supposing they do generalize, these patterns might help us develop a view of the default way in which major technological advancements happen. AI is unlikely to conform to this default, but it should help inform our priors and serve as a reference for how AI differs from other technologies. Finally, I think there is value in discussing and creating common knowledge around which aspects of past technological progress are most informative and decision-relevant to mitigating AI risk.

What I mean by ‘major technological advancements’

Throughout this post I am referring to a somewhat narrow class of advances in technological progress. I won’t try to rigorously define the boundaries for classification, but the kinds of technologies I’m writing about are associated with important new capabilities or clear turning points in how civilization solves problems. This includes, among many others, flight, the telegraph, nuclear weapons, the laser, pe...
Contra Infinite Ethics, Utility Functions
by kyleherndon — 2022-02-02T22:32:24.897Z

Suppose you're a human, and you think we're onto something with modern physics. Humans have a brain that does not exceed a meter in radius. Humans have a brain that weighs less than 100kgs. By the

Bekenstein bound

, we know the average human brain could have at most 2.6*10^42 bits before forming a black hole, and for the upper bounds I gave, that number is larger, but note, it is still finite.

Now let's say you want to make decisions. What does it mean to make decisions? That's tricky, but we can say it has something to do with performing actions based on sense-perceptions. Let's assume that you are fully aware of your entire brain and its information contents as your sense-perception, and you can completely modify its information contents as your action space. We now have a finite action space and finite perception space, which allows for a finite decision space wherein we define a function that tells you the best choice in action space from every element in perception space. This function can be given by a finite length program.

All a useful decision theory needs is to tell you what one of the best actions for a given perception is. Maybe it's not always unique, but there is no need to even assign utility functions to consider what the best action is. All you need is to form a partial order for your preferences over the collection, so let's only talk about preferences (which could be given by utility functions, but don't have to be).

"Now, now!" You say. "What about potentially larger actors, such as actors that I could modify myself into?" Well, first of all, you still have a finite decision space and you would have to be choosing between what actors to become. Just because you can think about an infinite path of actors that you could transform into, step after step, and perhaps achieve something increasingly large doesn't mean that there is a useful corresponding decision theory for you now.

We can also use

Bremermann's Limit

to show that even if you somehow managed to survive for the entire expected length of the universe (as our very useful thermodynamic models give) you would still have a finite amount of computation that you can perform before taking that action. This means you can at best make a finite-accuracy prediction of possible futures. At some fidelity, your ability to predict infinite processes accurately fails, and you weigh the possible futures...
January 2022 Time Tracking
by jefftk — 2022-02-02T22:00:06.311Z

Time tracking can be a good way to see how you've been spending your
time and how it compares to how you would like to be spending it. It's
something that Julia and I have done a few times (

December 2017,

June 2017,

2015,

2011) and I've generally found it pretty
helpful. This time we tracked two weeks, 2022-01-05 through
2022-01-11 and 2022-01-18 through 2022-01-24. These were pretty
normal work weeks for the two of us these days:

This is with three kids, 7y, 5y, and 7m.

Paid childcare 9-5 each workday, with the exception of one day
where our nanny called in sick.

Nora was sick one of the days. Julia spent more time nursing
her, including during the workday, because she was refusing the
bottle.

No travel or dance gigs.

No big house projects.

I took one afternoon off work to take the older kids
sledding.

Categories:

Sleep: Lights out to lights on. Julia tracked an average of
0:24/night of post-nursing insomnia that's included in the total here.
I also had some insomnia after nighttime wakings, but wouldn't have
been able to track it without making the insomnia worse. I subtracted
0:15 each night as a guess for how long Julia and I talk before
falling asleep. Does not include night nursing; that's under
"childcare".

Work: The entire workday, including lunch, incidental personal
stuff mixed in with working, pumping milk etc. Julia works from home;
I worked 4/10 days from the office, and included the commute here
(0:09 of the total; 0:35 per day I commuted).

Housework: cooking, cleaning, tidying, fixing things, dealing
with taxes.

J+J: time together without any of the kids.

Family: relaxing/playing with others in the family

Childcare: taking care of the kids. Julia's time includes
1:55/day of nursing (62% of her total). Nursing while hanging out
with family is under "family", though.

Personal: blogging, making a go
board, exercising, reading things, showering, free time generally.

Some distinctions:

When doing multiple things at once, like both cooking and
looking after a fussy Nora, we counted these as "housework" >
"childcare" > "work". That is, doing housework and childcare at the
same time coded as "housework".

"Family" vs "childcare" is a bit fuzzy. In 2017 we counted all
time solo with the kids as "childcare", but as they have gotten older
that no longer seems like a good fit.

...
OpenAI Solves (Some) Formal Math Olympiad Problems
by Michaël Trazzi — 2022-02-02T21:49:36.722Z

Epistemic status: I have just skimmed through OpenAI's blogpost and paper, I do not fully understand the details.

From the blogpost

We built a neural theorem prover for Lean that learned to solve a variety of challenging high-school olympiad problems, including problems from the AMC12 and AIME competitions, as well as two problems adapted from the IMO.

[...]

The prover uses a language model to find proofs of formal statements. Each time we find a new proof, we use it as new training data, which improves the neural network and enables it to iteratively find solutions to harder and harder statements.

From the paper

We explore the use of expert iteration in the context of language modeling applied to formal mathematics. We show that at same compute budget, expert iteration, by which we mean proof search interleaved with learning, dramatically outperforms proof search only. We also observe that when applied to a collection of formal statements of sufficiently varied difficulty, expert iteration is capable of finding and solving a curriculum of increasingly difficult problems, without the need for associated ground-truth proofs. Finally, by applying this expert iteration to a manually curated set of problem statements, we achieve state-of-the-art on the miniF2F benchmark, automatically solving multiple challenging problems drawn from high school olympiads.

Method

Uses the Lean formal environment instead of the Metamath used in GPT-f.
Uses "decoder-only Transformers similar to GPT-3" with 774M trainable parameters
Pre-trained "successively on GPT-3’s postprocessed version of CommonCrawl (for 300B tokens) and an updated version of WebMath (for 72B tokens)"
"proof search interleaved with learning"

The two IMO-adapted problems

Problem 1: Suppose a, b, c are the sides of a triangle. Prove that a^2(b + c − a) + b^2(c + a − b) + c^2(a + b − c) ≤ 3abc.

Problem 2: For a, b, c reals, prove that (a^2 + ab + b^2)(b^2 + bc + c^2)(c^2 + ca + a^2) ≥ (ab+bc+ ca)^3.

Both solutions to those problems use "nlinarith" applied to the right arguments, which, as far as I understand, is a tactic from mathlib for solving nonlinear arithmetic problems by adding more assumptions to the context of the solver. (source)

The right arguments for the first problem are said in the blogpost to come (informally) from Schur's inequality, which gives

nlinarith [sq_nonneg (b - a), sq_nonneg (c - b), sq_nonneg

...
Is asymptomatic transmission less common after vaccination?
by Brendan Long — 2022-02-02T20:53:42.188Z

How I understand it, the problem with the first time you're exposed to COVID is that your immune system hasn't seen it before and it takes a while to ramp up and fight it off, allowing it to do more damage.

Vaccines solve this by exposing your immune system to something that looks like COVID so it can ramp up faster if/when you're exposed to real COVID.

One of the things the immune system does is cause symptoms (fever, coughing, running nose, etc.).

My question is, is this "slow immune system ramp-up" the reason for so much asymptomatic spread (since you become contagious before your immune system catches up), and post-vaccination does your immune system ramp up fast enough that you have symptoms earlier (before or while contagious)?

I've had a lot of trouble searching for this since so many people are surprised that vaccinated people aren't 100% immune and it's hard to find anyone who can be bothered to collect data (and of course, running actual experiments is crazy talk).
Thoughts on AGI safety from the top
by jylin04 — 2022-02-02T20:06:06.161Z

In this post, I'll summarize my views on AGI safety after thinking about it for two years while on the Research Scholars Programme at FHI -- before which I was quite skeptical about the entire subject. [1]

My goals here are twofold: (1) to introduce the problem from scratch in a way that my past self might have found helpful, and (2) to round up my current thoughts on how one might make progress on it, with an eye towards figuring out what I want to work on next. This note is currently split into 1+2 parts, aimed at addressing these two things.

More precisely,

in Part 1 (more), I will introduce the alignment problem and explain why I've become interested in it;
in Part 2 (more), I will discuss the epistemics of thinking about AGI safety at a time when AGI doesn't yet exist; and
in Part 3 (more), I will explain why the considerations of Part 2 lead me to want to work on the intersection of mechanistic interpretability and deep learning theory -- despite doubting that current algorithms will scale up to AGI.

The three parts are optimized for different audiences, with later sections being more "in the weeds" and technical. In particular, I expect that Part 1 will mostly not be new to longtime readers of this forum, and people who are already convinced that AI safety is worth thinking about may want to skip ahead to Part 2. On the other hand, Part 3 will mostly be an attempt to justify a particular research direction and not an introduction to what anyone else is doing.

Thanks to Spencer Becker-Kahn, Lukas Finnveden, Steph Lin, and Chris van Merwijk for feedback on a late-stage version of this post, and to other members of RSP for feedback on an earlier draft.

1. Introduction to the alignment problem

In this first part, I will introduce the alignment problem and explain why one might care about it from an EA point of view. This section will overlap considerably with other recent intros, especially Richard Ngo's excellent

AGI safety from first principles

.

1.1. What is the alignment problem?

Suppose that in the future

We will be able to build AI systems that are more competent than humans at performing any tasks that humans can perform, including intellectual ones such as producing corporate strategies, doing theoretical science research, etc. (let's call them "AGIs");
We will actually build such systems;
We will build them in such a way that they c

...