trentmkelly/LessWrong-43k huggingface dataset:
Recreating logic in type theory
by Thomas Kehrenberg — 2022-12-21T15:19:18.275Z

This is the second entry in my series on type theory. In this post, we will talk about how to recreate logic in type theory.

As you might know, in order to define ZFC set theory, you first define first-order (FO) logic separately, and then specify the ZFC axioms in the language of FO logic. But we want to be more ambitious here! We want to encode FO logic with the type concepts we already have! If that doesn’t make any sense right now, I hope at the end of this article it does.

But before we get there, a quick refresher on type-valued functions.

Type-valued functions

In the definition of sum types, we briefly mentioned that B(t) is a type-valued function:

(t,x):∑t:AB(t) .

We can of course precisely specify B’s type:

B:∏t:AU .

What is maybe surprising about this is that an instance of a type, like the function B, becomes part of a type. But this is how it works in type theory. In fact, we could have written the type of all groups a little differently:

f(T):≡(∏x:T∏y:TT)×TGroup:≡∑T:Uf(T)

The function f here doesn’t only return a type; it also takes a type as input! A function on types can use the operators ∑, ∏, +, and ×, and can get quite complicated. And, given any function f with codomain U (and arbitrary domain X), the following is a type: ∑x:Xf(x), namely, the disjoint union over all types in the image of f.

What to use for truth values?

OK, why do I mention all this? Because it will be relevant to how we will represent truth values (booleans in programming languages) in our encoding of logic.

The truth values in first-order logic are, shockingly, true and false. Propositions are statements that have a truth value – they're either true or false. (So, boolean variables/expressions in a programming language.)

And as mentioned in the beginning, we can talk about subsets of, say, N by talking about predicate functions (or just predicates) on N. A predicate function is a function that takes a single argument and returns a truth value – true or false.

So, you see, truth values are needed everywhere. How to represent them?

Our first naïve approach might perhaps be to represent “true” and “false” with the values of the type 2, which is the type that has exactly two elements. Let’s call them 02 and 12 (distinct from 0:N ...
You become the UI you use
by Viliam — 2022-12-21T15:04:17.072Z

Epistemic status: exploration

*

People instinctively copy other people they interact with. But the communication tools we use determine what we see and don't see (what we can copy and what we cannot copy) and often also who we interact with (whom we copy).

Written communication is a filter that lets you know that someone disagreed with you, but doesn't let you see their kind smile. Thus people get more quarrelsome.

Reading someone's article or a book lets your see the presentation of their ideas, but not all the hard work they put into figuring all that out. Thus people get overconfident and assume that any idea they can present is equally valuable.

Advertising is a way to interact with many people. Each one of them will only copy a little bit of your message, but there are thousands of them, so it may be worth it. Even better, the communication is one-sided, so you do not instinctively copy their opinions in turn. (As opposed to a salesman, who must learn to ignore other people's feedback, which is difficult for most people.)

Banning and blocking people are ways to prevent copying. Voting is a way to encourage copying of content that appeals to people who vote.

Before internet, people copied those they saw on TV, and those who succeeded to write books. On social networks they copy those who write most (unless they are so annoying they get blocked).

*

...this all feels like something we could pay attention to, and maybe use to improve the ways we communicate, and thus who we become by communicating to each other...

*

As a random crazy thought, most of us probably prefer reading text to watching videos, but what if we required every article to start with a 10 second video that would play automatically when you open the article? For example, the video of the author reading the summary.

That would probably be annoying for many reasons, both for the authors and the audience. So this is not an actual proposal. But I wonder if it would somehow change the emotional background of the website.

(I specifically say a 10 second video, because my objection against videos is that they take too much time, so I would instinctively try to avoid them. But if I knew that on this website, each introductory video is exactly 10 seconds long, I probably wouldn't mind as a reader.)

Another crazy idea is to integrate the website with something like Khan Academy or Duolingo. Like, every time you r...
Price's equation for neural networks
by tailcalled — 2022-12-21T13:09:16.527Z

Price's equation is a fundamental equation in genetics, which can be used to predict how traits will change due to evolution. It can be phrased in many ways, but for the current post I will use the following simplified continuous-time variant:

ddtx=gcov(x,f)=(∇gE[x|g])cov(g,g)⋅(∇gE[f|g])

Here, x represents some genetic trait, f represents the fitness of the organism, g represents the genes of an organism, and gcov represents the genetic covariance between the trait and the fitness. Usually people only use the ddtx=gcov(x,f) part of the equation[1], but I've written out the definition

gcov(a,b)=(∇gE[a|g])cov(g,g)⋅(∇gE[b|g])

because that will make the analogy to neural networks easier to see.

Neural network training and Price's equation

Suppose we train a neural network's weights w using the following equation, where L represents the loss for the network:

ddtw=−∇wL(w)

In that case, if we have some property x(w) of the network (e.g. x could represent how a classifier labels an image, or how an agent acts in a specific situation, or similar), then we can derive an equation for x's evolution over time:

ddtx=(∇wx(w))ddtw=−(∇wx(w))⋅(∇wL(w))

Similar to how we have a concept of genetic covariance to represent the covariance linked to genes, we should probably also introduce a covariance concept linked to neural network weights, to make it cleaner to talk about. I'll call that ntcov (short for neural tangent covariance), defined as:

ntcov(a,b)=(∇wa(w))⋅(∇wb(w))

Furthermore, to make it closer to being analogous, we might replace L with U=−L, yielding the following equation for predicting the evolution of any property x with training under gradient descent:

ddtx=ntcov(x,U)

This makes a bunch of idealistic assumptions about the training process, e.g. that we have an exact measure of the full gradient. It might be worth relaxing the math to more realistic assumptions, and check how much still applies. But for now, let's just charge ahead with the unrealistic assumptions.

Covariance niceties

Covariances play nicely with linear causal effects. If F and G are linear transformations, then cov(Fx,Gy)=Fcov(x,y)G⊤.

For instance, suppose you have a reinforcement learner that has learned to drink juice when close to it. Suppose further that now ...
Decisions: Ontologically Shifting to  Determinism
by Chris_Leong — 2022-12-21T12:41:30.884Z

In a previous post, I posited that a significant portion of the confusion surrounding counterfactuals can be traced back to an ontological shift. In this post, I will seek to elaborate on this argument and explain how we should reinterpret decision theory problems in light of this insight.

Our Naive Ontology Assumes Libertarian Free Will

Consider our naive ontology, by which I mean how humans tend to think without substantial philosophical reflection. In our naive ontology, we assume both that:

That we possess libertarian free will, ie. that our choices are not predetermined before we make them.
That when we're making a decision, both the past and present state of the universe are fixed independent of that decision.

Unfortunately, scientific evidence[1] seems to indicate:

3. At the point of a decision, each brain-state only corresponds to one possible choice[1]

It is impossible for all of these statements to be true simultaneously. If they were, there would be multiple possible choices we could make, each of which would necessitate a different brain-state. However, this would also require the current state of the universe, and therefore our brain state, to be fixed independent of these choices.

From my perspective, it is clear[2] that the mistake was in assuming the existence of libertarian free will (assumption 1). Given this, I'll take it for granted that we have to revise our naive conception of decisions so as to be rooted in a deterministic ontology.

Adapting to Determinism by Augmenting with Counterfactuals

Determinism dictates that whatever decision is ultimately made was the only decision that could have ever been made. This immediately poses a challenge: how do we avoid any decision from ending up as a trivial one?

The obvious answer is that we must find a way to augment the factual with a counterfactual for each option we desire beyond the first. But how should we construct this and what kind of object should it even be?

In Favour of Consistent Counterfactuals

As I previously argued in my post Counterfactuals are an Answer, Not a Question, the definition of a counterfactual will depend on the types of questions we seek to use them to answer. For example, while Causal Decision Theory results in counterfactual scenarios that are inconsistent, they can still be useful in practice, because the inconsistencies are often not relevant to ...
A Comprehensive Mechanistic Interpretability Explainer & Glossary
by Neel Nanda — 2022-12-21T12:35:08.589Z
Link: https://neelnanda.io/glossary

This is a linkpost for a very long doc defining, explaining, and giving intuitions and conceptual frameworks for all the concepts I think you should know about when engaging with mechanistic interpretability. If you find the UI annoying, there's an HTML version here

Why does this doc exist?

The goal of this doc is to be a comprehensive glossary and explainer for Mechanistic Interpretability (focusing on transformer language models), the field of studying how to reverse engineer neural networks.

There's a lot of complex terms and jargon in the field! And these are often scattered across various papers, which tend to be pretty well-written but not designed to be an introduction to the field as a whole. The goal of this doc is to resolve some research debt and strives to be a canonical source for explaining concepts in the field

I try to go beyond just being a reference that gives definitions, and to actually dig into how to think about a concept. Why does it matter? Why should you care about it? What are the subtle implications and traps to bear in mind? What is the underlying intuition, and how it fits into the rest of the field?

I also go outside pure mechanistic interpretability, and try to define what I see as the key terms in deep learning and in transformers, and how I think about them. If you want to reverse engineer a system, it's extremely useful to have a deep model of what's going on inside of it. What are the key components and moving parts, how do they fit together, and how could the model use them to express different algorithms?

How to read this doc?

The first intended way is to use this a reference. When reading papers, or otherwise exploring and learning about the field, coming here and looking up any terms and trying to understand them.

The second intended way is to treat this as a map to the field. My hope is that if you're new to the field, you can just read through this doc from the top, get introduced to the key ideas, and be able to dig into further sources when confused. And by the end of this, have a pretty good understanding of the key ideas, concepts and results!

It's obviously not practical to fully explain all concepts from scratch! Where possible, I link to sources that give a deeper explanation of an idea, or to learn more.

More generally, if something’s not in this glossary, you can often find something good by googling it o

...
Google Search loses to ChatGPT fair and square
by Shmi — 2022-12-21T08:11:43.287Z
Link: https://www.surgehq.ai//blog/googles-existential-threat-chatgpt-matches-googles-performance-on-informational-search-queries-and-smashes-it-on-coding

I mentioned before that Google search is less than satisfactory even on uncomplicated queries: https://www.lesswrong.com/posts/naAs59xiGfr7fPjej/google-search-as-a-washed-up-service-dog-i-halp

The linked article compares accuracy and usefulness of Google search with ChatGPT on some quite reasonable queries, and the latter seems to win hands down, even gimped by the lack of internet access (h/t https://twitter.com/emollick/status/1605420852635488258). I am not sure what to make of it, since Google has its own AI lab that is apparently at least as advanced and with better staffing and funding...
Sazen
by Duncan Sabien (Inactive) — 2022-12-21T07:54:51.415Z

Purpose of post: describe and (hopefully) popularize a concept I've found highly useful.

Last year, my partner Logan Strohl wrote a sequence to introduce the "naturalism" concept they've been developing and teaching for the past decade or so.

That sequence was structured around a single, short sentence. The first essay introduced the sentence, and the remaining essays were primarily about explaining what each of the important concepts in that short sentence actually meant.

So, for the sentence "knowing the territory takes direct and patient observation," there was a full essay on what was intended (and, more crucially, what was not intended) by the word "knowing," and another on "the territory," and another on "observation," and so on.

This format was largely inspired by a conversation in which I asked Logan to describe naturalism briefly, and they said "I totally can, but you'll get the wrong idea."

Together, we realized that there is a curious one-way sort of property to many sentences, in which they work as pointers or summaries after the fact, but fail to generate the-thing-they're-summarizing if used as standalone seeds.

(One could argue that every sentence has some of this property, but some sentences have a lot of it.)

I'd like to be able to point directly at this property, and as a result of historical accident that I'll explain in a footnote, the handle I've ended up with in my own head is sazen[1].

Example I: "Duncan Sabien is a teacher and a writer."

This is a true sentence. People who know me very, very well, upon hearing this sentence, will nod. It's a good fit, retrospectively, for the data.

However, if you are attempting to give someone a sense of me up-front, saying "Duncan Sabien is a teacher and a writer" is an unusually bad start. The thing that most people will think of when they hear "teacher" or "writer" is specifically unlike me—I'm a very weird sort of teacher and a very weird sort of writer, and so anchoring people on the representative stereotypes is almost actively misleading.

The sentence "Duncan Sabien is a teacher and a writer" is a sazen.

Example II: Peanut butter and jelly sandwiches

There's a classic challenge in which a teacher asks a bunch of students to write down unambiguous and complete instructions for how to make a peanut butter and jelly sandwich. The gimmick is that, to grade each paper, the teacher will follow only the actions w...
New AI risk intro from Vox [link post]
by JakubK — 2022-12-21T06:00:06.031Z
Link: https://www.vox.com/the-highlight/23447596/artificial-intelligence-agi-openai-gpt3-existential-risk-human-extinction

"AI experts are increasingly afraid of what they’re creating" by Kelsey Piper. It came out several weeks ago, but I only found it now and thought other people would want to see it. I think it's one of the best introductory articles on AI risk for people unfamiliar with AI (for more, see this doc). Here are my thoughts:

When I hover over the tab in my browser, a tooltip appears saying "A guide to why advanced AI could destroy the world - Vox." I think this text would be a better title than the current title.
Slightly misleading line: "College professors are tearing their hair out because AI text generators can now write essays as well as your typical undergraduate." The link describes how AI chatbots could, in theory, do college homework. But it doesn't show that "college professors are tearing their hair out."
There's a link to "Scientists Increasingly Can’t Explain How AI Works." I love how this title almost sounds like the headline for a satire piece, although I didn't find the corresponding article very convincing.
I really like following John Carmack's announcement of "AGI or bust, by way of Mad Science"
... with "This particular mad science might kill us all. Here’s why." It's difficult to be convey a sense of urgency and importance without sounding alarmist and annoying people, but I think this sequencing and phrasing does the job.
The section "Computers that can think" gives a short history of AI research and where it stands today, with an argument at the end for the importance of interpretability research.
There are links to "

Is Power-Seeking AI an Existential Risk?

" and "

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

." I don't think these are the best AI risk analyses to present to AI researchers (the recommendations in

Resources I sent to AI researchers about AI safety

are probably better since they rely less on philosophy-style argumentation), but this is fine since I assume the target audience is people unfamiliar with AI research.
I think Stuart Russell's "k < n" variables example flies over many people's heads if they're unfamiliar with AI or related fields (math, CS, stats, etc).
Perhaps unintentionally, the first paragraphs in the "Asleep at the wheel" section paint DeepMind as more safety-conscious than OpenAI, which might be false.
I think the statistic from the 2022 AI Impacts surve

...
Local Memes Against Geometric Rationality
by Scott Garrabrant — 2022-12-21T03:53:28.196Z

Earlier in the sequence, I presented the claim that humans are evolved to be naturally inclined towards geometric rationality over arithmetic rationality, and that around here, the local memes have moved us too far off this path. In this post, I will elaborate on that claim. I will argue for this claim very abstractly, but I think the case is far from sound, and some of the arguments will be rather weak.

Naturally Occurring Geometric Rationality

When arguing for Kelly betting, I emphasized respecting your epistemic subagents, and not descending into internal politics. There is a completely different, more standard, argument for Kelly betting, which is that given enough time Kelly bettors will almost certainly end up with all the money. It is thus natural to expect natural selection might select for geometric rationality. This hypothesis is backed up by the way people seem naturally risk averse. Further, people have a tendency towards probability matching, as discussed here, which is a bug that seems more natural if people are naturally inclined towards geometric rationality.

I think the hypothesis that evolution favors geometric rationality is probably best considered empirically, and I haven't gone deep into the data. If anyone else wants to dig into this, I would be curious to hear what they find.

This post, however, will focus on local memes that pull us away from geometric rationality. I will discuss four reasons that I think we are pulled away from geometric rationality: the end of the world, astronomical stakes, utility theory, and low probability of success in our main shared endeavor.

The Endgame

Kelly betting is usually justified by looking at long run outcomes. If you make enough bets, an expected wealth maximizer will almost certainly end up with no money. My fairness orientation to Kelly betting is a nonstandard one.

People around here tend to think the world is ending. If there are only a few rounds of betting left, it is hard to justify optimizing for long run outcomes.

Similarly, Thompson sampling is about exploration. If you don't have time to learn from new things, you should be in exploit mode, and just do the best thing.

Further, cooperative and fair equilibria are a lot easier to maintain in a repeated game. People actually often defect at the end of a finitely repeated prisoners' dilemma.

You don't actually need the end of everything t...
Logging Shell History in Zsh
by jefftk — 2022-12-21T03:30:03.180Z

By default most shells don't log your history in a detailed durable
way, which gives up one of the big advantages of working on the
command line. Good history lets you look back at things you did
months or years ago, in a searchable and skimmable fashion, so you can
answer questions like "how did I generate this number?", "what was
that trick I used?", or "if I'm doing something similar now what
should I recall from last time?"

I use bash as my shell and have long
used:

.bashrc:
promptFunc() {
echo "$(date +%Y-%m-%d--%H-%M-%S) $(hostname) $PWD $(history 1)" \
>> ~/.full_history
}
PROMPT_COMMAND=promptFunc

I was recommending this to folks at work, but they use
zsh. In zsh you can get a lot of the way
here by setting INC_APPEND_HISTORY (append to history
immediately instead of waiting for the shell to exit),
SAVEHIST=1000000000 (effectively don't limit the history
size on disk), and EXTENDED_HISTORY (to store timestamps
with history entries). But you risk losing most of your history if you
ever accidentally invoke zsh without these set, and it
doesn't write the directory you ran the command in (which is metadata
I reference a lot).

Mike tweaked my snippet to run in
zsh:

.zshrc:
precmd() {
echo "$(date +%Y-%m-%d--%H-%M-%S) $(hostname) $PWD $(history -1)" \
>> ~/.full_history
}

The two changes are that zsh uses precmd
instead of PROMPT_COMMAND, and that you need
history -1 instead of history 1.

You can still use the same histgrep command on both:

function histgrep {
local n_lines=10
if [[ "$1" =~ ^[0-9]*$ ]]; then
n_lines="$1"
shift
fi
grep "$@" ~/.full_history | tail -n "$n_lines"
}

Note that this doesn't replace your shell's built-in history tooling,
and I wouldn't recommend turning that off. This just a very cheap
additional layer of logging with additional metadata and less risk of
accidental deletion.