trentmkelly/LessWrong-43k huggingface dataset:
12 career-related questions that may (or may not) be helpful for people interested in alignment research
by Orpheus16 — 2022-12-12T22:36:21.936Z

Epistemic status: Some people tell me that these kinds of questions are helpful sometimes.

At EAGxBerkeley, people interested in alignment research often asked me for career advice. I am not a technical alignment researcher, but some people claim they find my style helpful (my “style” usually is heavy on open-ended questions, reflecting things back at people, and noticing things that people aren’t considering).

I noticed a few questions that came up frequently in my 1-1s. Here are some examples:

On Plans

What are your transformative AI timelines (and to what extent do your plans currently make sense given your timelines?)

Example: If you put a ~50% chance of TAI within the next 10-15 years, does your current plan let you contribute in time? If your current plan involves gaining credentials for 5-10 years, how much time does that leave you to contribute? Have you considered alternatives that involve spending a greater proportion of remaining time on direct work?

Forget about your current plan. Think about the goal you’re trying to achieve. If you work backwards from the goal, what are the most efficient ways to achieve it?

What steps will

actually

make you likely to achieve your goal

?
Is there any way you can replace your current plan with one that’s 10X more ambitious? What would a desirable tail outcome look like?
Is there any way you can take your current plan and achieve it 10X faster?
Are there any options outside of the “default path” that you might be neglecting to consider? Are there any options that are unconventional or atypical that you might want to consider more carefully? (e.g., taking a gap year, traveling to AIS hubs)

Note of course that sometimes the “default path” is actually the best option. I ask this question because I think people generally benefit from carefully considering options, and “weird” options are often the options that people have most neglected to consider.

Have you considered potential downsides of your current plan? Who might be able to help you identify more of them? How might you mitigate them?

On getting feedback

Who would you ideally get mentorship from?

Could you reach out to them and ask, “is there anything I can do (or produce) to help you evaluate whether or not I’d be a good fit for your mentorship?”

Who would you ideally get feedback from?

Could you reach out to them with a 1-3 page google doc that describes

...
Concept extrapolation for hypothesis generation
by Stuart_Armstrong — 2022-12-12T22:09:46.545Z

Posted initially on the Aligned AI website. Authored by Patrick Leask, Stuart Armstrong, and Rebecca Gorman.

There’s an apocryphal story about how vision systems were led astray when trying to classify tanks camouflaged in forests. A vision system was trained on images of tanks in forests on sunny days, and images of forests without tanks on overcast days.

To quote Neil Fraser:

In the 1980s, the Pentagon wanted to harness computer technology to make their tanks harder to attack…

The research team went out and took 100 photographs of tanks hiding behind trees, and then took 100 photographs of trees—with no tanks. They took half the photos from each group and put them in a vault for safe-keeping, then scanned the other half into their mainframe computer. [...] the neural net correctly identified each photo as either having a tank or not having one.

Independent testing: The Pentagon was very pleased with this, but a little bit suspicious. They commissioned another set of photos (half with tanks and half without) and scanned them into the computer and through the neural network. The results were completely random. For a long time nobody could figure out why. After all nobody understood how the neural had trained itself. Eventually someone noticed that in the original set of 200 photos, all the images with tanks had been taken on a cloudy day while all the images without tanks had been taken on a sunny day. The neural network had been asked to separate the two groups of photos and it had chosen the most obvious way to do it—not by looking for a camouflaged tank hiding behind a tree, but merely by looking at the color of the sky…

⁠“Neural Network Follies”⁠, Neil Fraser, September 1998

We made that story real. We collected images of tanks on bright days and forests on dark days to recreate the biased dataset described in the story. We then replicated the faulty neural net tank detector by fine tuning a CLIPViT image classification model on this dataset.

Below are 30 images taken from the training set ordered from left to right by decreasing class certainty. Like the apocryphal neural net, this one perfectly separates these images into tank and no-tank.

Figure 1: Trained classifier, labeled images

To replicate the Pentagon’s complaint, we then simulated the deployment of this classifier into the field with an unlabeled dataset of similar images, that doesn’t have...
Let’s go meta: Grammatical knowledge and self-referential sentences [ChatGPT]
by Bill Benzon — 2022-12-12T21:50:31.809Z

Language is a medium that humans use for sharing thoughts, attitudes, and feelings. In order to do so it has to be embedded in some physical medium, whether sound, visual signs, of physical gestures. But the thoughts, attitudes, and feelings are not in the physical medium, they are not attached to the physical tokens in the way that the handle is attached to the head a hammer. Rather, they are associated with the physical medium through very complex and diffuse circuit patterns in people’s brains.

But the fact that we embody those thoughts, attitudes, and feelings in a physical medium allows us to get “meta” to them, to treat them as objects. How/why? Because the physical medium is an object that we can perceive like any other physical object. We can perceive the speech stream like we perceive bird song, thunder, a babbling brook, or the sounds of a horse galloping on the plain. And so it is with writing and signing.

This gives rise to what Roman Jakobson termed the metalingual function of language, language about language. Metalanguage is enormously useful in various ways. On the one hand, for example, it allows to “debug” conversations in process, thereby improving communication. But it also allows us to bootstrap abstract concepts over strings of dialog, as David Hays has argued (in e.g. On “Alienation”) And to evolve ever more sophisticated conceptual systems over the long course of history (Benzon and Hays, The Evolution of Cognition).

With that in mind, I want to take a look at ChatGPT’s grasp of metalinguality, if you will. First, I ask it about Jakobson’s idea. Then I probe its ability to “think” about language.

The metalingual function

What’s going on here seems self-evident to me and is not surprising. The web has many texts about Jakobson and his ideas, so ChatGPT had material to work from.

Do you know about Roman Jakobson, the linguist?

Yes, Roman Jakobson was a Czech-born Russian and American linguist and literary theorist. He was one of the founders of the Moscow Linguistic Circle and a key figure in the development of structuralism and the structural analysis of language. Jakobson's work on language and communication had a significant influence on fields such as linguistics, semiotics, and communication studies.

What did Roman Jakobson mean by the metalingual function of language?

In his work on the functions of language, Jakobson proposed that there a

...
D&D.Sci December 2022 Evaluation and Ruleset
by abstractapplic — 2022-12-12T21:21:08.781Z

This is a followup to the D&D.Sci post I made ten days ago; if you haven’t already read it, you should do so now before spoiling yourself.

Here is the web interactive I built to let you evaluate your solution; below is an explanation of the rules used to generate the dataset (my full generation code is available here, in case you’re curious about details I omitted). You’ll probably want to test your answer before reading any further.

Ruleset

Snark Sub-Species

There are thirteen distinct types of Snark; three of these are Boojums. Typical characteristics for each sub-species (which are frequently deviated from; see my generation code for details) are summarized in the table below:

Name	Freq	Boojum?

Average

Waking-Time

Other Characteristics
Vorpal	19%	No	2:27pm

Hollow yet Crisp taste

Extreme Fondness

Moderate Cleanliness

Moderate Phobia

Frumious	7%	No	2:00pm

Crumbling yet Blunt taste

Mild/Moderate Fondness

Moderate Cleanliness

Extreme Phobia

Slythy	14%	No	4:20pm

Hollow/Artless taste

Crisp/Neat taste

Mild Everything

Mimsy	4%	No	4:10pm

Artless/Meagre taste

Bright taste

Moderate Everything

Manxome	4%	No	2:41pm

Hollow/Haunting taste

Blunt taste

Unusually specific sleep schedule

(Very!) Mild Fondness

Moderate Cleanliness

Extreme/Moderate Phobia

Whiffling	6%	No	1:39pm

Hollow yet Bright taste

Relatively specific sleep schedule

Extreme Fondness

(Very!) Mild Cleanliness

Moderate Phobia

Burbling	8%	No	4:33pm

Artless yet Crisp/Clear taste

Extreme Fondness

Moderate Phobia

Gyring	10%	No	3:22pm

Meagre yet Neat taste

Moderate Fondness

Moderate Cleanliness

Extreme Phobia

Gimbling	11%	No	3:23pm

Artless/Meagre taste

Extreme Fondness

Moderate Cleanliness

Mild Phobia

Cromulent	5%	Yes	4:01pm

Hollow/Crumbling taste

Blunt taste

Mild Cleanliness

Phobia almost never Moderate

Snippid	2%	Yes	2:44pm

Meagre/Haunting taste

Clear taste

Moderate Fondness

Moderate/Extreme Phobia

Scrumbling	3%	Yes	4:22pm

Crumbling yet Blunt taste

Mild Fondness

Moderate/Mild Cleanliness

Moderate/Mild Phobia

Which Snarks are Hunted?

2% of all sighted Snarks are left unhunted due to logistical problems or gluts of potential targets.

Conventional wisdom in the Snark-hunting community is that Snarks with a Taste containing “Crumbling” and (to a lesser extent) “Blunt” are much more likely to be Boojums, and so should not be hunted. Six-sevenths of Snark-hunters follow this advice regarding “Blunt”, and everyone follows it regarding “Crumbling”.

Strategy

The risk associated with each Snark in the list is as follows:

Snark	R

...
Log-odds are better than Probabilities
by Robert_AIZI — 2022-12-12T20:10:29.968Z
Link: https://aizi.substack.com/p/log-odds-are-better-than-probabilities

[This is a cross-post from my blog at aizi.substack.com. I'm sure someone has made a point like this before, but I don't know any specific instances and I wanted to give my take on it.]

At my previous job I worked on ML classifiers, and I learned a useful alternative way to think about probabilities which I want to share. I’m referring to

log-odds aka logits

, where a probability p is represented by logit(p):=log(p/(1-p))[1].

I claim that, at least for Bayesian updates and binary prediction, it can be better to think in terms of log-odds than probabilities, and this post is laying out that case.

Log-odds simplifies Bayesian calculations

Do you do

Bayesian updates

in your head? I didn’t, in part because the classic Bayes formula is kinda bad to work with:

\[P(H|E)= \frac{P(H)P(E|H)}{P(E)}\]

The first problem is that you need to know P(E), the chance that E is true at all. But the value of P(E) should be irrelevant since we know we live in a timeline where E is true! Of course you can use a formula like this to hide P(E) but at a complexity cost:

\[P(H|E)=\frac{P(H)P(E|H)}{P(E|H)P(H)+P(E \neg H)P(\neg H)}\]

For me, this calculation requires too many operations and cached numbers to do easily in my head.

But more importantly, these formula don’t emphasize how P(H) was updated. Sure, you can say P(H) is being multiplied by P(E|H)/P(E), but that number isn’t really comparable across priors. For instance, if P(E|H)/P(E)=2, that’s a small update if your prior is P(H)=.1 (taking you from 10% to 20%), a huge update if P(H)=.5 (taking you from a coinflip to certainty), and impossible for P(H)>.5. So “P(E|H)/P(E)=2” isn’t a meaningful intermediary calculation.

Now let’s compare the log-odds version. I’ll write L(H) for logit(P(H)):

\[\begin{eqnarray*} L(H|E) &=& \log\left( \frac{P(H|E)}{P(\neg H | E)}\right)\\ &=& \log\left( \frac{\left(\frac{P(H)P(E|H)}{\cancel{P(E)}} \right)}{\left( \frac{P(\neg H)P(E| \neg H)}{\cancel{P(E)}}\right)}\right)\\ &=&\log(\left(\frac{P(H) P(E|H)}{P(\neg H)P(E|\neg H)} \right)\\ &=& \log \left( \frac{P(H)}{P(\neg H)} \right)+\log \left( \frac{P(E|H)}{P(E|\neg H)} \right) \\ &=& L(H)+ \log \left( \frac{P(E|H)}{P(E|\neg H)} \right) \end{eqnarray*}\]

Omitting intermediate steps:

\[\begin{eqnarray*} L(H|E) &=& L(H)+ \log \left( \frac{P(E|H)}{P(E|\neg H)} \right) \end{eqnarray*}\]

Now that’s clear! A Ba...
Psychological Disorders and Problems
by adamShimi — 2022-12-12T18:15:49.333Z

There are two types of psychological issues: disorders and problems.

Disorders are so massive, so life-altering that we can’t miss them: schizophrenia, crippling social anxiety, destructive addiction… They scream “danger” in neon red blinking letters. We know and fear them because they destroy lives. People with such disorders are often unable to function in society, and see themselves ostracized, shunned, and institutionalized.

On the other hand, psychological problems appear quite benign and harmless: fear of rejection, not liking social gatherings, stress eating/smoking… They’re not good, but they’re also not that big of a deal. Everybody has them, and most people function just fine despite a bunch of psychological problems. They don’t forbid a normal life.

Yet they subtly restrict you.

Take low social anxiety. Maybe you don’t really like meeting new people. Maybe you don’t like group settings. Or you’re uncomfortable talking with others, because you might say something stupid. Whatever your reason, this anxiety makes you:

Avoid parties if you can
Stay less time if you have to go
Be less enjoyable in conversations when there

None of these consequences will ruin your life. Even when put all together, psychological problems still don’t amount to disorders. But as soon as you

start to optimize

, they get in your way. You start bending the best action that you can find in order to deal with your problems, instead of going for it directly. If networking is the best use of your time but you have social anxiety, you’ll find alternatives that don’t reap all the benefits, often without even realizing it. You’ll just flinch away.

Psychological disorders are obvious liabilities. But if you neglect psychological problems, they will invisibly block your path again and again. Avoiding this doesn’t look like waiting until all your psychological problems are solved, but instead to acknowledge them and endeavour to fix them when they get in the way.
Confusing the goal and the path
by adamShimi — 2022-12-12T16:42:40.508Z
Link: https://epistemologicalvigilance.substack.com/p/confusing-the-goal-and-the-path

Say you are working on formulating a new scientific theory. You’re not there yet, but you broadly know what you want: a simple theory that powerfully compresses the core phenomenon, and suggests a myriad of new insights.

If you’re anything like me, at least part of you now pushes for focusing on simplicity from the get go. Let’s aim for the simplest description that comes easily, and iterate from that.

Did you catch the jump?

I started with a constraint on the goal — a simple theory — and automatically transmuted it into a constraint on the path — simple intermediary steps.

I confused “Finding a simple theory” with “Finding a simple theory simply”.

After first uncovering this in my own reasoning, I now see this pattern crop everywhere:

(Well-calibration) I want to be well-calibrated. And it feels like I should thus ensure to only make judgments and claims I’m sure are well-calibrated. Yet if I do that, I’ll end up protecting the parts of my model where the confusion and the uncertainty lies, and starving my brain of updates.
(Coherence) Another valuable property of mature scientific theories is coherence. And it feels like I should thus ensure that all my intermediary theories are coherent. Yet many examples in the history of science show that incoherent frames and theories can prove key in generating coherent and useful ones: Newton’s early calculus, Heaviside’s operational calculus, Kekule’s structure of benzene, Bohr’s atom, Dirac’s delta).
(Winning) In competitive games (sports, board games, esports), the goal is to win. And it feels like becoming a winner means consistently winning. Yet if I aim to only win, the best course of action avoids stronger opponents and the valuable lessons they can explicitly and implicitly teach.

In the language of moral philosophy, this pattern amounts to adding a deontological spin on a consequentialist value. Even though it can prove valuable (in AI alignment for example), doing this unconsciously and automatically leads to problems after problems.

For it conjures obstacles that were never there.

This post is part of the work done at Conjecture.

Thanks to Clem for giving me the words to express this adequately.
Meaningful things are those the universe possesses a semantics for
by Abhimanyu Pallavi Sudhir — 2022-12-12T16:03:31.413Z

Logic courses are very risk-averse to talking about philosophy. This is bad, because philosophy is the motivation/intuition for logic & TCS. This essay gives a scout's view.

Epistemic status: A computer scientist would nod through Chapters 2-5, at least after first looking at the logician for approval. Subsequent chapters take a philosophical stance, namely that expressed in the title.

Contents

Motivation and a cringey rant [skip for the technical meat]
Gödel's first incompleteness theorem
Gödel's second incompleteness theorem
Semantics and truth
Exercises
Reflection and ordinals
Chaitin and complexity
Empiricism and the Löbstacle

[A rough version of Chs 1-5 post initially appeared as a math stackexchange answer and on my blog; I wrote it as I learned, and am posting more refined writing here.]

Motivation and a cringey rant [skip for the technical meat]

There are several immediate philosophical questions a beginner may ask when first hearing of Gödel's incompleteness theorem, such as:

Does Gödel's theorem apply to humans? If so, how are we able to tell that the Gödel sentence is true, or that some program doesn't halt?
Does Gödel's theorem imply that there are questions about the universe, or about mathematics, whose answers we can never learn? Are these questions at least comprehensible to us?
What's with the suggestive analogy between logic and computation, and how can we make it more precise? If Gödel applies to humans, are we formal systems or programs?
If "true" is different from "provable", what does "true" even mean?
How do we know that (e.g.) ZFC is consistent? We keep talking about the "true" model, how do we even identify this true model?

Unlike other areas of science that lay-people find weird and philosophers say absurd things about (such as relativity and quantum mechanics), these questions are honestly legitimate and relevant for AI work (the only such question that isn't legitimate is "Does Gödel preclude a TOE?").

Some very wrong ways to answer these questions/to introduce mathematical logic:

Gödel's theorem applies to specific types of formal systems, and says nothing about human reasoning. It just means we can't apply an algorithm to answer every question, not that we can't do so by other, more creative means! [Dude, do something non-algorithmic and show me.]
Gödel's theorem is no biggie! It's just like how commutativity is independent of t

...
Tradeoffs in complexity, abstraction, and generality
by remember — 2022-12-12T15:55:19.064Z

Complexity, abstraction, and generalization are powerful:

Complexity lets us explain a wider range of phenomena
Abstraction allows us to reason more easily by filtering unnecessary noise or details
Generalization lets us reason over larger domains.

And yet, complexity, abstraction, and generality have costs, which creates a tradeoff between these benefits that they bring and the costs we incur.

Adding complexity makes our models more expressive by adding more details to the models. The extra detail added to the model makes it harder to understand, potentially making interventions harder to perform or the effects of interventions harder to understand. At times, we need our models to capture more of reality, but if we only care about narrower descriptions, the tradeoff may not be worth it.

For example, when programming, adding code to handle lots of different cases can increase the scope of your program but also makes it harder to understand. More complex code also increases the difficulty of making changes and the odds a change will break something elsewhere. Laws and contracts are another good example of highly complex systems that both have high costs and situationally provide large benefits. By adding complexity to laws by specifying potential legal outcomes in many different circumstances, the law can better protect against arbitrary interpretation and increase confidence in its judgments. The complexity of law comes at a cost, though: it can be quite difficult for people without training or time to learn to understand all the intricacies.

Abstraction moves our models from concrete details to abstract concepts. Abstracting trades contact with reality for ideas that can be much easier to work with. Reality can be too detailed, too messy to handle well, which demands that we abstract. But if we abstract too much, we risk losing contact with reality. This can lead to our abstractions no longer tracking reality and to our abstractions no longer capturing important details about reality.

An example of this is how we use and model technology in our daily lives. Often, we don’t need to understand the underlying tech to use it, but abstraction can obfuscate important details. People who have a very abstract model of technology can struggle to use it in practice. Still, depending on their needs, the ease of modeling objects they interact with may outweigh the inability to...
Green Line Extension Opening Dates
by jefftk — 2022-12-12T14:40:03.429Z

Talk of extending the Green Line from Lechmere to Ball Square and
Tufts goes back

a very
long way, but it's finally open!

In celebration, I wanted to go back and collect the completion dates
we've been given over the years:

1990-12-19:
2010

1994-10-04:
2011-12-31

2006-11-17:
2014-12-31

2010-07-10:
2015-10

2011-08-02:
2018 (Fall) to 2020

2013-07-29:
2019 or 2020.

2016-12-08: 2021

2021-06-21: 2022-05

2022-02-24:
2022 (Late Summer)

2022-08-05:
2022-11 (Late in the month)

2022-11-17: 2022-12-12

(Thanks to u/Master_Dogs on reddit for helping
me compile these.)