trentmkelly/LessWrong-43k huggingface dataset:
The Cage of the Language
by Martin Sustrik — 2022-04-13T05:20:54.319Z

Wittgenstein: Die Grenzen meiner Sprache bedeuten die Grenzen meiner Welt.

Passer-by: Where's that guy from? Is he Dutch?

I've never thought much of the Wittgenstein's argument. First, there's non-verbal thinking that can be used to escape the limits of the language. Second, once the knowledge is gained by non-verbal thinking, it can be used to extend the language itself and thus extend the limits of everyone's world.

However, I've been writing newspaper articles and a blog in Slovak lately and the experience made me think of the Wittgenstein quote.

Namely, I had hard time expressing some ideas and making some arguments. Slovak, you see, is a language of 5 million speakers, with ~200 years of history as a literary language. A lot of terms that one can use in English simply do not exist.

I can't say, for example:

I support policy X because it enables economies of scale and I am against policy Y because it introduces a single point of failure.

There are no widely accepted equivalents to "economies of scale" and "single point of failure".

I can say:

Som za X, pretože umožňuje úspory z rozsahu a som proti Y, pretože zavádza jediný bod zlyhania.

It's hard to relate how that sounds. Enough to say that "úspory z rozsahu" doesn't literally translate to "economies of scale" but rather to "savings from the extent". One gets an impression of a crackpot speaking gibberish, making a crackpot argument.

It should be said that this is not a problem in one-to-one conversations. One can use the English term if the other is aware of it, or, if not, he can clarify the meaning of the term in advance. Where the problem hits is the public discussion. There you can't expect the prior knowledge or afford extensive explanations. Attention span in public discussion is, after all, very limited. If you waste your one minute of public attention by explaining what a single point of failure is, you've already lost.

It can be argued that the problem lies not with the language, but rather with the language speakers: Slovaks are simple people, uneducated, they don't have the concept of economies of scale. Therefore, they should be educated and the term would force its way into the language all by itself.

But that, while true, is missing the point.

Most English speakers don't know what economies of scale are either. Yet, with 1.5 billion English speakers there's a large enough minority that does....
How dath ilan coordinates around solving alignment
by Thomas Kwa — 2022-04-13T04:22:25.643Z

Summary

Eliezer Yudkowsky created dath ilan in 2014 as a fictional world where Yudkowsky is the median person. Since then, he has written hundreds of pages of fiction in this setting, mostly "rational fiction", which means that Yudkowsky thinks most of the things he writes about in dath ilan would actually happen given the premise. This post is a series of quotes from various pieces of glowfic (a form of role-play fiction).
Dath ilan "successfully coordinated around maintaining a higher level of ability to solve coordination problems".
When the idea of AGI was first discovered "a few generations" ago, the world government of dath ilan took action to orient their entire civilization around solving the alignment problem, including getting 20% of their researchers to do safety research, and slowing the development of multiple major technologies. AGI safety research has been ongoing on dath ilan for generations.
The possibility of AGI and other x-risks is considered an infohazard of the highest importance. It was even kept from government representatives at first. At our current tech level, the average dath ilani has not heard about the possibility of AGI; even so, education gives dath ilani intuitions that might be useful for alignment research.
Dath ilan prediction markets that include Keepers (who know about AGI) say they have a 97% chance of reviving cryonically preserved people. Yudkowsky has confirmed[1] this implies dath ilan has a >97% chance of solving alignment.
Yudkowsky thinks Earth will probably fail at alignment. It's not clear whether the likely success of dath ilan is due to its higher average research ability, or its widespread coordination around civilizational priorities and infohazards, but I strongly suspect it's the latter. Being fiction, Yudkowsky's utopia is not the closest world to Earth where alignment is solved, but perhaps it can inspire us nonetheless.

The discovery of x-risk from AGI

This is dath ilan; and a few generations before Thellim's time, the highest of the Keepers[2] called together the Nine Legislators in a secret meeting. Shortly after, the highest Keeper and the Nine called an emergency closed assembly of the 324 Representatives.

And the highest Keeper said, with the Nine Legislators at the peak of dath ilan standing beside her, that the universe had proven to be a harsher and colder and more dangerous place th

...
What more compute does for brain-like models: response to Rohin
by Nathan Helm-Burger — 2022-04-13T03:40:34.031Z

This is a response to a comment made by Rohin Shah on Daniel Kokotajlo's post Fun with +12 OOMs of Compute. I started trying to answer some questions and assumptions he had, then realized there was more of an inferential gap that needed filling in. Also, as I attempted to estimate the OOMs of compute above GPT-3/PaLM needed for each method, I realized I was just going off of vague guesses rather than grounded estimates based on recent benchmarks. So, since other people might also be lacking the same info and be curious about my answer, I decided to put a bit more work into answering and turn it into a full post.

[Edit: 2 years later, and neuron level models continue to be very slow. https://www.frontiersin.org/articles/10.3389/fninf.2020.00016/full ]

Introducing the cast

First, I'd like to note that I don't endorse trying to get to AGI through any of these methods. I think they are potentially worse for interpretability in addition to being less compute efficient. My goal here is to point out that I think it could be done if the world were suddenly given lots more compute. In other words, I shall make the argument that given lots of compute, issues of limited data and potential scaling plateaus of artificial neural nets can be bypassed via other less compute efficient methods. Many roads lead to AGI, and specific predictions about the failure of one specific path (e.g. Transformers) don't necessarily mean all the paths are affected by that predicted failure mode.

The main contenders

Numenta - moderately detailed, most realistic-task proven. Designers made carefully chosen abstractions which may or may not be right.
Spiking Neural Network simulators (spikingNNs) - [I mentioned Nengo previously because its the one I'm most familiar with, but researching this showed me that there are other better performing options with similar abstraction levels such as BindsNet and brian2genn ] moderately detailed, moderately compute efficient, moderately task proven. Fewer abstractions than Numenta, more than Blue Brain. Less chance that an important detail was omitted, but still some chance.
Blue Brain - highly detailed, very compute inefficient, not task proven. Very few abstractions, so relatively high chance that it contains all necessary details for a functioning neocortex.

Supporting roles

The field of computational neuroscience has generated lots and lots of very narrowly focu...
“Fragility of Value” vs. LLMs
by Not Relevant — 2022-04-13T02:02:29.900Z

It seems like we keep getting LLMs that are better and better at getting the point of fairly abstract concepts (e.g. understanding jokes). As compute increases and their performance improves, it seems increasingly likely that human “values” are within the class of not-that-heavily-finetuned LLMs.

For example, if I prompted a GPT-5 model fine-tuned on lots of moral opinions about stuff: “[details of world], would a human say that was a more beautiful world than today, and why?” I… don’t think it’d do terribly?

The same goes for e.g. how the AI would answer the trolley problem. I’d guess it’d look roughly like humans’ responses: messy, slightly different depending on the circumstance, but not genuinely orthogonal to most humans’ values.

This is obviously vulnerable to adversarial examples or extreme OOD settings, but then robustness seems to be increasing with compute used, and we can do a decent job of OOD-catching.

Is there a modern reformulation of “fragility of value” that addresses this obvious situational improvement? Because as of now, the pure "Fragility of Value" thesis seems a little absurd (though I’d still believe a weaker version).
Commensurable Scientific Paradigms; or, computable induction
by samshap — 2022-04-13T00:01:22.820Z

Epistemic status - could use more rigor.

Thomas Kuhn's 1962 book "The Structure of Scientific Revolutions" (SSR) is a foundational work in the sociology and history of science.

SSR makes several important claims:

Scientific disciplines (e.g. astrophysics) undergo a cycle of paradigmatic 'normal science' and non-paradigmatic 'revolutionary science'.
During 'normal' periods, scientists in that field operate from a common scientific paradigm (e.g. Ptolemaic geocentric universe) - a common body of models, analysis techniques, and evidence-gathering tools.
As anomalies - discrepancies between observations and the paradigm - accrue, the paradigm enters a crisis. Visionary scientists start developing new paradigms (e.g. Copernican heliocentric universe), initiating the revolutionary period. Eventually one of these wins out, ushering in a new normal period.
(most controversially) These paradigms are incommensurable; there is no single metric by which the two paradigms can be judged as to which is a better map of reality. (He actually denotes multiple other types of incommensurability as well, but I find these far less convincing to anyone who has read the sequences).
In a later work Kuhn provided a set of five criteria that should be considered in judging the quality of a paradigm.

Like all models, Kuhn's is wrong. But is it useful? Many social scientists seem to think so: according to google scholar, SSR has been cited over 136 thousand times.

In this article/sequence, I will argue that despite issues with all these claims, paradigms are a useful construct. I will attempt to formalize the concept of a paradigm, within the context of Solomonoff induction, and demonstrate how these formalized paradigms can be used to aid in induction.

I will then demonstrate that this new formalism provides us with a metric that makes all paradigms commensurable, and that this metric (mostly) aligns with Kuhn's five criteria.

Stephen Toulmin had an eminently reasonable critique of SSR, namely that 'normal' science has plenty of smaller paradigm shifts that differ more in degree than in kind from Kuhn's 'revolutionary' shifts. The formalized paradigms agree with Toulmin here, as they are flexible enough to allow and infinitude of sub- and super-paradigms that allow for greater and lesser changes in scope.

Finally, I'll attempt to explain why this is important, both for understanding the cou...
Convincing People of Alignment with Street Epistemology
by Logan Riggs — 2022-04-12T23:43:57.873Z

Having a group of people who have experience convincing people of alignment helps us:

1. Convince people of alignment
2. Make explicit common reasons for skepticism and unwillingless
3. Coach people who want to have these conversations w/ their friends/colleagues/advisors/etc.

If you're not convinced of alignment, and would be willing to donate 1 hour of your time talking to me, I'd appreciate if you messaged me to schedule a call these next couple of weeks.

If you'd like to help convince people of alignment and build a curriculum w/ me, I'd also appreciate a message and schedule a call to chat.

What is Street Epistemology?

Street Epistemology is a set of tools that helps you have better conversations about difficult topics.

It's a way of getting to the core disagreements of a conversation, instead of talking in circles, past each other, and never getting anywhere.

Recently I had a conversation that went like this (note: details are vague, but this was the general flow of the conversation. They are a friend and had read Superintelligence but were not working in alignment):

L: What would convince you to start working on alignment?

C: There's not really anything actionable to solve the problem.

L: Have you looked at people's research agendas who work on this?

C: No.

L: Okay, say we have an actionable plan that you could work on, would you then want to work on alignment?

C: ... there's definitely a pay-aspect, so I couldn't take much of a pay cut than what I'm working on now (ie software engineer)

L: Okay, say there's an actionable plan and you have funding at the same pay as your current job, then would you want to work on alignment?

[Note that we could've gone off on tangents like explaining different people's research agendas or saying "if you truly believed AI could cause an existential risk, then a pay cut's not a big deal"]

C: I'm not sure if I'd be happy working on it. When I'm studying ML, there's a monotony to it.

L: I think you're the type of person who'd be happy working on this type of problem; it's not all ML. Though there are definitely parts that are a slog, but programming and learning is like that sometimes too.

C: Oh, I definitely can find joy in things that are slog, but things like this and global warming make me feel bad.

[asking to elaborate eventually yields]:

C: "...even if I made a perfect solution to alignment, it wouldn't change anything. You could h

...
Useful Vices for Wicked Problems
by HoldenKarnofsky — 2022-04-12T19:30:22.054Z

I've claimed that the best way to learn is by writing about important topics. (Examples I've worked on include: which charity to donate to, whether life has gotten better over time, whether civilization is declining, whether AI could make this the most important century of all time for humanity.)

But I've also said this can be "hard, taxing, exhausting and a bit of a mental health gauntlet," because:

When trying to write about these sorts of topics, I often find myself needing to constantly revise my goals, and there's no clear way to know whether I'm making progress. That is: trying to write about a topic that I'm learning about is generally a wicked problem.

I constantly find myself in situations like "I was trying to write up why I think X, but I realized that X isn't quite right, and now I don't know what to write." and "I either have to write something obvious and useless or look into a million more things to write something interesting." and "I'm a week past my self-imposed deadline, and it feels like I have a week to go, but maybe it's actually 12 weeks - that's what happened last time."

Overall, this is the kind of work where I can't seem to tell how progress is going, or stay on a schedule.

This post goes through some tips I've collected over the years for dealing with these sorts of challenges - both working on them myself, and working with teammates and seeing what works for them.

A lot of what matters for doing this sort of work is coming at it with open-mindedness, self-criticality, attention to detail, and other virtues. But a running theme of this work is that it can be deadly to approach with too much virtue: holding oneself to self-imposed deadlines, trying for too much rigor on every subtopic, and otherwise trying to do "Do everything right, as planned and on time" can drive a person nuts. So this post is focused on a less obvious aspect of what helps with wicked problems, which is useful vices - antidotes to the kind of thoroughness and conscientiousness that lead to unreachable standards, and make wicked problems impossible.

I've organized my tips under the following vices, borrowing from Larry Wall and extending his framework a bit:

Laziness. When some key question is hard to resolve, often the best move is to just ... not resolve it, and change the thesis of your writeup instead (and change how rigorous you're trying to

...
Does the rationalist community have a membership funnel?
by Alex_Altair — 2022-04-12T18:44:48.795Z

I was talking to someone the other day about the ways in which I've noticed the [Berkeley] rationalist community changing. One of the main ways was that group houses seemed to be disappearing. People were getting older, moving away, or just moving into their own houses to have kids. It then occurred to me that it doesn't seem like this is happening with the EA side of the community. Thinking about it more, it seems to me that EA has a quite strong funnel in the form of student groups. I semi-regularly hear about events, companies, projects, or just impressive people that are coming from EA student groups. Meanwhile I'm not even aware of a rationalist student group (although I'm sure there are some).

When I think about where rationalists came from, my answer is 1) EY writing the original sequences, and 2) EY writing HPMOR. It feels like those things happened, tons of people joined, and then they stopped happening, and people stopped joining. Then people got older, and now we have a population pyramid problem.

I think this is something of a problem for the mission of preventing AI x-risk. It is of course great to have lots of EAs around, but I think that people that the rationalist community would differentially appeal to would provide a lot of value that EA-learning people would be a lot less likely to provide (focus on AI, obsessive investigation into understanding confusing yet important subjects, etc.).

Do others agree with the pattern? Do you also see it as a problem? Any suggestions for what we could do about it? Why aren't there many rationalist student groups?
A Small Negative Result on Debate
by Sam Bowman — 2022-04-12T18:19:25.927Z

Some context for this new arXiv paper from my group at NYU:

We're working toward sandwiching experiments using our QuALITY long-document QA dataset, with reading time playing the role of the the expertise variable. Roughly: Is there some way to get humans to reliably answer hard reading-comprehensions questions about a ~5k-word text, without ever having the participants or any other annotators take the ~20 minutes that it would require to actually read the text.
This is an early writeup of some negative results. It's earlier in the project that I would usually write something like this up, but some authors had constraints that made it worthwhile, so I'm sharing what we have.
Here, we tried to find out if single-turn debate leads to reliable question answering: If we give people high-quality arguments for and against each (multiple-choice) answer choices, supported by pointers to key quotes in the source text, can they reliably answer the questions under a time limit?
We did this initial experiment in an oracle setting; We had (well-incentivized, skilled) humans write the arguments, rather than an LM. Given the limits of current LMs on long texts, we expect this to give us more information about whether this research direction is going anywhere.
It didn't really work: Our human annotators answered at the same low accuracy with and without the arguments. The selected pointers to key quotes did help a bit, though.
We're planning to keep pursuing the general strategy, with multi-turn debate—where debaters can rebut one another's arguments and evidence—as the immediate next step.
Overall, I take this as a very slight update in the direction that debate is difficult to use in practice as an alignment strategy. Slight enough that this probably shouldn't change your view of debate unless you were, for some reason, interested in this exact constrained/trivial application of it.
US Taxes: Adjust Withholding When Donating?
by jefftk — 2022-04-12T15:50:02.226Z

A friend recently asked:

I am considering increasing my donation amount to a 2-digit percentage
of income, and was wondering about the logistics of federal/state
income tax withholdings.

Do you have your employer reduce withholdings upfront? Was it simple
to have them do so? Or rather ask the IRS for a refund a year later?

This is a good question! When you start a job in the US, your
employer will generally withhold Federal and State taxes. Their goal
is that when you file taxes you will be close to neutral: neither
owing the government additional tax nor getting a large refund. Since
the amount of tax that you owe will depend on your individual tax
situation, however, your employer needs some information from you to
figure out how much to withhold, and they'll have you fill out Form W4
(pdf).

This used to be pretty awkward, with a system of "allowances", but a
few years ago the form was reworked and is much better. There is now
a section for "deductions", and you can put your planned donations
there. The IRS has an online
calculator that is pretty good.

Since you normally only fill this out when you start a new job, if
you're making a big change to your taxes, like a large increase in
donations, you should ask your employer for a new W4 to fill out.

If your state also has an income tax (most states) then it likely has
a different form, (ex: in MA it's Form
M-4, in CA it's DE
4), and unfortunately they generally still use an "allowances"
system. If your state doesn't allow you to deduct donations (ex: MA
and NJ) these forms are relatively easy to figure out because you just
put down your information like anyone else. If your state does (ex:
CA, NY), it probably has a much lower limit than the federal
government (table),
so when following the instructions only consider the amount that you
plan to donate that your state will count.

Overall, if you've done this correctly, when it's time to file taxes
next year you should be looking at a small bill/refund. If you are
off, however, as long as you adjust for next year you should be fine:
the rules for
underpayment are relatively friendly as the IRS goes.

Comment via: facebook