trentmkelly/LessWrong-43k huggingface dataset:
Baby Nicknames
by jefftk — 2021-12-13T02:20:01.861Z

Lily and Anna rarely call Nora by her name. Instead, it's always
nicknames, with lots of variety. For a week starting on Thanksgiving
I counted how often they used different terms:

21
Norkit (Anna favorite)

20
Nwetzel (Lily favorite)

16
Norky

9
Cuddleezel

7
Norka

7
Nori

4
Norit

4
Cuddleitz (former favorite)

3
Norweasel

3
Norklet

2
Norkla

2
Cuddlet

Terms I only recorded once: Cuddleez, Cuddlehat, Cuddleoo, NW,
Norahats, Noranoo (former
favorite), Norba, Norcuddly, Norfi, Norfkit, Norflea, Norfo, Norfu,
Norito, Norkakit, Norkaloo, Norkanoo, Norkita, Norkito, Norko,
Norkwinza, Norkwitz, Norquo.

I counted each term once per "session", where if they said "Norkit
Norkit Notkit" I would still only count that once, while if an hour
later they said "market" again I would give it another tally. Doesn't
include any sessions where I had my hands full or otherwise wasn't
able to easily tally.

It took me a while to get around to posting this, during which both
kids, but especially Lily, have started using "Kaia" a lot. Asking
Lily, she says it is "short for Cuddler".

Comment via: facebook
Why do governments refer to existential risks primarily in terms of national security?
by Evan_Gaensbauer — 2021-12-13T01:05:29.722Z

I learned from the public policy section from the Wikipedia article about the AI control problem that earlier this year the UK government published its ten-year National AI Strategy. For specifically the control problem and x-risks from AGI, the strategy report reads:

The government takes the long term risk of non-aligned Artificial General Intelligence, and the unforeseeable changes that it would mean for the UK and the world, seriously.

That's an improvement from a few years ago when some world governments declared they're prioritizing the control problem only in terms of national security. On its face, that's of course ludicrous because prioritizing only national security misrepresents the scope of x-risk from AI. If national governments understand the control problem, they understand that it's pointless in a literal sense to minimize their focus to only national security.

Yet so far they've still mainly expressed their focus on the control problem in terms of national security. One obvious reason why is that there may be a nationalist tendency to a government's ideology that provokes them to add a boilerplate about national security or self-interest to everything without reflection. Another obvious reason is that governments are trying to send social signals. Yet I'm asking this question to check what more precise or less intuitive answers I might be missing.

A related question is: What might have changed in the last few years that has a national government express concern about the control problem in terms of global security too?

The National AI Strategy report from the UK is the first time I've seen a government express concerns about the control problem in terms of global security. This could be important because it means governments may also be trying to signal they're more willing to coordinate with other governments on the control problem. Understanding what motivated one government to signal they're more multilateralist than before could be applied to motivate other governments to do the same.
[Resolved] Who else prefers "AI alignment" to "AI safety?"
by Evan_Gaensbauer — 2021-12-13T00:35:12.215Z

Edit:

User davidad wrote a comprehensive overview of other actors in the field who have begun using AI alignment instead of AI safety as the standard terminology to refer to the control problem. It seems this is a general or growing preference in the field that isn't a complete consensus. That may only be because of inertia from several years ago before AI alignment and the control problem were as distinguished as well in the field. Sometimes the term control problem is simply used instead of either of the other terms.
I originally characterized the control/alignment problem as synonymous with any x-risks from AI. User antimonyanthony clarified the control problem is not the only way AI may pose an existential risk. I've edited this post accordingly.

During conversations about x-risks from AI among a community broader than the rationality or x-risk communities, such as in effective altruism or social media, I've seen Eliezer Yudkowsky and Ben Pace clarify that the preferred term to refer to the control problem is "AI alignment." I understand this is to distinguish other ethical and security concerns about AI, which is what "AI safety" has come to mean, from specifically existential risks from AI. Yet I've only seen those involved in x-risk work coming from the rationality community saying this is the preferred term. That main reason for that might be that maybe the majority of people I know working on anything that could be called either AI alignment or AI safety are also in the rationality community.
Is there any social cluster in the professional/academic/whatever AI communities other than the x-risk reduction cluster around the rationality community who prefers this terminology?
Working through D&D.Sci, problem 1
by Pablo Repetto — 2021-12-12T23:10:39.553Z
Link: https://pabloernesto.github.io/2021/12/11/dndsci-1.html

A while back a series of statistics/data analysis problems loosely inspired by Dungeons and Dragons were posted on LessWrong under the title D&D.Sci. I'm working through them while avoiding spoilers and posting my work in advance of seeing the solutions.

The first post in the series is my stab at problem 1. I still haven't seen the published solution, but I'll writeup my thoughts when I do.

I have very little knowledge or experience with statistics, so I'm going to be doing lots of just-in-time learning.
Teaser: Hard-coding Transformer Models
by MadHatter — 2021-12-12T22:04:53.092Z

Transformer models are incredibly powerful for natural language tasks (and they are starting to find uses in many other fields of machine learning). Unfortunately, it is nigh-impossible to interpret what goes on inside them. OR IS IT???

In this post, I am trying to gauge potential community interest in a strand of research that I have been doing in my spare time off and on for the past year and a half (roughly).

I have found that I can, with a fair amount of effort, hard-code the weights of a transformer model in order to perform some very crude versions of linguistic tasks. So far I have achieved English-to-French translation (on a toy corpus of about 150 sentences), text classification (is a sentence grammatical or not? on a toy corpus of a couple hundred sentences), and sentiment analysis (again on a limited corpus). These results are obviously not impressive compared to the state of the machine learning field, but I am pretty sure that they can all be drastically scaled up with the investment of some time and energy. Unfortunately, I have a fairly demanding day job, and haven't found the time and energy yet.

All of this is done by inspection (no gradient descent!). The process is a lot like programming, although it is more difficult than programming, at least right now for me. I am fairly certain that better tools and better notation can be developed to make the process easier. It is also almost certainly possible to combine hard-coding with gradient descent approaches to be able to scale these methods up in a slightly less labor-intensive way.

I think that these ideas could prove useful in alignment research - if we understand how a language model works in excruciating detail, it seems drastically more likely that we will be able to reason about and predict various misunderstandings rooted in the ambiguity of language. Given that language is (arguably) a fully general means of interacting with an artificial intelligence, it seems plausible to me that this work is on the critical path to alignment.
Redwood's Technique-Focused Epistemic Strategy
by adamShimi — 2021-12-12T16:36:22.666Z

Imagine you’re part of a team of ML engineers and research scientists, and you want to help with alignment. Everyone is ready to jump in the fray; there’s only one problem — how are you supposed to do applied research when you don’t really know how AGI will be built, what it will look like, not even the architecture or something like that? What you have is the current state of ML, and a lot of conceptual and theoretical arguments.

You’re in dire need of a bridge (

an epistemic strategy

) between the experiments you could run now and the knowledge that will serve for solving alignment.

Redwood Research

is in this precise situation. And they have a bridge in mind. Hence this post, where I write down my interpretation of their approach, based on conversations with Redwood’s

Buck Shlegeris

. As such, even if I’m describing Redwood’s strategy, it’s probably biased towards what sounds most relevant to both Buck and me.

Thanks to Buck Shlegeris for great discussions and feedback. Thanks to Nate Thomas for listening to me when we talked about epistemic strategies and for pushing me to send him and Buck my draft, starting this whole collaboration. Thanks to Seraphina Nix for feedback on a draft of this post.

Techniques, not Tasks or Issues

My first intuition, when thinking about such a bridge between modern experiments and useful practical knowledge for alignment, is to focus on tasks and/or issues. By task, I mean the sort of things we want an aligned AGI to do (“learn what killing means and not do it” might be an example), whereas issues are… issues with an AGI (deception for example). It sounded obvious to me that you start from one of those, and then try to make a simpler, analogous version you can solve with modern technology— the trick being how to justify your analogy.

This is not at all how Buck sees it. After my initial confusion wore off, I realized he thinks in terms of techniques: potential ways of aligning an AGI. If tasks are “what we want” and issues are “what might go wrong”, techniques focus on “how” — how we solve the task and avoid the issues.

Buck’s favorite example of a technique (and the one driving Redwood’s current work) is a form of

adversarial training

where the main model receives adversarial examples from a trained adversary, and has its response judged by an overseer to see if they’re acceptable or not.

Another example is

Debate

: the alignment...
For and Against Lotteries in Elite University Admissions
by Sam Enright — 2021-12-12T13:41:04.868Z

People (whose names often rhyme with Palcolm Sadwell) sometimes suggest replacing the admissions systems of highly selective universities with lotteries. The proposal is that universities would mark a pool of students as ‘good enough’ and then students from that pool would be accepted at random. Here are some arguments for and against this idea, inspired by Julia Galef’s unpopular ideas series. I don’t agree with all these arguments (or even fully endorse any); the point here is just to categorise the arguments worth considering on both sides.

For:

Competitive university admissions generate enormous amounts of waste (through Goodhart’s law) with time and money being spent on extracurriculars and private tutoring for what is essentially a zero-sum game. It may also be incentivising unethical behaviour: the proportion of students who admitted to cheating in high school went from 34% in 1969 to 74% in 2002.
Current admissions are racist and classist, especially when they rely on interviews, because admissions officers are unconsciously biased.
Noise is systematically much higher than people suspect so the system is already, to a moderate extent, random.
With a lottery, admitted students would be from more racially and socioeconomically diverse backgrounds. Different countries and universities have differing levels of remedial discrimination policies, so a lottery would make some colleges less diverse, but the net effect would be to make them more diverse.
The current system penalises students who spend their teen years doing things they actually enjoy. There’s adverse selection for people who are unoriginal, uncreative, and place little value on their time.
Students applying to top universities are very stressed out, e.g. 46% of Cambridge students either have depression or suspect they might. Easing the competition would help with mental health problems and lead them to put less pressure on themselves.
If getting into top universities is more arbitrary, it will cease to be a reliable signal of competence to employers. Therefore, the labour market will rely more on non-credential signals of competence, bursting the higher education bubble.
Lotteries will make universities more intellectually diverse. Conservatives are highly underrepresented in top universities, and (at least in America) this is in part because of ideological admissions officers that discriminate against

...
Nuclear war anthropics
by smountjoy — 2021-12-12T04:54:10.637Z

In the most recent episode of Rationally Speaking ("Humanity on the precipice", 10 Dec. 2021), Julia Galef and Toby Ord discuss the risk of a nuclear exchange during the Cold War:

J.G. [55:28]: "If you had asked me to put a probability on each case, like taking nuclear near misses—for example, like the Cuban Missile Crisis or the time the Russians could have retaliated against us if not for Arkhipov—I would have put a probability of maybe 25% on a lot of those. But it starts to add up. If you take all of the near misses that I would've assigned 25% to, they add up enough that it starts to look really surprising that we didn't end up with a nuclear war, which kind of throws into question my ability to assign good probabilities to all of these near misses."

T.O. [55:02]: "You're right that if your attempts to do this are producing very large numbers, then that does suggest that the attempts might be going wrong, and I think it's very easy to have them go wrong; I think it's an extremely difficult thing, and I think there's been very little written, really, about how to do this kind of retrospective prediction or trying to assign probabilities to past events where we actually know what happened.

If anthropic effects were distorting the historical record, this is exactly what you'd expect to see: Julia's probabilities on nuclear exchange would seem reasonable individually but too high in aggregate. Why does neither of them mention the anthropics explanation? Is it somehow common knowledge that this explanation doesn't hold water?

The Precipice has a brief section on anthropics (part of Chapter 7), in which Toby says, "From what we know, it doesn't look like [anthropic] selection effects have distorted the historical record much, but there are only a handful of papers on the topic and some of the methodological issues have yet to be resolved." The only nearby citation is this paper by Cirkovic, Sandberg & Bostrom, who seem to conclude that selection effects from anthropogenic hazards are difficult to determine; they don't make any argument that those effects are small.

Also related: https://www.lesswrong.com/posts/EkmeEAB646Yf2DNzW/anthropics-doesn-t-explain-why-the-cold-war-stayed-cold
Some abstract, non-technical reasons to be non-maximally-pessimistic about AI alignment
by Rob Bensinger — 2021-12-12T02:08:08.798Z

I basically agree with Eliezer’s picture of things in the

AGI interventions post

.

But I’ve seen some readers rounding off Eliezer’s ‘the situation looks very dire’-ish statements to ‘the situation is hopeless’, and ‘solving alignment still looks to me like our best shot at a good future, but so far we’ve made very little progress, we aren’t anywhere near on track to solve the problem, and it isn’t clear what the best path forward is’-ish statements to ‘let’s give up on alignment’.

It’s hard to give a technical argument for ‘alignment isn’t doomed’, because I don’t know how to do alignment (and, to the best of my knowledge in December 2021, no one else does either). But I can give some of the more abstract reasons I think that.

I feel sort of wary of sharing a ‘reasons to be less pessimistic’ list, because it’s blatantly filtered evidence, it makes it easy to overcorrect, etc. In my experience, people tend to be way too eager to classify problems as either ‘easy’ or ‘impossible’; just adding more evidence may cause people to bounce back and forth between the two rather than planting a flag in the middle ground.

I did write a version of 'reasons not to be maximally pessimistic' for a few friends in 2018. I’m warily fine with sharing that below, with the caveats ‘holy shit is this ever filtered evidence!’ and ‘these are my own not-MIRI-vetted personal thoughts’. And 'this is a casual thing I jotted down for friends in 2018'.

Today, I would add some points (e.g., 'AGI may be surprisingly far off; timelines are hard to predict'), and I'd remove others (e.g., 'Nate and Eliezer feel pretty good about MIRI's current research'). Also, since the list is both qualitative and one-sided, it doesn’t reflect the fact that I’m quantitatively a bit more pessimistic now than I was in 2018.

Lo:

[...S]ome of the main reasons I'm not extremely pessimistic about artificial general intelligence outcomes.

(Warning: one-sided lists of considerations can obviously be epistemically bad. I mostly mean to correct for the fact that I see a lot of rationalists who strike me as overly pessimistic about AGI outcomes. Also, I don't try to argue for most of these points in any detail; I'm just trying to share my own views for others' stack.)

1. AGI alignment is just a technical problem, and humanity actually has a remarkably good record when it comes to solving technical problems. It's historica...
Magna Alta Doctrina
by jacob_cannell — 2021-12-11T21:54:36.192Z

Continuing topics from an earlier 2015 blog post, The Brain as a Universal Learning Machine, this new series will begin with updated distillations of deep learning's best learnings, then interweave brain (reverse)engineering, advanced simulation tech, crypto-economic mechanism design, and more, interject a bit of novelty, then future project those intertwined threads into an ambitious systemic proto-plan for a new research economy to build advanced sims and aligned AGI. The telic ambition of this sequence - on it's best boldest days - is to evolve into that plan; or if failing that, at least engage as futurism.

This sequence will explore answers to three intertwined key questions:

How can we build powerful scalable predictive world models?
How can we efficiently plan/steer/optimize the future using said models?
How can we best ensure said powerful optimization is aligned to our values?

These problems and their solutions are not wholly distinct, but rather deeply interconnect in a fractal recurrence. But before exploring that, first we must establish common knowledge of the foundations, and before that: a quick reckoning.

Introduction: a Reckoning

“Having been trained as a computer scientist in the 90s, everybody knew that AI didn’t work. People tried it, they tried neural nets and none of it worked.”

-Sergey Brin, Davos 2017

Few foresaw the Connectivist's sudden sweeping victory. Why?

In this post I will reveal and explore the key foundations of Deep Learning's success. And reveal is the correct word; some of these insights are rare even within the field. I will argue that these insights predict/postdict the success of DL, and towards the end I will venture further to argue that any practical AGI will necessarily be built on a DL-like foundation (regardless of map nomenclature), and perhaps stake on that with some novel predictions.

In a popular AI textbook from a bit before the DL revolution (the Norvig/Russell one), I recall a chapter discussing Artificial Neural Networks as a technique near a chapter on Evolutionary Computation, perhaps nearby chapters labeled Bayesian Inference or Decision Trees, and all the other separated magisteria. The chapter on ANNs made sure to point out how they only did marginally better than SVMs (support vector machines) on key benchmarks like MNIST at the time, despite being far less theoretically sound (which should only make them more...
EA Dinner Covid Logistics
by jefftk — 2021-12-11T21:50:02.587Z

Yesterday we hosted an

EA dinner,
the first largeish (~22 people) gathering inside our house since
before the pandemic. It went well, and it was really good to see
people. I wanted to write a bit about what sort of precautions we
took for covid.

Open window in the main area with a fan pointing out. Windows partly
open in other parts of the house, trying to balance the airflow so as
much air as possible traveled past the guests. I preheated the house
earlier in the day, and during the event had the heat on maximum. It
was a bit chilly near the end, but still tolerable.

Air purifier cubes in each room.

In the main area, which isn't a good fit for one, I made a bit of a
tunnel above the bookcase.

Masks on, except when eating. People were good about this; I
didn't see people hanging out with mask off and a plate but not
eating.

Rapid testing at the door ($7/person).
We lined up a volunteer to handle this, because Julia was dealing with
hosting and I was watching the kids upstairs. With each test they
wrote down the person's name, the time where it needed to be read and
their cellphone number. When each test had been sitting for 15
minutes, we texted the person the result. We asked people not to take
their mask off to eat until they had their results. If any tests had
been positive, we would have informed the group. False positive rate
is ~0.05%. We
are keeping the cell phone numbers for a few days, so we will be able
to text everyone if anyone later notifies us they have it.

We told people vaccination was required, though not boosters. We
intended to check vaccination cards (accepting photos) at the door,
but forgot. If I was organizing some thing like this a few weeks from
now, when there has been more time for people to get boosters, I might
require them.

We considered requiring masks to be surgical or better (offering
free ones) but decided not to. I think this probably would have
been worth doing, except that we added several of these precautions
after announcing the gathering, and it felt like too much to add
another one.

Occasional crowd management, encouraging people to move from denser
spaces to emptier ones