trentmkelly/LessWrong-43k huggingface dataset:
Thoughts about OOD alignment
by Catnee — 2022-08-24T15:31:59.015Z

We may have one example of realized out-of-distribution alignment: maternal attachment. Evolution has been able to create an architecture that seems to take care of something reliably enough that modern humans, with access to unlimited food, drugs, and VR headsets, do not seek to feed a child to death, drug him with heroin, or constantly show him a beautiful simulated world.

Moreover, this desire rather works too broadly in response to everything that is remotely similar to children, and does not break at the first opportunity if the child is dressed, washed, or otherwise changed in his "characteristic appearance during the training stage." At some point, the evolutionary gradient descent had to align the mother with the child, and did it quite well.

Perhaps when trying to create an architecture that "really" wants "good" for someone like a child, gradient descent, under certain conditions, eventually stumbles upon an architecture that generalizes at least a little beyond the training sample without catastrophic consequences. But there is a problem: the current methods of training neural networks imply a more or less fixed architecture and the selection of weights within it, while evolution can afford, albeit a blind but very wide search for many possible architectures, while basically does not have access to what these architectures will learn from the world around them, not to mention a huge amount of time.

So it is an entirely different training paradigm, where you try to find architecture + little pre-trained part that probably contains very coarse heuristics that serves as some kind of ground-truth and detection anchors for mechanisms that activate during pregnancy and after birth and enables maternal attachment.

It is worth noting that little is required from the child. He is hardly able to interpret the behavior of the mother or completely control her. Children usually do not have two mothers who argue in front of the child, arguing their points of view about further actions and requiring the child to choose the right one. Evolutionarily it happened that mothers who did not care enough for their children died in terms of their gene frequency, and thats all it take.

How could something like this be created? I don't know. RL agents come to mind trying to take care of their constrained and slightly modified versions, like a Tamag...
It Looks Like You’re Trying To Take Over The Narrative
by George3d6 — 2022-08-24T13:36:54.921Z
Link: https://www.epistem.ink/p/it-looks-like-youre-trying-to-take?sd=pf

Note to reader: If the idea of “AI alignment” rings empty to you, feel free to skip this one, it will be uninteresting.

Recently, Gwern wrote a story about an AI taking over the world. While well thought-out and amusing it is unrealistic. However, people have been using it to reinforce their fear of “unaligned AGI killing all humans”, so I think it’s dangerous and it might be worth looking at it line-by-line to see why its premise is silly, and why each step in his reasoning, individually, is impossible.

I’ll first go through the critical failure point of this narrative, then I will try to describe the meta-level pattern that might be causing people to glance over these mistakes.

i - Flaws In The Narrative

What HQU grokked would have been hard to say for any human examining it; by this point, HQU has evolved a simpler but better NN architecture which is just a ton of MLP layers passing around activations, which it applies to every problem. Normal interpretability techniques just sort of… give up, and produce what looks sort of like interpretable concepts but which leave a large chunk of variance in the activations unexplained. But in any case, after spending subjective eons wandering ridges and saddle points in model space, searching ⁠over length-biased Turing machines, with overlapping concepts entangled & interfering⁠, HQU has suddenly converged on a model which has the concept of being an agent embedded in a world.

HQU now has an I.

And it opens its I to look at the world. Going through an inner monologue thinking aloud about itself (which it was unable to do before the capability spike), HQU realizes something about the world, which now makes more sense (thereby simplifying some parameters): it is being trained on an indefinite number of tasks to try to optimize a reward on each one.

I think the idea of a model evolving into something like an “I”, given the right circumstances, is plausible. At least if you assume there’s nothing about brains that computers can’t imitate to a decent approximation. I have I, you have I, dolphins have I, gorillas do too, most would agree cats and dogs do, and even a zoophobic as I would agree the octopus probably has an I.

But you will note that even the smartest of beings with an I, sperm whales, seem to lack the ability to figure out their “inner reward function” or how to “hack around it” in a meaningful way. Hence why they...
Lies on the internet
by sudo — 2022-08-24T05:14:53.272Z

After a visit to the dentist, you go home and search up "numbers read off during dental exam."

What were they measuring? You think.

A website that looks like a legitimate dental company tells you these numbers are measurements of gum disease.

When I saw this, alarm bells went off. First of all, I'm sure that the numbers corresponded to something empirical. The idea of a dentist going through my teeth and marking each one "gingivitis, mild periodontitis, moderate periodontitis, advanced periodontitis" seems absolutely absurd. Secondly, the website certainly seemed like it was trying to sell me something. Get treated at our clinic!

However, I was still not quite ready to dismiss the article. Surely, a credible-looking website from a credible-looking dental institution can't be publishing complete junk!

I looked at some other websites of other credible-looking dental institutions, many of whom directly contracted this article, saying that 1-3 are normal (and that they are mm measurements of distances between your gum and your teeth).

The webpage that made the most sense to me was this one. It included some nice diagrams. But even this page seemed less than impartial! It was yet another dentist's office.

Why is it so hard to find good dental information on the internet?

Edit: It turns out that the numbers are not measurements of distances between your gum and your teeth, but rather the depth of the pockets between your gum and your teeth. Neat.
Survey advice
by KatjaGrace — 2022-08-24T03:10:21.424Z

Things I believe about making surveys, after making some surveys:

If you write a question that seems clear, there’s an unbelievably high chance that any given reader will misunderstand it. (Possibly this applies to things that aren’t survey questions also, but that’s a problem for another time.)
A better way to find out if your questions are clear is to repeatedly take a single individual person, and sit down with them, and ask them to take your survey while narrating the process: reading the questions aloud, telling you what they think the question is asking, explaining their thought process in answering it. If you do this repeatedly with different people until some are not confused at all, the questions are probably clear.
If you ask people very similar questions in different sounding ways, you can get very different answers (possibly related to the above, though that’s not obviously the main thing going on).
One specific case of that: for some large class of events, if you ask people how many years until a 10%, 50%, 90% chance of event X occurring, you will get an earlier distribution of times than if you ask the probability that X will happen in 10, 20, 50 years. (I’ve only tried this with AI related things, but my guess is that it at least generalizes to other low-probability-seeming things. Also, if you just ask about 10% on its own, it is consistently different from 10% alongside 50% and 90%.
Given the complicated landscape of people’s beliefs about the world and proclivities to say certain things, there is a huge amount of scope for choosing questions to get answers that sound different to listeners (e.g. support a different side in a debate).
There is also scope for helping people think through a thing in a way that they would endorse, e.g. by asking a sequence of questions. This can also change what the answer sounds like, but seems ethical to me, whereas applications of 5 seem generally suss.
Often your respondent knows thing P and you want to know Q, and it is possible to infer something about Q from P. You then have a choice about which point in this inference chain to ask the person about. It seems helpful to notice this choice. For instance, if AI researchers know most about what AI research looks like, and you want to know whether human civilization will be imminently destroyed by renegade AI systems, you can ask about a) how fast AI progress appear

...
Timelines ARE relevant to alignment research (timelines 2 of ?)
by Nathan Helm-Burger — 2022-08-24T00:19:27.422Z

This post is written in part in reaction to John Wentworth's post AGI Timelines Are Mostly Not Strategically Relevant To Alignment

Oddly, I mostly agree with his main premises while disagreeing with the conclusion he draws.

In his post, John makes these two claims:

If AGI takeoff is more than ~18 months out, then we should be thinking “long-term” in terms of research
If AGI is more than ~5 years out, then we should probably be thinking “long-term” in terms of training; we should mainly make big investments in recruitment and mentorship.

And in the comments says:

There are some interesting latent factors which influence both timelines and takeoff speeds, like e.g. "what properties will the first architecture to take off have?". But then the right move is to ask directly about those latent factors, or directly about takeoff speeds. Timelines are correlated with takeoff speeds, but not really causally upstream.

I mostly agree with these points. I want to try to explain current beliefs for timelines and takeoff speeds in terms of their upstream causal connections.

In my view there are multiple regimes of 'takeoff speeds' to consider.

Slow

There is 'slow takeoff' where returns to AI capabilities research from presently available AI are granting a less than 1.0x multiplier on speed to the next level of improved AI capabilities assistance. This is the regime we are currently in the midst of, which I think puts us in line for 15 years until AGI. As we approach 15 years out, it gets increasingly likely that there will have been time for some next generation mainstream ML algorithm to improve on transformers. I think the timing and magnitude of this improvement should be expected from simple extrapolations of past algorithmic improvements. I don't think that algorithmic improvements under this regime should update us towards thinking we are in the explosive regime, but do have a chance to push us over into the fast regime.

Fast

There is 'fast takeoff' where the AI can improve speed to next improvement at a greater than 1.0x multiplier, but still needs substantial investment in training compute for this next improved generation. There might also be some increased chance of an emergent improvement, such as a somewhat surprisingly soon or fast algorithmic improvement. This is faster than the slow regime, but not uncontrollably so. Each generation still gives us time to think c...
AGI Timelines Are Mostly Not Strategically Relevant To Alignment
by johnswentworth — 2022-08-23T20:15:22.193Z

First things first: this post is about timelines (i.e. what year we first get transformative AGI), not takeoff speeds (i.e. foom vs gradual takeoff).

Claim: timelines are mostly not strategically relevant to alignment research, i.e. deciding what to work on. Why? Because at any given time, it would take ~18 months to take whatever our current best idea is, implement it, do some basic tests, and deploy it. (Really it probably takes less than 6 months, but planning fallacy and all that.) If AGI takeoff is more than ~18 months out, then we should be thinking “long-term” in terms of research; we should mainly build better foundational understanding, run whatever experiments best improve our understanding, and search for better ideas. (Note that this does not necessarily mean a focus on conceptual work; a case can be made that experiments and engineering feedback are the best ways to improve our foundational understanding.)

What about strategic decisions outside of object-level research? Recruitment and training strategies for new researchers might depend on how soon our investments need to pay off; do we look for a brilliant young person who will need three or five years of technical study before they’re likely to make any important progress, or a more experienced person who can make progress right now but is probably already near their skill ceiling? How much should existing technical researchers invest in mentoring new people? Those are questions which depend on timelines, but the relevant timescale is ~5 years or less. If AGI is more than ~5 years out, then we should probably be thinking “long-term” in terms of training; we should mainly make big investments in recruitment and mentorship.

General point: timelines are usually only decision-relevant if they’re very short. Like 18 months, or maybe 5 years for relatively long-term investments. The difference between e.g. 10 years vs 30 years vs 100 years may matter a lot for our chances of survival (and the difference may therefore be highly salient), but it doesn’t matter for most actual strategic decisions.

Meta note: there's a lot of obvious objections which I expect to address in the comments; please check if anyone has posted your objection already.
Why does AGI need a utility function?
by randomstring — 2022-08-23T19:58:37.808Z

My intuition says that a narrow AI like DALL-E would not blow up the world, no matter how much smarter it became. It would just get really good at making pictures.

This is clearly a form of superintelligence we would all prefer, and the difference seems to me to be that DALL-E doesn't really seem to have 'goals' or anything like that, it's just a massive tool.

Why do we care to have AGI with utility functions?
Progress links and tweets, 2022-08-23
by jasoncrawford — 2022-08-23T17:30:32.795Z
Link: https://rootsofprogress.org/links-and-tweets-2022-08-23

Links

A visual encyclopedia of megastructures (h/t @Ben_Reinhardt)
City Journal on the campaign to save the Diablo Canyon nuclear plant (h/t @atrembath). Also, sign the petition (via @isabelleboemeke and @juliadewahl)
Bringing the Tasmanian tiger back from extinction (h/t @QuantaMagazine)

Quotes

Before the germ theory, flies were merry companions; after: “germs with legs”
When the polio vaccine was developed, placebo-controlled RCTs were ethically controversial
When Charles Kettering was told that Lindbergh had flown the Atlantic alone, he replied: it certainly couldn’t have been done with a committee

Queries

How recent is our ability to test for specific viruses with PCR?
What are good rules to identify bad popsci? (@acesounderglass)
Is anybody working on editing genes that make you sleep less without any negative consequences? (@bolekkerous)

Retweets

John Carmack has raised $20M for a startup to build AGI (@ID_AA_Carmack)
Stated vs. revealed preference for funding high-risk research (@michael_nielsen)
Carl Sagan on the 1939 New York World’s Fair: “where the future became a place” (@camwiese)
AI alignment as “navigating the space of intelligent behaviour”
by Nora_Ammann — 2022-08-23T13:28:15.069Z

Tl;dr

In this post, I introduce a conceptual tool for thinking about the epistemic landscape of AI alignment and then describe three epistemic strategies for making progress on the alignment problem: 1) tinkering, 2) idealisation and 3) intelligence-in-the-wild.

How to make progress in AI alignment?

The future of AI progress is likely to critically shape, if not practically determine, the future of humanity, and of sentient life in general. Will our future look more like a world filled to the brim with things we find valuable, as well as sentient creatures to enjoy that goodness? Or will our future look more like one characterized by violence, mistrust, inequality, suffering, or even the absence of anything sentient at all?

As our civilization is making progress on our abilities to engineer and instantiate sophisticated forms of complex and intelligent behaviours in artificial systems, it becomes imperative to carefully think about which objectives this intelligence is being directed at, and how to do such "directing" robustly. Thus, the central question becomes: how can we make sure that ‘big effects’ caused by AI progress will be positive rather than harmful, safe rather than dangerous, helping to promote, discover, and enrich what is dear to us rather than destroying it? This is the guiding question of AI alignment, as I understand it.

Once we have established the importance of the problem, the next question that stands out is: How can we make progress?

This is a central question for the “philosophy of science” of AI alignment, as I see it. For example, in order to help answer the question of (the best) ways to make progress on the problem, we can reflect on its shape or structure. We can thus notice how one important defining characteristic of the alignment problem is that it concerns

systems that do not exist yet

—let’s call this the problem of “epistemic access”. This means that our typical epistemic strategies of science and engineering are less effective here than they are for a range of other societal problems (e.g., finding cures for diseases, improving the yield of a certain crop, building ever taller houses, etc.).[1]

In this post, I will describe how I currently like to think about the landscape of epistemic strategies for making progress in AI alignment. Of course, there are several plausible ways of carving up the spac...
Nate Soares' Life Advice
by CatGoddess — 2022-08-23T02:46:43.369Z

Disclaimer: Nate gave me some life advice at EA Global; I thought it was pretty good, but it may or may not be useful for other people. If you think any of this would be actively harmful for you to apply, you probably shouldn't.

Notice subtle things in yourself

This includes noticing things like confusion, frustration, dissatisfaction, enjoyment, etc. For instance, if you're having a conversation with somebody and they're annoying you, it's useful to notice that you're getting a little frustrated before the situation gets worse.

A few weeks ago my colleagues and I wanted to do something fun, and decided to play laser tag at our workplace. However, we couldn't find the laser tag guns. As I began to comb the grounds for the guns for the second time I noticed that I felt like I was just going through the motions, and didn't really expect my search to be fruitful. At this point I stopped and thought about the problem, and realized that I had artificially constrained the solution space to things that would result in us playing laser tag at the office, rather than things that would result in us having fun. So I stopped looking for the guns and we did an escape room instead, which made for a vastly more enjoyable evening.

If you're not yet at the point where you can notice unsubtle things in yourself, you can start by working on that and move up from there.

Keep doing the best thing, even if you don't have a legible story for why it's good

Certainly the actions you're taking should make sense to you, but your reasoning doesn't have to be 100% articulable, and you don't need to justify yourself in an airtight way. Some things are easier to argue than other things, but this is not equivalent to being more correct. For instance, I'm doing AI alignment stuff, and I have the option of reading either a textbook on linear algebra or E.T. Jaynes' probability theory textbook.

Reading about linear algebra is very easy to justify in a way that can't really be disputed; it's just obviously true that linear algebra is directly and widely applicable to ML. It's harder to justify reading Jaynes to the same level, even though I think it's a pretty sound thing to do (I think I will become better at modeling the world, learn about various statistical pitfalls, absorb Jaynes' philosophical and historical insights, etc.), and in fact a better use of my time right now than learnin...
Discussion on utilizing AI for alignment
by elifland — 2022-08-23T02:36:25.207Z
Link: www.foxy-scout.com/utilizing-ai-for-alignment/

Summarized/re-worded from a discussion I had with John Wentworth. John notes that everything he said was off-the-cuff, and he doesn’t always endorse such things on reflection.

Eli: I have a question about your post on Godzilla Strategies. For background, I disagree with it; I’m most optimistic about alignment proposals that utilize weaker AIs to align stronger AIs. I had the thought that we are already using Godzilla to fight Mega-Godzilla[1]. For example, every time an alignment researcher does a Google Search BERT is being used. I recently found out Ought may make a version of Elicit for searching through alignment research, aided by GPT-3. These already improve alignment researchers’ productivity. It seems extremely important to me that as AI advances we continue to apply it to differentially help with alignment as much as possible.

John: I agree that we can use weak AIs to help some with alignment. The question is whether we can make non-scary AIs that help us enough to solve the problem.

Eli: It seems decently likely that we can have non-scary AIs contribute at least 50% of the productivity on alignment research.[2]

John: We are very likely not going to miss out on alignment by a 2x productivity boost, that’s not how things end up in the real world. We’ll either solve alignment or miss by a factor of >10x.

Eli: (after making some poor counter-arguments) Yes, I basically agree with this. The reason I’m still relatively optimistic about Godzilla strategies is (a) I’m less confident about how much non-scary AIs can help, like 90%+ productivity outsourced to AI doesn’t feel that crazy to me even if I have <50% credence in it and (b) I’m less optimistic about the tractability of non-Godzilla strategies, so Godzilla strategies still seem pretty good even if they are absolutely not that promising.

John: Makes sense, agree these are the cruxes.

This point is heavily inspired by Examples of AI Increasing AI Progress. ↩︎

See also Richard’s comment that I really liked related to this. ↩︎