trentmkelly/LessWrong-43k huggingface dataset:
AI pause/governance advocacy might be net-negative, especially without a focus on explaining x-risk
by Mikhail Samin — 2023-08-27T23:05:01.718Z

I found myself repeating the same words to multiple people, hence a short post.

I think some of the AI pause/governance advocacy might be net-negative. Three reasons:

Most importantly, it's easy to get regulation implemented for reasons different from addressing x-risk, which leads to all sorts of failure modes, where it becomes actually harder to prevent x-risk with further regulation, and we all simply die a bit later[1];
Less importantly, when talking about a dangerous technology, it's easy to incentivise governments to race to invest in that technology instead of preventing everyone from getting it;
To keep in mind, when talking about non-x-risk concerns that can help the pause, you might be outside your area of expertise and say something that any technical expert would say is wrong and consider you not to know what you're talking about.

Edit: I have somewhat updated my views around that since writing my post; in particular, see this comment.

Edit 2: I further changed my mind; I think this post and the comment still communicate something useful, but don't fully represent my current views.

Epistemic status: idk, handwavy models, a bunch of relevant experience; some people who disagreed with me have changed their mind when I talked about these points and they haven't made good points in response; I've seen docs that would've been harmful if important people saw them; the authors agreed with some of my object-level objections and changed the texts. Seems good to put out there.

If AI regulation isn't explicitly aimed at x-risk, it can be net-negative

What I think:

It’s pretty important to remember what the aim is. It’s not to slow down AI but to prevent an existential catastrophe. “Slowing AI” might help somewhat, but it’s not enough, and some kinds of "slowing down AI" can make it much harder to get policymakers to also introduce regulation that prevents x-risk.

Some strategies involve advocating for/introducing AI regulations without mentioning x-risk, with the hope of locally slowing down AI progress, building frameworks that can later be used to address x-risk, or fostering relationships with policymakers. Many of them carry significant downside risks and are net-negative.

Many people don't seem to consider politicians and voters to be grown-ups who can listen to arguments for why AI poses an x-risk, and implement AI regulation that slows down AI for the reasons we...
Will issues are quite nearly skill issues
by dkl9 — 2023-08-27T16:42:11.264Z
Link: https://dkl9.net/essays/will_issue.html

This post contains content that might be regarded as spoilers for the free will problem

I cannot jump off a cliff. I do not know any cliffs near me, but that's not the reason I care about, so we'll counterfactually skip over that. The more interesting reason I can't jump off a cliff is that I don't want to. When we confidently know that I don't want to jump off a cliff, and that no one in the near future would coerce or manipulate me to, we can conclude a negligibly tiny probability for my jumping off a cliff, or, in practice, that I will not do that. Once we accept that "could" expresses an option's plausibility, we're saying that "it cannot happen that I'd jump off a cliff", and, equivalently, "I couldn't jump off a cliff".

There is a straightforward sequence of muscle commands from my brain to my legs that would lead me to jump. But the existence of such motor outputs is not what we typically mean by "ability". If there was a cliff near me, but I didn't know about it (probably true anyway), that pattern of leg movements would just as much exist, but I'd be unable (in the conventional sense, not just "my" "weird" sense) to go to it and jump.

Not wanting to do something inhibits the person from doing it just as effectively as an explicit "inability". You can't just object "but free will!" (and be correct), both sith I just anticipated it, and so you wouldn't want to (unless you're trying to be annoying), and sith human minds are not mystically unpredictable blobs of chaos. People's minds have a theoretically-measurable internal state that constrains what they do.

If someone (honestly, after thorough thought, and not opposing their established habit) resolves to not do something (nontrivial and easily avoided), they will tend to not do it, unless they are affected by (sufficiently convincing) new input. A failure to accurately constrain someone's future actions is a matter of your limited knowledge about what they think or will encounter, not their inscrutable freedom to do whatever they know how, even when it arbitrarily contradicts any or all of their preferences.

Remarking on someone's intent, then saying "but they could do otherwise!" shows your uncertainty about that person's plans or future inputs, not any fundamental nondeterminism in human behaviour. Or it may show your failure to neglect small probabilities for practical purposes. If you ask me to recite...
Xanadu, GPT, and Beyond: An adventure of the mind
by Bill Benzon — 2023-08-27T16:19:58.916Z

Cross posted from New Savanna.

I've posted a new working paper. Title above; links, abstract, table of contents, and introductory material below.

Download at:

Academia: https://www.academia.edu/106001453/Xanadu_GPT_and_Beyond_An_adventure_of_the_mind

SSRN: https://ssrn.com/abstract=4553351

ResearchGate: https://www.researchgate.net/publication/373433939_Xanadu_GPT_and_Beyond_An_adventure_of_the_mind

Abstract: This article recounts an intellectual journey that began in curiosity about the structure of Coleridge’s “Kubla Khan” in the late 1960s and has led to an interest in large language models at the present time. A close analysis of the poem revealed its two parts each to have a nested structure (think of a matryoshka doll) that suggested the operation of an underlying computational process (nested loops). That led to the study of computational linguistics (semantic networks), followed by neuroscience (Karl Pribram’s neural holography), and cultural evolution. In the 2010s I began following work digital humans had been doing with machine learning. When GPT-3 was released in 2020 I was ready, though it took me awhile to establish a link, however tentative, between that conceptual universe and that of “Kubla Khan.”

Encountering Coleridge’s “Kubla Khan” 3
Romantic states of consciousness 3
Matryoshka dolls and the escape from Xanadu 5
Semantic networks and a Shakespeare sonnet 8
Karl Pribram, neural holography, and the brain 10
The wandering years 11
Through GPT to the future 13
Mind and world in text 15
The Text of “Kubla Khan,” including Coleridge’s prefatory note 17
A note about the cover image 19

Encountering Coleridge’s “Kubla Khan”

I became hooked on Coleridge’s “Kubla Khan” in the Spring of 1969, my last semester as an undergraduate at Johns Hopkins. Three years later “Kubla Khan” had become the standard against which I measured my understanding of the human mind. That is why I am telling a story about how my interest in the mind has evolved through “Kubla Khan” to include, most recently, ChatGPT. Strange as it may seem, that poem is the vehicle through which I am coming to terms with this new technology and arriving at a sense of its potential.

There is a sense in which the story of that great poem can be traced back to the 11th century invasion of Britain by the Norman French, for that culture-crossing is what gave rise to the English language. A centu...
High level overview on how to go about estimating "p(doom)" or the like
by Aryeh Englander — 2023-08-27T16:01:07.467Z

This is from a comment I left on a recent blog post from Gary Marcus, but I thought it might be of general interest here. Feedback and critiques definitely welcome!

------

For what it's worth, here's a slightly longer overview on my own current preferred approach to estimating "p(doom)", "p(catastrophe)", or other extremely uncertain unprecedented events. I haven't yet quite worked out how to do this all properly though - as Gary mentioned, I'm still working on this as part of my PhD research and as part of the MTAIR project. The broad strokes are more or less standard probabilistic risk assessment (PRA), but some of the details are my own take or are debated.

Step 1: Determine decision thresholds. To restate the part Gary quoted from our email conversation: We only really care about "p(doom)" or the like as it relates to specific decisions. In particular, I think the reason most people in policy discussions care about something like p(doom) is because for many people higher default p(doom) means they're willing to make larger tradeoffs to reduce that risk. For example, if your p(doom) is very low then you might not want to restrict AI progress in any way just because of some remote possibility of catastrophe (although you might want to regulate AI for other reasons!). But if your p(doom) is higher then you start being willing to make harder and harder sacrifices to avoid really grave outcomes. And if your default p(doom) is extremely high then, yes, maybe you even start considering bombing data centers.

So the first step is to decide where the cutoff points are, at least roughly - what are the thresholds for p(doom) such that our decisions will change if it's above or below those points? For example, if our decisions would be the same (i.e., the tradeoffs we'd be willing to make wouldn't change) for any p(doom) between 0.1 and 0.9, then we don't need any more fine-grained resolution on p(doom) if we've decided it's at least within that range.

How exactly to decide where the proper thresholds are, of course, is a much more difficult question. This is where things like risk tolerance estimation, decision making for non-ergodic processes, and decision making under deep uncertainty (DMDU) come in. I'm still trying to work my way through the literature on this.

Step 2: Determine plausible ranges for p(doom), or whatever probability you're trying to forecast. Use avail...
Trying a Wet Suit
by jefftk — 2023-08-27T15:00:08.728Z

I get cold very quickly in the water, enough that unless it's close to
body temperature I get chilled through within ~15min. This mostly
wasn't a problem, because I'd take a quick dip to cool off and then
hang out on the beach, but now that I have kids they (and I) want lots
of swimming together time. When I touched on this

a few weeks ago

people
recommended

trying
a wetsuit, and yesterday evening I did for the first time!

It was different in a bunch of ways, but on balance I like it a lot.
Some things I noticed:

Initially there was some air in my suit, which felt funny
bubbling out.

The suit is slightly buoyant, taking some getting used to.

It still felt cold getting in, until my body had time to heat
up the trapped layer of water.

I didn't get cold in the water! This was with maybe ~78F water
and ~82F air. I could play with my kids until they wanted to get out
of the water.

I bought separate pants and a vest, which I wore under my
swimsuit. The pants worked very well, and the vest worked ok: I got a
bit of water rotating through where they met, and I occasionally
needed to pull the vest down. Possibly a full body suit would have
been better? But those seem more annoying to get in and out of, and
the amount of water moving through was pretty low with the kind of
playing we were doing.

Once I got out of the water, I stayed wet a lot longer. This
was warmer than usual for the first part, since it was a warm sort of
wet, though after we got to the stage where I normally would have been
fully dry (~45m?) I was mildly colder.

Overall I'm happy, and am looking forward to swimming with it again in
the future!

(My kids are now asking if they can get ones too, which is fine with
me!)

Comment via: facebook, mastodon
Apply to a small iteration of MLAB to be run in Oxford
by RP — 2023-08-27T14:21:18.310Z

TLDR: We’re running a small iteration of MLAB (~10 participants) in Oxford towards the end of September. If you’re interested in participating, apply here by 7 September. If you’re interested in being a TA, please email us directly at oxfordmlab@gmail.com

Edit: The dates are now confirmed for the 23 September- 7 October.

Background

MLAB is a program, originally designed by Redwood Research, to help people upskill for alignment work. We think it’s a good use of time if you want to eventually get into technical alignment work, or if you want to work on theoretical alignment or related fields and think understanding ML would be useful. The program we’re running is slightly shorter than the full MLAB—two weeks instead of three. We’ve condensed the curriculum similarly to how WMLB was condensed last year.

We plan to have just under 10 participants, and 2-3 TAs.

Curriculum

This curriculum might change slightly. Depending on participant interest, we might also have two optional days before the course to work through prereqs (the W0 materials) together.

W0D1 - pre-course exercises on PyTorch and einops (CPU)
W1D1 - practice PyTorch by building a simple raytracer (CPU)
W1D2 - build your own ResNet (GPU preferred)
W1D3 - build your own backpropagation framework (CPU)
W1D4 - model training Part 1: model training and optimizers (CPU) Part 2: hyperparameter search (GPU preferred)
W1D5 - GPT Part 1: build your own GPT (CPU) Part 2: sampling text from GPT (GPU preferred)
W2D1&2 - transformer interpretability (CPU)
W2D3 - transformer interpretability on algorithmic tasks (CPU)
W2D4 - intro to RL Part 1: multi-armed bandit (CPU) Part 2: DQN (CPU)
W2D5 - policy gradients and PPO (CPU)

Other activities will include guest speakers and reading groups.

Logistics

Dates: 23 September - 7 October

Location: Oxford

Housing will be covered for participants not already living in Oxford.

Travel from within the UK is covered. Travel from outside the UK is not covered.

Questions

Feel free to comment questions below or DM any of us.
The Game of Dominance
by Karl von Wendt — 2023-08-27T11:04:36.661Z

Concerns about AI safety rely on the assumption that a sufficiently powerful AI might take control of our future in an undesirable way. Meta’s head of AI, Yann LeCun, correctly points this out in a recent tweet, but then argues that this assumption is wrong because “intelligence” and “dominance” are independent of each other and only humans have a natural tendency to dominate, so we will remain the “apex species”. Here is the tweet in full:

Once AI systems become more intelligent than humans, humans we will *still* be the "apex species."

Equating intelligence with dominance is the main fallacy of the whole debate about AI existential risk.

It's just wrong.

Even *within* the human species It's wrong: it's *not* the smartest among us who dominate the others.

More importantly, it's not the smartest among us who *want* to dominate others and who set the agenda.

We are subservient to our drives, built into us by evolution.

Because evolution made us a social species with a hierarchical social structure, some of us have a drive to dominate, and others not so much.

But that drive has absolutely nothing to do with intelligence: chimpanzees, baboons, and wolves have similar drives.

Orangutans do not because they are not a social species. And they are pretty darn smart.

AI systems will become more intelligent than humans, but they will still be subservient to us.

They same way the members of the staff of politicians or business leaders are often smarter than their leader.

But their leader still calls the shot, and most staff members have no desire to take their place.

We will design AI to be like the supersmart-but-non-dominating staff member.

The "apex species" is not the smartest but the one that sets the overall agenda.

That will be us.

Edit: Even ChatGPT 3.5 could point out the - to me - obvious problems with this argument when I asked it to criticize the tweet, putting the full text in the prompt. An LLM shouldn’t be smart enough to dismantle the argument of a Turing Award winner, but it had no problem finding significant flaws with it. Among other things, it criticizes that "assuming that AI will remain subservient to humans oversimplifies the potential risks associated with advanced AI":

While AI systems would be designed with specific objectives, there's a concern that once AI becomes highly intelligent, it could develop its own motivations or interpretations of its

...
Mesa-Optimization: Explain it like I'm 10 Edition
by brook — 2023-08-26T23:04:21.280Z

There are a couple of explanations of mesa-optimization available. I think Rob Miles' video on the topic is excellent, but I think existing written descriptions don't make the concept simple enough to be understood thoroughly by a broad audience. This is my attempt at doing that, for those who prefer written content over video.

Summary

Mesa-optimization is an important concept in AI alignment. Sometimes, an optimizer (like gradient descent, or evolution) produces another optimizer (like complex AIs, or humans). When this happens, the second optimizer is called a 'mesa-optimizer'; problems with the alignment (safety) of the mesa-optimizer are called 'inner alignment problems'.

What is an optimizer?

I'll define an optimizer as something that looks through a 'space' of possible things and behaves in a way that 'selects' some of those things. One sign that there might be an optimizer at work is that weird things are happening, by which I mean 'things that are very unlikely if the system was behaving randomly'. For instance, humans do things all the time that would be very unlikely to happen if we behaved randomly, and humans would certainly not exist at all if evolution worked by just making new organisms with totally random genes.

The human brain

The human brain is an optimizer -- it looks through the different things you could do and picks ones that get you things you like.

For instance, you might go to the shop, get ice cream, pay for the ice cream, and then come back. If you think about it, that's a really complex series of actions -- if you behaved completely randomly, you would never ever end up with ice cream. Your brain has searched through a lot of possible actions you could take (walking somewhere else, dancing in your living room, moving just your left foot up 30 degrees) and expects none of them will get you ice cream -- and has then selected one of the very few paths that will get you what you want.

Evolution

A stranger example of an optimizer is evolution. You've probably heard this already -- evolution 'optimizes inclusive genetic fitness'. But what does this mean?

Organisms change randomly-- their genes change due to mistakes in the process of making new organisms. Sometimes, that change helps the organism to reproduce by allowing it to survive longer, by making it more attractive, etc. When that happens, the next generation has more of t...
Aumann-agreement is common
by tailcalled — 2023-08-26T20:22:03.738Z

Thank you to Justis Mills for proofreading and feedback. This post is also available on my substack.

Aumann's agreement theorem is a family of theorems which say that if people trust each other and know each other's opinions, then they agree with each other. Or phrased another way, if people maintain trust with each other, then they can reach agreement. (And some variants of the theorem, which take computational factors into consideration, suggest they can do so quite rapidly.)

The original proof is pretty formal and confusing, but a simpler heuristic argument is that for an honest, rational agent, the mere fact of them professing an opinion can be strong evidence to another rational agent, because if the speaker's probabilities are higher than the speaker's prior, then they must have seen corresponding evidence to justify that opinion.

Some people find this confusing, and feel like it must be wrong because it doesn't apply to most disagreements. I think these people are wrong because they are not sufficiently expansive in what they think of as a disagreement. The notion of disagreement that Aumann's agreement theorem applies to is when the people assign different probabilities to events; this is a quite inclusive notion which covers many things that we don't typically think of as disagreements, including cases where one party has information about a topic and the other party has no information.

My vacation in Norway relied tons on Aumann agreements

Recently, I had a vacation in Norway with my wife.

In order to get there, and to get around, we needed transport. At first we disagreed with people who provided transport there, as we didn't know of many specific means of transport, only vaguely that there would be some planes and ships, without knowing which ones. But my wife had heard that there was something called the "Oslo ferry", so we Aumann-agreed that this was an option, and decided to investigate further.

We disagreed with the company that provided the Oslo ferry, as we didn't know what their website is, so we asked Google, and it provided some options for what the ferry might be, and we Aumann-agreed with Google and then went investigating from there. One website we found claimed to sell tickets to the ferry; at first we disagreed with the website about when we could travel as we didn't know the times of the ferry, but then we read which times it claimed was ...
Ramble on STUFF: intelligence, simulation, AI, doom, default mode, the usual
by Bill Benzon — 2023-08-26T15:49:47.781Z

Cross posted from New Savanna. I've published a number of these rambles over at New Savanna. They are a way for me to think about what's on my mind and consist of a variety of quick and crude remarks on various topics: Epistemic status: I'm still thinking about it.

x x x x x

Time for another one of these. Lots on my mind. Difficult to sort it out. So, some quick hits just to get them all into the same space.

The discourse of intelligence

As far as I can tell the notion of intelligence that’s instantiated in such phrases as “intelligent life” or “artificial intelligence” is relatively new, dating back to the late 19th and early 20th centuries with the emergence of tests to estimate general cognitive capacity, that is, intelligence, as in “intelligence quotient.” This conception is pervasive in the modern world. It pervades the educational system and our discourse on race and public policy, including eugenics. AI seems to be a cross between this idea and computation, the idea and its realization in digital computers.

How does this discourse show up in fiction? I’m thinking of such characters as Sherlock Holms or, much more recently, Adrian Monk? These characters are noted for their intelligence, but are otherwise odd and in Monk’s case, extremely odd.

Bostrom’s simulation hypothesis seems very thin to me

The more I think about it, it is not at all clear just what he means by simulation. He focuses on human experience. But what about animals? They are part of human experience. Are animals just empty shells, existing only as appearances for humans. But what about their interactions with one another? What drives their actions? Are they (quasi) automatous agents or are their actions entirely subordinate to the mechanisms which present appearances to humans? What happens when animals or humans reproduce? Do we have simulated DNA, fertilization, embryonic development? That seems to be rather granular and would impose a heavy computational burden. But if the simulation doesn’t do that, then how are the characteristics of the offspring related to those of its parents? What mechanism is used of that? Lots of problems.

Other posts on the simulation argument.

Performance and competence in AI test-taking

https://rodneybrooks.com/the-seven-deadly-sins-of-predicting-the-future-of-ai/ Rodney Brooks has introduced a distinction between competence and performance into his on...