trentmkelly/LessWrong-43k huggingface dataset:
A freshman year during the AI midgame: my approach to the next year
by Buck — 2023-04-14T00:38:49.807Z

I recently spent some time reflecting on my career and my life, for a few reasons:

It was my 29th birthday, an occasion which felt like a particularly natural time to think through what I wanted to accomplish over the course of the next year 🙂.
It seems like AI progress is heating up.
It felt like a good time to reflect on how Redwood has been going, because we’ve been having conversations with funders about getting more funding.

I wanted to have a better answer to these questions:

What’s the default trajectory that I should plan for my career to follow? And what does this imply for what I should be doing right now?
How much urgency should I feel in my life?

How hard should I work?
How much should I be trying to do the most valuable-seeming thing, vs engaging in more playful exploration and learning?

In summary:

For the purposes of planning my life, I'm going to act as if there are four years before AGI development progresses enough that I should substantially change what I'm doing with my time, and then there are three years after that before AI has transformed the world unrecognizably.
I'm going to treat this phase of my career with the urgency of a college freshman looking at their undergrad degree--every month is 2% of their degree, which is a nontrivial fraction, but they should also feel like they have a substantial amount of space to grow and explore.

The AI midgame

I want to split the AI timeline into the following categories.

The early game, during which interest in AI is not mainstream. I think this ended within the last year 😢
The midgame, during which interest in AI is mainstream but before AGI is imminent. During the midgame:

The AI companies are building AIs that they don’t expect will be transformative.
The alignment work we do is largely practice for alignment work later, rather than an attempt to build AIs that we can get useful cognitive labor from without them staging coups.
For the purpose of planning my life, I’m going to imagine this as lasting four more years. This is shorter than my median estimate of how long this phase will actually last.

The endgame, during which AI companies conceive of themselves as actively building models that will imminently be transformative, and that pose existential takeover risk.

During the endgame, I think that we shouldn’t count on having time to develop fundamentally new alignment insights or techniques (except mayb

...
Against AI Understanding and Sentience: Large Language Models, Meaning, and the Patterns of Human Language Use
by  — 2023-04-13T23:29:05.927Z
Link: http://philsci-archive.pitt.edu/21983/

Abstract

Large language models such as ChatGPT are deep learning architectures trained on immense quantities of text. Their capabilities of producing human-like text are often attributed either to mental capacities or the modeling of such capacities. This paper argues, to the contrary, that because much of meaning is embedded in common patterns of language use, LLMs can model the statistical contours of these usage patterns. We agree with distributional semantics that the statistical relations of a text corpus reflect meaning, but only part of it. Written words are only one part of language use, although an important one as it scaffolds our interactions and mental life. In human language production, preconscious anticipatory processes interact with conscious experience. Human language use constitutes and makes use of given patterns and at the same time constantly rearranges them in a way we compare to the creation of a collage. LLMs do not model sentience or other mental capacities of humans but the common patterns in public language use, clichés and biases included. They thereby highlight the surprising extent to which human language use gives rise to and is guided by patterns.
Financial Times: We must slow down the race to God-like AI
by trevor — 2023-04-13T19:55:26.217Z
Link: https://www.ft.com/content/03895dc4-a3b7-481e-95cc-336a524f2ac2

Update April 14th 2023: Apparently, this was the most read story on FT.com a day or two ago.

_________________________________________________________

The article itself is paywalled, so here you go (you can often circumvent paywalls by typing the article's name into google search's news tab). This isn't actually bad news; it means that the FT will reach a smaller and more elite audience (and hopefully with above average quant skills), rather than the NYT which simply maximizes the number of views by being the most popular outlet.

Notably, this story not only gives AI alignment positive coverage, but it is also extremely close to the front page on Financial Times's website, and with a very striking image to boot (possibly the most important factor). Of course, it's still social media spread that largely decide the fate of these articles, not the news outlet's website, so we can't know for sure how helpful it is.

This article was written by Ian Hogarth, "yesterday" according to the page. It's important to bear in mind that being published in a major news outlet is a stamp of approval that most people take extremely seriously when entering a field for the first time, and that factor is a much bigger deal than whether the author got everything right on their first try.

As a side note, Raemon has recently recommended these sources as a good way to explain AI risk to someone for the first time:

Superintelligence FAQ (very accessible to layfolk)
The Alignment Problem from a Deep Learning Perspective (written with ML researchers in mind)

(I'll work on compiling more of these soon)

_________________________________________________________

On a cold evening in February I attended a dinner party at the home of an artificial intelligence researcher in London, along with a small group of experts in the field. He lives in a penthouse apartment at the top of a modern tower block, with floor-to-ceiling windows overlooking the city’s skyscrapers and a railway terminus from the 19th century. Despite the prime location, the host lives simply, and the flat is somewhat austere.

During dinner, the group discussed significant new breakthroughs, such as OpenAI’s ChatGPT and DeepMind’s Gato, and the rate at which billions of dollars have recently poured into AI. I asked one of the guests who has made important contributions to the industry the question that often co...
R0 Is Not Counterfactual
by jefftk — 2023-04-13T19:50:04.824Z

Someone who helps organize an

N95-required
dance recently wrote to a group of organizers:

R0 is the number of people that an infected individual is likely to
infect. R0 was 5.4
in Dec 2022. ... People who assume increased risk for themselves are
also assuming increased risk for 5.4 other people.

This is wrong in two main ways, and I responded on the list, but I
wanted to share my response here as well because the claim illustrates
two common misconceptions.

The first issue is that R0
is for an entirely "susceptible" population, one in which no one has
any gained any immunity from exposure or vaccination. What an R0 of
5.4 would mean is that if it had suddenly appeared in 2019, each
infected person would on average infect 5.4 others. This is very
different from the current situation where most people have had
several shots, plus most people have had covid at least once. The
term for the expected number of people an infected person will
directly infect given current conditions is Rt,
currently about 1. Which
is really just another way of saying that covid levels have been
changing relatively slowly: if Rt were 5.4 we'd have rapid growth
tearing through the population.

The other issue is that even Rt doesn't tell you how many infections
you getting sick would cause. You may know various things about your
behavior that make the expected number of people you'd directly infect
higher or lower, but that's not the main issue. Instead it's that (a)
people you infect can go on to infect other people and (b) people you
infect might otherwise have been infected by other people. These two
factors push in opposite directions, but both can be quite large.
Here are a pair of toy situations showing how, holding Rt fixed, one
infection can lead to either very many or almost no counterfactual
infections:

A new epidemic is starting, and Rt (which is R0 in this case)
is 5.4. There are very clear symptoms, and people are just starting
to catch on. Very soon there will be massive behavior changes to
suppress Rt, and at this stage those might or might not be enough.
One more person getting infected could have a 5% impact on the chance
that this becomes a pandemic infecting ~half the world. In which case
the expected number of additional infections is ~200M people. Here
(a) is the main factor.

An new pandemic is well under way, and it h

...
Subscripts for Probabilities
by niplav — 2023-04-13T18:32:17.267Z

cross-posted from niplav.github.io

Gwern has wondered about a use-case for subscripts in hypertext. While
they have settled on a specific use-case, namely years for
citations, I propose a different one:
reporting explicit probabilities.

Explicitely giving for probabilities in day-to-day English text is usually
quite clunky: "I assign 35% to North Korea testing an intercontinental
ballistic missile until the end of this year" reads far less smoothly
than "I don't think North Korea will test an intercontinental ballistic
missile this year".

And since subscripts are a solution in need of a problem, one can wonder
how well those two fit together: Quite well, I claim.

In short, I propose to append probabilities in subscript after a statement
using standard HTML subscript notation (or LATEX as a fallback if
it's available), with the probability possibly also being a link to a
relevant forecasting platform with the same question:

I think Donald Trump is going to be incarcerated before 203065%.

This is almost as readable as the sentence without the probability.

There are some complications with negations in sentences or multiple
statements. For the most part, I'll simply avoid such cases ("Doctor,
it hurts when I do this!" "Don't do that, then."), but if I had to,
I'd solve the first problem by declaring that the probability applies to
the literal meaning of the previous sentence, including all negations;
the problem with multiple statements is solved by delimiters.

As an example for the different kinds of negation: "The train won't
come more than 5 minutes late90%" would (arguendo) mean the
same thing as "I don't think the train will come more than 5 minutes
late90%" means the same as "The train will take more than 5
minutes to arrive10%" equivalent to "I assign 90% probability
to the train arriving within the next 5 minutes".

With multiple statements, my favorite way of delimiting is currently
half brackets: "I think ⸤it'll rain tomorrow⸥55%, but
⸤Tuesday is going to be sunny⸥80%, but I don't think
⸤your uncle is going to be happy about that⸥15%."

The probabilities in this context aren't quite
evidentials, but neither
are they veridicals nor
miratives, I propose the world
"credal" for this category.

Enumerating Possible Notations

The exact place of insertion is subtle: In sentences with a
single central statement, there are multiple locations one could place
the probabi...
The Virus - Short Story
by Michael Soareverix — 2023-04-13T18:18:48.068Z

There are a number of ways that our AI and technological development can turn out. This is a hypothetical story about a way where AI development is stopped and a 'pivotal act' that does not require AGI occurs.

The first thing I notice when I wake up is that the sun is shining through the window. "Oh sh--" I bite off a curse. I've probably missed my class.

I reach for my phone, and suddenly, I'm startled by how hot it is. Searingly hot. I tentatively tap the screen, but it doesn't turn on. "Did the battery explode?" I mutter. A sinking feeling washes over me. This is a terrible start to the day.

I don't know it yet, but waking up late is soon going to be the least of my worries.

I run downstairs, grabbing my backpack. I walk over the where my laptop is charging only to realize in horror that smoke is trickling from behind the screen.

"What the hell?"

This is really bad. Maybe there was a huge electrical surge? Both my phone and the laptop are fried.

I open the door, jog down the stairs, and see dozens of students anxiously gathered outside.

"What's happening?" I ask, jogging up to the group. All of them look scared.

"Something bad is going on." One of them gestures towards smoke in the distance. "All of the computers are fried, both in the college and the town. Nobody has any idea what's going on. None of the phones work."

A chill settles over me. "Is it a cyberattack?" I ask. I've never heard of anything like this. I've heard that detonating a nuke in the atmosphere could knock out an electrical grid, but I feel like I would have heard it.

I look up at the sky. It's a crisp morning, probably about 10:00 AM or so.

There is not a single plane in the sky. Not even the faint, ever-present rumbling of jet engines at forty-five thousand feet. The day suddenly seems eerily quiet.

I listen harder. I don't hear any cars either.

It is then that I begin to realize the magnitude of what is happening.

The next week happens in a terrifying blur. School is cancelled. A few people have radios, the really old kind. Apparently the power grid across the entire US, and possibly the entire world, is down. Every computer seems to have spontaneously combusted or is now otherwise entirely useless.

Most of the cars on the road have been rendered unusable. They won't even turn on because the computers are fried. There are still a lot of older cars, so people begin to travel and move by radio.

After a...
Polluting the agentic commons
by hamandcheese — 2023-04-13T17:42:05.619Z
Link: https://www.secondbest.ca/p/polluting-the-agentic-commons

The risks from developing superintelligent AI are potentially existential, though hard to visualize. A better approach to communicating the risk is to illustrate the dangers from existing systems, and to then imagine how those dangers will increase as AIs become steadily more capable.

An AGI could spontaneously develop harmful goals and subgoals, or a human could simply ask an obedient AGI to do things that are harmful. This second bucket requires many fewer assumptions about arcane results in decision theory. And even if those arcane results are where the real x-risk lies, it’s easier to build an intuition for the risks by working from the bottom-up, as scenarios in which “AGI gets an upgrade and escapes the lab” require a conceptual leap that’s easy to get distracted by.

After all, it didn't take long after GPT models became widely available for someone to build an autonomous agent literally called "ChaosGPT." We don't need to speculate about emergent utility functions. People will choose to unleash harmful agents just because.

The current trend is for these models to become smaller and smaller to the point where they will soon run locally on a smartphone. Once a GPT agent is compact enough to run on a personal computer without you noticing, it’s inevitable that it will be used to wreak havoc just like any other kind of malware. Let’s call these evil GPTs “mal-bots.” Now imagine the following:

A spurned lover gives a mal-bot a profile of his ex: photos, name, location, worst fears, food allergies, etc.

The bot then has one mission: find ways to terrorize the ex while avoiding detection and making copies of itself.

The ex's phone rings off the hook with vaguely synthetic voices that threaten to kill her dog. Food containing allergens is constantly delivered to her home. Dozens of credit card applications are applied for under her name.

She moves and changes her number, but the mal-bot deduces her new location from the contrails in an Instagram selfie.

The man eventually regrets what he's done but can't make it stop. The mal-bot is quietly running on thousands of computers around the world, coordinating with its own copies through cryptic posts from an anon account.

Seeking revenge, she gives in and downloads a mal-bot of her own.

Now solve for the equilibrium.

This scenario doesn't require superintelligence. These capabilities mostly already exist. GPTs can be used as...
Even if human & AI alignment are just as easy, we are screwed
by Matthew_Opitz — 2023-04-13T17:32:22.735Z

I get the sense that something like Eliezer's concept of "deep security" as opposed to "ordinary paranoia" is starting to seep into mainstream consciousness—not as quickly or as thoroughly as we'd like, to be sure. But more people are starting to understand the concept that we should not be aiming for an eventual Artificial Super-Intelligence (ASI) that is constantly trying to kill us, but is constantly being thwarted by humanity always being just clever enough to stay one step ahead of it. The 2014 film "Edge of Tomorrow" is a great illustration of the hopelessness of this strategy. If humanity in that film has to be clever enough to thwart the machines every single time, but the machines get unlimited re-tries and only need to be successful once, most people can intuit that this sort of "infinite series" of re-tries converges towards humanity eventually losing (unless humanity gets a hold of the reset power, as in the film).

Instead, "deep security" when applied to ASI (to dumb-down the term beyond a point that Eliezer would be satisfied with) is essentially the idea that, hmmmmm, maybe we shouldn't be aiming for ASI that is always trying to kill us and barely failing each time. Maybe our ASI should just work and be guaranteed to "strive" towards the thing we want it to do under every possible configuration-state of the universe (under any "distribution"). I think this idea is starting to percolate more broadly. That is progress.

The next big obstacle that I am seeing from many otherwise-not-mistaken people, such as Yann LeCun or Dwarkesh Patel, is the idea that aligning AIs to human values should, by default (as a prior baseliness assumption, unless shown strong evidence to update one's opinions otherwise), be about as easy as aligning human children to do so.

There are, of course, a myriad of strong arguments against that idea. That angle of argument should be vigorously pursued and propagated. However, here I'd like to dispute the premise that humans are aligned sufficiently with each other to give us reassurance even in the scenario in which it turned out that humans and AI were similarly alignable with human values.

There is a strong tendency to attribute good human behavior to intrinsic friendly goals towards other humans rather than extrinsic means towards other ends (such as attaining...
Intro to Ontogenetic Curriculum
by Eris — 2023-04-13T17:15:13.065Z

Disclaimer. This is the first post in a series about my personal research progress as a member of the Interpretable Cognitive Architectures team at AI Safety Camp.

Epistemic status. An educated take on combining existing knowledge into cohesive whole.

TL;DR

Ontogenetic curriculum is a map of developmental threads that describe the formation of a human mind
The most promising research lines are:

Ontogeny of values and concepts,
Curriculum learning,
Distribution of complexity between learning algorithm and training curriculum

What is OC?

Ontogenetic curriculum is a map of developmental threads that describe the formation of a human mind. That includes value learning, concept acquisition, skill building and development of cognitive capacities.

Some examples of relevant fields and research areas include:

Developmental psychology and studies on child development, which spanning from Piaget’s Theory of Cognitive Development to Dynamic Theory of Development of action and thought by Fischer and his work on constructive web.
Developmental robotics and attempts to construct artificial cognitive architectures to test psychological theories with their help.
General ethology research, such as Niche Construction Theory and studies on animal cognition

Why OC? Promising research questions

There are many possible lines of research under this topic, but the most promising questions are:

Ontogeny of values and concepts. Values and preferences in humans are formed gradually during upbringing, which resembles evolutionary dynamics with offspring-values born under the pressure of a new environmental niche. Getting better understanding of the rules governing this dynamics would help us design better value mechanism for artificial agents. Related questions:

What are the exact conditions leading to the emergence of positive attractors in humans?
What are the most powerful contributing factors in that dynamics?
How do values apply to a new situation? Would an agent’s extrapolation of values to a new situation match with humans’?

Curriculum learning. What new opportunities does the transition from batch training to ordered training open up? The main candidate for the answer is fine control over time when agent learns something. Choosing the right points of intervention suitable for specific value is in theory a good strategy for steering behaviour away from undesired territories.
Whe

...
Was Homer a stochastic parrot? Meaning in literary texts and LLMs 
by Bill Benzon — 2023-04-13T16:44:43.443Z

Cross posted from New Savanna.

The phrase “stochastic parrot” was coined, I believe, by Emily Bender, a computational linguist and one of the coauthors of the paper, On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Consider this passage:

That is, human language use takes place between individuals who share common ground and are mutually aware of that sharing (and its extent), who have communicative intents which they use language to convey, and who model each others’ mental states as they communicate. As such, human communication relies on the interpretation of implicit meaning conveyed between individuals. The fact that human-human communication is a jointly constructed activity [29, 128] is most clearly true in co-situated spoken or signed communication, but we use the same facilities for producing language that is intended for audiences not co-present with us (readers, listeners, watchers at a distance in time or space) and in interpreting such language when we encounter it. It must follow that even when we don’t know the person who generated the language we are interpreting, we build a partial model of who they are and what common ground we think they share with us, and use this in interpreting their words.

Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind.

I have been happy to accept this view, while at the same time denying that LLMs are stochastic parrots. That’s the view I took in my 2020 working paper, GPT-3: Waterloo or Rubicon? Here be Dragons, and which I have maintained until quite recently. Then, at the end of March, a discussion I had over at LessWrong has lead me to revise that view, if only provisionally, in a post, MORE on the issue of meaning in large language models (LLMs). There, following an argument that John Searle had made about certain digital devices, where he pointed out that they can be said to compute only in relation “to some actual or possible consciousness that can interpret the processes as computationally.” So, mutatis mutandis, the operations of an LLM are only meaningful relative to some actual or possible consciousness that can interpret the processes as being meaningful.

It seems to me, then, that the question of whether or not the language produced by LLMs is meaningful is up to us. Do you trust it? Do WE trust it? Why or ...