trentmkelly/LessWrong-43k huggingface dataset:
Consider Joining the UK Foundation Model Taskforce
by Zvi — 2023-07-10T13:50:05.097Z

A few months ago, Ian Hogarth wrote the Financial Times Op-Ed headlined “We must slow down the race to God-like AI.”

A few weeks ago, he was appointed head of the UK Foundation Model Taskforce, and given 100 million pounds to dedicate to AI safety, to universal acclaim. Soon there will also be a UK Global AI Summit.

He wrote an op-ed in The Times asking everyone for their help, with accompanying Twitter thread. Based on a combination of sources, I am confident that this effort has strong backing for the time being although that is always fragile, and that it is aimed squarely at the real target of extinction risk from AI, with a strong understanding of what it would mean to have an impact on that.

Once again: The real work begins now.

The UK Taskforce will need many things in order to succeed. It will face opposition within and outside the government, and internationally. There is a narrow window until the AI summit to hit the ground running and establish capability and credibility.

The taskforce represents a startup government mindset that makes me optimistic, and that seems like the best hope for making government get things done again, including on other vital causes that are not AI, and not only in the UK. We likely only get one shot at this. If the taskforce fails, there will probably not be another such effort.

Right now, the main bottleneck is that the taskforce is talent constrained. There is an urgent need to scale up rapidly with people who can hit the ground running and allow the taskforce to orient.

If you are in position to help, then with the possible exception of creating your own organization at scale, I believe this is the highest leverage opportunity currently available.

To reach out and see if you can help, you can fill out this Google Form here.
“Reframing Superintelligence” + LLMs + 4 years
by Eric Drexler — 2023-07-10T13:42:09.739Z

Background

In January 2019, FHI published Reframing Superintelligence,[1] a book-length technical report on prospects for advanced AI. OpenAI published the first paper on GPT-2 a month later. Advances since then have been strange and rapid, and I’d like to revisit the report in light of what we have learned. In brief, I think that the abstract conceptual model of AI development and organization proposed in Reframing fits today’s reality quite well, even though LLM-based technologies have diverged far from anything I’d anticipated.

Below, you'll find an abstract of the abstract of the report, followed by a series of section-level mini-summaries[2] with update comments. I’ve omitted sections that are either outside the intended focus of this article or are too broad and forward-looking to summarize.

A significant impetus behind “Reframing Superintelligence” was to challenge a prevailing notion of advanced AI (equating superintelligent-level AI with a superintelligent agent), which has, in my view, been assigned disproportionate weight and skewed the balance of alignment research. The report offers an alternative framework that includes both risks and opportunities that are overlooked by agent-centric perspectives.

Note that this reframing is additive rather than subtractive: My intention is not to disregard agent-focused concerns — their importance is assumed, not debated.[3] Indeed, the AI services model anticipates a world in which dangerous superintelligent agents could emerge with relative ease, and perhaps unavoidably. My aim is to broaden the working ontology of the community to include systems in which superintelligent-level capabilities can take a more accessible, transparent, and manageable form, open agencies rather than unitary agents. This framework highlights different risks and expands the the solution-space for familiar problems.

Finally, when I refer “LLMs”, please read this as encompassing multimodal models (GPT-4!) with considerations that carry over to a wider range of foundation models.

Abstract of the Abstract

“Reframing Superintelligence” reviews the concept of superintelligent AI systems as utility-driven agents and suggests expanding our ontology of superintelligence to include compositions of AI systems that can best be understood through their structures, relationships, development processes, and the services they can pr...
Open-minded updatelessness
by Nicolas Macé — 2023-07-10T11:08:22.207Z

Summary

Bounded agents might be unaware of possibilities relevant to their decision-making. That is, they may not just be uncertain, but fail to conceive of some relevant hypotheses entirely. What's more, commitment races might pressure early AGIs into adopting an updateless policy from a position of limited awareness. What happens then when a committed AGI becomes aware of a possibility that’d have changed which commitment it’d have wanted to make in the first place? Motivated by this question, we develop "open-minded" extensions of updatelessness, where agents revise their priors upon experiencing awareness growth and reevaluate their commitment to a plan relative to the revised prior.

Introduction

Bounded agents may be unaware of propositions relevant to the decision problem they face.[1] That is, they don’t merely have uncertainty, but also fail to conceive of the full set of possibilities relevant to their decision-making. (For example, when playing a board game, one might be unaware of some crucial rule. Moreover, one’s awareness might grow, e.g. when one discovers such a rule midgame.)[2]
Awareness growth raises questions for commitment. What if one commits, and then discovers an important consideration, whose conception would have changed the plan one would have wanted to commit to? The earlier one commits, the less time one has to think about the relevant considerations, and the more likely this problem is to arise.

We are interested in preventing AGI systems from making catastrophic strategic commitments. One reason that not-fully-aware AGI systems could make bad commitments is that important hypotheses are missing for their priors. For example, they might fail to conceive of certain attitudes towards fairness that bargaining counterparts might possess. One might think that AGI agents would quickly become aware of all relevant hypotheses, and make commitments only then. But commitment race dynamics might pressure early AGIs into making commitments before thinking carefully, and in particular, in a position of limited awareness. From The Commitment Races problem (emphasis ours):

If two consequentialists are playing a game of Chicken, the first one to throw out their steering wheel wins. […] More generally, consequentialist agents are motivated to make commitments as soon as possible, since that way they can influence the behavior of other conseq

...
Consciousness as a conflationary alliance term for intrinsically valued internal experiences
by Andrew_Critch — 2023-07-10T08:09:48.881Z

Tl;dr: In this post, I argue from many anecdotes that the concept of 'consciousness' is more conflated than people realize, in that there's a lot of divergence in what people mean by "consciousness", and people are unaware of the degree of divergence. This confusion allows the formation of broad alliances around the value of consciousness, even when people don't agree on how to define it. What their definitions do have in common is that most people tend to use the word "consciousness" to refer to an experience they detect within themselves and value intrinsically. So, it seems that people are simply learning to use the word "consciousness" to refer to whatever internal experience(s) they value intrinsically, and thus they agree whenever someone says "consciousness is clearly morally valuable" or similar.
I also introduce the term "conflationary alliance" for alliances formed by conflating terminology.

Executive Summary

Part 1: Mostly during my PhD, I somewhat-methodically interviewed a couple dozen people to figure out what they meant by consciousness, and found that (a) there seems to be a surprising amount of diversity in what people mean by "consciousness", and (b) they are often surprised to find out that other people mean different things when they say "consciousness". This has implications for AI safety advocacy because AI will sometimes be feared and/or protected on the grounds that it is "conscious", and it's good to be able to navigate these debates wisely.

(Other heavily conflated terms in AI discourse might include "fairness", "justice", "alignment", and "safety", although I don't want to debate any of those cases here. This post is going to focus on consciousness, and general ideas about the structure of alliances built around confused concepts in general.)

Part 2: When X is a conflated term like "consciousness", large alliances can form around claims like "X is important" or "X should be protected". Here, the size of the alliance is a function of how many concepts get conflated with X. Thus, the alliance grows because of the confusion of meanings, not in spite of it. I call this a conflationary alliance. Persistent conflationary alliances resist disambiguation of their core conflations, because doing so would break up the alliance into factions who value the more precisely defined terms. Th...
The world where LLMs are possible
by Ape in the coat — 2023-07-10T08:00:11.556Z

In Artificial Addition, Eliezer used the ability to do arithmetic as a metaphor for intelligence. I really like this essay. It's witty and enlightening. And yet I have to admit it aged not so well. Among several confused ways to think about artificial addition and by metaphor about artificial intelligence as well - he mentioned these:

"It's a framing problem - what 'twenty-one plus' equals depends on whether it's 'plus three' or 'plus four'. If we can just get enough arithmetical facts stored to cover the common-sense truths that everyone knows, we'll start to see real addition in the network."
"But you'll never be able to program in that many arithmetical facts by hiring experts to enter them manually. What we need is an Artificial Arithmetician that can learn the vast network of relations between numbers that humans acquire during their childhood by observing sets of apples."
"No, what we really need is an Artificial Arithmetician that can understand natural language, so that instead of having to be explicitly told that twenty-one plus sixteen equals thirty-seven, it can get the knowledge by exploring the Web."

Now we know that this approach to artificial intelligence actually works. LLMs are trained on a huge corpus of texts from the internet to learn the vast network of relations between concepts, which gives them the ability to understand natural language, and as a result they can perform well in a vast array of intelligence tasks. Ironically, they are still not very good at calculation, though.

What can we say about it in hindsight? What mistake in reasoning led to this bad prediction? Why did past Eliezer fail to anticipate LLMs? What lesson can we learn from it?

First of all, let's remind ourselves why past Eiezer's reasoning made sense.

Understanding language is a proxy target. You can map mathematical facts to language and treat them in a roundabout way, but this is going to be less accurate then addressing them directly in the medium specifically optimized for them.
Knowing a collection of facts satisfying a rule isn't the same as knowing that rule. One can get the rule from the collection of facts via induction, but this a separate intellectual ability that you will have to embed into your system. It's easier to figure out the rule yourself, as you already possess the ability to do induction and then embed the rule.
Addition is pla

...
The virtue of determination
by Richard_Ngo — 2023-07-10T05:11:00.412Z

At the end of Replacing Guilt, Nate talks about the virtues of

desperation

,

recklessness

and

defiance

. None of these really hook neatly into my motivation system, but there’s a nearby virtue which does: determination, and in particular three facets of it which resonate strongly for me. I want to finish this sequence by talking about them.

The first is that I’m determined not to be a dumbass. I picture the trillions upon trillions of potential future people who could exist, looking back at us from the distant stars.

As Carl Sagan put it

:

“They will marvel at how vulnerable the repository of all our potential once was, how perilous our infancy, how humble our beginnings, how many rivers we had to cross, before we found our way.”

Or

as Joe Carlsmith put it

:

I imagine them going: “Whoa. Basically all of history, the whole thing, all of everything, almost didn’t happen.” I imagine them thinking about everything they see around them, and everything they know to have happened, across billions of years and galaxies — things somewhat akin, perhaps, to discoveries, adventures, love affairs, friendships, communities, dances, bonfires, ecstasies, epiphanies, beginnings, renewals. They think about the weight of things akin, perhaps, to history books, memorials, funerals, songs. They think of everything they love, and know; everything they and their ancestors have felt and seen and been a part of; everything they hope for from the rest of the future, until the stars burn out, until the story truly ends.

All of it started there, on earth. All of it was at stake in the mess and immaturity and pain and myopia of the 21st century. That tiny set of some ten billion humans held the whole thing in their hands. And they barely noticed.

And then I think about them focusing on me in particular, and my choices and decisions; all the good work I’ve done, and all the things I could have done better. And I don’t think they’ll blame me for the latter, not if they’ve internalized the types of thinking that I’ve described in the rest of this sequence. Regardless of what I do, they’ll think of me as a child in many ways, without the ability to see and shape the world to anywhere near the extent they can. But I’d prefer they think of me the way I think of a 10-year-old who can at least make meaningful plans and intentions, rather than a 5-year-old who is too inc...
Some reasons to not say "Doomer"
by Ruby — 2023-07-09T21:05:06.585Z

"Doomer" has become a common term to refer to people with pessimistic views about outcomes from AI. I claim this is not a helpful term on net, and generally will cause people to think less clearly.

Reification of identity + making things tribal

I finally realized today why politics and religion yield such uniquely useless discussions...

I think what religion and politics have in common is that they become part of people's identity, and people can never have a fruitful argument about something that's part of their identity. By definition they're partisan...

More generally, you can have a fruitful discussion about a topic only if it doesn't engage the identities of any of the participants. What makes politics and religion such minefields is that they engage so many people's identities. But you could in principle have a useful conversation about them with some people. And there are other topics that might seem harmless, like the relative merits of Ford and Chevy pickup trucks, that you couldn't safely talk about with others.
The most intriguing thing about this theory, if it's right, is that it explains not merely which kinds of discussions to avoid, but how to have better ideas. If people can't think clearly about anything that has become part of their identity, then all other things being equal, the best plan is to let as few things into your identity as possible.
- Paul Graham in Keep Your Identity Small[1]

I think a big risk of the "Doomer" label is it moves something from a "given the arguments and evidence I have, I believe X" into an essential deeper property (or commitment) of a person. It reifies it as an identity. You correspondingly then get people who are "not-Doomers", and more recently, I've heard the term "Foomer" too.

Because people's beliefs are quite correlated with those immediately around them, those more pessimistic and less pessimistic about AI tend be clusters, meaning it gets easy to point at clusters and make things tribal and political.

I'm in this tribe, they're in that tribe. My tribe is good, that tribe is bad. I think this makes us stupider in various ways:

We now start talking about people rather than object-level beliefs/evidence/reason.

It's much easier to dismiss people than arguments.

We put up social barriers to changing your mind. People don't like to be an odd one out among their friends, and if your friends identify as bei

...
The Seeker’s Game – Vignettes from the Bay
by Yulia — 2023-07-09T19:32:58.717Z

Introduction

Last year, one conversation left a lasting impression on me. A friend remarked on the challenges of navigating "corrupting forces" in the Bay Area. Intrigued by this statement, I decided to investigate the state of affairs in the Bay if I had the chance. So when I got the opportunity to visit Berkeley in February 2023, I prepared a set of interview questions. Can you share an experience where you had difficulty voicing your opinion? What topics are hard to clearly think about due to social pressures and factors related to your EA community or EA in general? Is there anything about your EA community that makes you feel alienated? What is your attitude towards dominant narratives in Berkeley? [1] In the end, I formally interviewed fewer than ten people and had more casual conversations about these topics with around 30 people. Most people were involved in AI alignment to some extent. The content for this collection of vignettes draws from the experience of around ten people. [2] I chose the content for the vignettes for one of two reasons – potential representativity and potential extraordinariness. I hypothesized that some experiences represent the wider EA Berkeley community accurately. Others, I included because they surprised me, and I wanted to find out how common they are. All individuals gave me their consent to post the vignettes in their current form.

How did I arrive at these vignettes? It was a four-step process. First, I conducted the interviews while jotting down notes. For the more casual conversations, I took notes afterwards. The second step involved transcribing these notes into write-ups. After that, I obscured any identifying details to ensure the anonymity of the interviewees. Lastly, I converted the write-ups into vignettes by condensing them into narratives and honing in on key points while trying to retain the essence of what was said.

I tried to reduce artistic liberties by asking participants to give feedback on how close the vignettes were to the spirit of what they meant (or think they meant at the time). It is worth noting that I bridged some gaps with my own interpretations of the conversations, relying on the participants to point out inaccuracies. By doing that, I might have anchored their responses. Moreover, people provided different levels of feedback. Some shared thorough, de...
Why have exposure notification apps been (mostly) discontinued?
by VipulNaik — 2023-07-09T19:07:50.576Z

I recently noticed that exposure notifications for COVID-19 in the app that I had been using, called CA Notify, had stopped working (I had attended an event a few days prior, and was feeling ever-so-slightly unwell, so I thought to check to see if there were any exposure notifications from the event; anyway it looks like my ever-so-slight not-feeling-well was unrelated to the event and it self-recovered quickly).

After digging further, I found that most of the COVID-19 contact tracing apps stopped sending exposure notifications around May 11, 2023, when the COVID-19 pandemic was officially declared over. See e.g. https://www.cnbc.com/2023/05/12/covid-most-americans-wont-receive-exposure-alerts-on-phones.html

I think this is a bit ... sad? I feel like contact tracing / exposure notifications is one of the kinds of precautionary measures that provides benefits at ~0 personal cost, unlike things like social distancing, masking, and vaccinations. In fact I would have liked exposure notifications to be expanded beyond just COVID-19 to cover more respiratory diseases and also include the ability to report new ailments before they spread widely. Seems like keeping an exposure notification system going and having people regularly use it to report infections would be good practice for both known and unknown ailments.

I hope that companies and health institutions are secretly working on some more robust longer-term solution such as this, but have not seen any evidence of it so far.

What are the reasons this might have been shut down? I couldn't find much discussion online, so I tried to come up with my own ideas, and here they are; I am not personally convinced of any of them:

Maybe the cost of running the backends was very high for the institutions managing the app (the California government in case of CA Notify).

Maybe people think "ignorance is bliss" when it comes to getting notified of COVID exposures, insofar as after having received such notifications, they may be morally obliged to cancel meetings with other people, incurring financial and social costs of doing so. They'd rather not be notified and incur the increased risk of transmitting COVID.

Maybe the exposure notification system was considered a security risk or liability risk of some sort, so shutting it down was considered prudent as a way to reduce the risk of some major scandal. (I think, even if th

...
The Necessity of Privacy: A Condition for Social Change and Experimentation?
by Blake — 2023-07-09T18:42:00.805Z

Recently, I came across a New Yorker article titled "Taking Back Your Privacy," featuring "Moxie" Marlinspike, a key figure behind Signal. According to Moxie, privacy is indispensable for both social change and experimentation. His view suggests that without privacy, we risk being pigeonholed by our publicly shared information, a characterization that's often difficult to escape.

For instance, let's say I make a controversial tweet. "I hate Trans people." I'm immediately labeled based on that tweet, and this label may persist, regardless of context or whether the sentiment was an offhand remark. However, is it accurate to claim that we inherently become what we express?

Can't we offer further explanations to contextualize a statement, allowing for a better understanding of one's views? Although this approach opens us up to critique, it also provides a platform for dialogue and growth.[1]

Another aspect worth considering is the potential misuse of information by malicious entities. The inception of Moxie's encrypted messaging work coincided with the Arab Spring, a period when oppressive governments frequently exploited information to undermine their opponents. But do we have concrete evidence that secure messaging promotes experimentation that leads to more democratic outcomes?

This question is not intended to undermine the importance of privacy or secure communication. Instead, I'm trying to understand the necessity of privacy in the context of social experimentation and change. (Ben Frankly, for instance, willfully sent "secrets" from France though he knew he was surrounded by spies.) I'm looking for examples that clearly illustrate how private experimentation can lead to widespread public benefit. If you have any insights or thoughts, I would love to hear them.

^

For an example of re-contextualizing of public sentiment, listen to BBC's Great Lives: JS Mill episode where Max Mills of "sick Nazi orgy" fame view contextualizes his branding.