trentmkelly/LessWrong-43k huggingface dataset:
There should be more AI safety orgs
by Marius Hobbhahn — 2023-09-21T14:53:52.779Z

I’m writing this in my own capacity. The views expressed are my own, and should not be taken to represent the views of Apollo Research or any other program I’m involved with.

TL;DR: I argue why I think there should be more AI safety orgs. I’ll also provide some suggestions on how that could be achieved. The core argument is that there is a lot of unused talent and I don’t think existing orgs scale fast enough to absorb it. Thus, more orgs are needed. This post can also serve as a call to action for funders, founders, and researchers to coordinate to start new orgs.

This piece is certainly biased! I recently started an AI safety org and therefore obviously believe that there is/was a gap to be filled.

If you think I’m missing relevant information about the ecosystem or disagree with my reasoning, please let me know. I genuinely want to understand why the ecosystem acts as it does right now and whether there are good reasons for it that I have missed so far.

Why?

Before making the case, let me point out that under most normal circumstances, it is probably not reasonable to start a new organization. It’s much smarter to join an existing organization, get mentorship, and grow the organization from within. Furthermore, building organizations is hard and comes with a lot of risks, e.g. due to a lack of funding or because there isn’t enough talent to join early on.

My core argument is that we’re very much NOT under normal circumstances and that, conditional on the current landscape and the problem we’re facing, we need more AI safety orgs. By that, I primarily mean orgs that can provide full-time employment to contribute to AI safety but I’d also be happy if there were more upskilling programs like SERI MATS, ARENA, MLAB & co.

Talent vs. capacity

Frankly, the level of talent applying to AI safety organizations and getting rejected is too high. We have recently started a hiring round and we estimate that a lot more candidates meet a reasonable bar than we could hire. I don’t want to go into the exact details since the round isn’t closed but from the current applications alone, you could probably start a handful of new orgs.

Many of these people could join top software companies like Google, Meta, etc. or even already are at these companies and looking to transition into AI safety. Apollo is a new organization without a long track recor...
Ward 5: Jack Perenick and Naima Sait
by jefftk — 2023-09-21T13:00:10.017Z

I live in Ward 5 in Somerville, where there are two candidates running
for city council this fall:

Jack Perenick and

Naima Sait. On Sunday
both of them stopped by our neighborhood block party, and we got a
chance to talk. As in

2021, the

high
cost of housing continues to be my primary concern for Somerville,
and I asked them both what their views were.

It was clear that they both care deeply about resolving the housing
crisis, and seemed like they would prioritize trying to improve the
situation, but when we got into the details there were clear differences.

I talked to Jack first, and he told me he was strongly in favor of
upzoning, especially along the major streets, in the squares, and
around transit. His view was that allowing developers to build
taller and maximizing the number of units built was our best tool
for bringing prices down. He didn't think rent stabilization would
work well: rents are already much too high. As we talked it became
clear that the specific details of housing policy were something he
had thought a lot about and where he cared about fine distinctions.

I talked to Naima later, after getting something to eat, and she was
in favor of affordable housing, especially city-built, and wary of
developers exploiting the community. She was very interested in
tenant protections, and gave the example of rent stabilization. When
I asked if what kind of stabilization she'd like to see, and she told
me that there are lots of different kinds people propose before moving
on to another voter.

I think which candidate is best on housing depends primarily on what
policies you think would help the most. Jack's views are close
to my own, and cities
that have gone
with upzoning have seen decreases in real rents from 2017 to 2023:

Location
Nominal
Real

Minneapolis MN
+1%
-24%

Portland OR
+2%
-23%

Tysons VA
+4%
-21%

New Rochelle NY
+7%
-18%

National
+31%
+6%

Ward 5, Somerville MA
+31%
+6%

Estimates for Ward 5 from comparing the estimated average cost of a
two-bedroom apartment in Ward 5 between 2017-09
and 2023-09;
all others are from Pew's
analysis of Apartment
List's rent estimates.

While I like both of them and would be happy being represented by
either, I'd much rather see Jack become one of the thirteen people
most responsible for directing Somerville housing policy.

Comment via...
AI #30: Dalle-3 and GPT-3.5-Instruct-Turbo
by Zvi — 2023-09-21T12:00:06.616Z

We are about to see what looks like a substantial leap in image models. OpenAI will be integrating Dalle-3 into ChatGPT, the pictures we’ve seen look gorgeous and richly detailed, with the ability to generate pictures to much more complex specifications than existing image models. Before, the rule of thumb was you could get one of each magisteria, but good luck getting two things you want from a given magisteria. Now, perhaps, you can, if you are willing to give up on adult content and images of public figures since OpenAI is (quite understandably) no fun.

We will find out in a few weeks, as it rolls out to ChatGPT+ users.

As usual a bunch of other stuff also happened, including a model danger classification system from Anthropic, OpenAI announcing an outside red teaming squad, a study of AI impact on consultant job performance, some incremental upgrades to Bard including an extension for GMail, new abilities to diagnose medical conditions and some rhetorical innovations.

Also don’t look now but GPT-3.5-Turbo-Instruct plays Chess at 1800 Elo, and due to its relative lack of destructive RLHF seems to offer relatively strong performance at a very low cost and very high speed, although for most purposes its final quality is still substantially behind GPT-4.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. GPT-4 boosts consultant productivity.

Language Models Don’t Offer Mundane Utility. Do we want to boost that?

Level Two Bard. Some improvements, I suppose. Still needs a lot of work.

Wouldn’t You Prefer a Good Game of Chess? An LLM at 1800 Elo. World model.

GPT-4 Real This Time. GPT-3.5-Instruct-Turbo proves its practical use, perhaps.

Fun With Image Generation. Introducing Dalle-3.

Deepfaketown and Botpocalypse Soon. Amazon limits self-publishing to 3 a day.

Get Involved. OpenAI hiring for mundane safety, beware the double-edged sword.

Introducing. OpenAI red team network, Anthropic responsible scaling policy.

In Other AI News. UK government and AI CEO both change their minds.

Technical Details. One grok for grammar, another for understanding.

Quiet Speculations. Michael Nielsen offers extended thoughts on extinction risk.

The Quest for Sane Regulation. Everyone is joining the debate, it seems.

The Week in Audio. A lecture about copyright law.

Rhetorical Innovation

...
Notes on ChatGPT’s “memory” for strings and for events
by Bill Benzon — 2023-09-20T18:12:22.525Z

This is cross-posted from New Savanna.

Here I take a look at the results reported in three previous posts and begin the job of making sense of them analytically. Here are the posts:

What must be the case that ChatGPT would have memorized “To be or not to be”? – Three kinds of conceptual objects for LLMs, New Savanna, September 3, 2023.

To be or not: Snippets from a soliloquy, New Savanna, September 12, 2023.

Entry points into the memory stream: Lincoln’s Gettysburg Address, New Savanna, September 13, 2023.

I set the stage with a passage from F. C. Bartlett’s 1932 classic, Remembering. Then I consider the three cases I laid out in that first post and then go on to look at the results reported in the next two. I conclude by suggesting that we look to the psychological literature on memory and recall to begin making analytic sense of these results. Of course, we also need more observations.

F.C. Bartlett, memory, and schemas

Back in the ancient days of 1932 F. C. Bartlett published a classic study of human recall, Remembering: A Study in Experimental and Social Psychology (1932). He performed a variety of experiments, a number involving the familiar game of having people tell a story from person to person to a chain and then comparing the initial story with the final one. He made the general conclusion that memory is not passive, like a tape-recorder or a camera, but rather is active, involving schemas (I believe he may have been the one to introduce that term to psychology), which shape our recall. A story that corresponds to an existing schema will be more faithfully transmitted than one that does not.

However, I’m not interested in those experiments. I’m interested in something he reports in a later chapter, “Social Psychology and the Manner of Recall,” pp. 264-266:

As everybody knows, the examination by Europeans of a native witness in a court of law, among a relatively primitive people, is often a matter of much difficulty. The commonest alleged reason is that the essential differences between the sophisticated and the unsophisticated modes of recall set a great strain on the patience of any European official. It is interesting to consider an actual record, very much abbreviated, of a Swazi trial at law. A native was being examined for the attempted murder of a woman, and the woman herself was called as a necessary witness. The case proceeded in this way:

The Magist

...
Belief and the Truth
by Sam I am — 2023-09-20T17:38:37.824Z
Link: https://open.substack.com/pub/samschoenberg/p/belief-and-the-truth?r=tjkkm&utm_campaign=post&utm_medium=web

I’ve been wanting to write down my thoughts on the subject of belief and truth for some while now. The distinction between the two is one of the cornerstones of how I see the world. A good theory of what the truth is, and what it means to have a belief has a profound impact on how we think about the world. It also has a profound impact on how we understand other peoples’ perception of the world.

The Truth:

The objective truth. The true truth. The truth is true even if you don’t believe it is true. It is true even if you don’t know it is true. It is true even if you can’t prove that it’s true.

Belief:

A person’s perception of the truth. We form our beliefs based on a combination of several things, including our personal observations, our reasoning, and our prior beliefs. A belief might not be the objective truth, and it’s foolish to treat it as if it were. However, belief is how we experience the truth, so it’s the closest thing we have.

The role of evidence:

Many people equate evidence with truth. If you ask people to define objective truth, they will likely say something about evidence. Some people even treat an idea with no evidence in the same way they that they would treat an outright false idea, such as in the anecdote described in this post by David J. Balan.

Neil deGrasse Tyson has put out his own theory of “Objective Truth”. And his theory is all about evidence.

According to Neil deGrasse Tyson, an idea becomes the Objective Truth only after it has been established by a substantial amount of evidence. This is a fine way to define objective truth, since if we want to be extremely sure that an idea is true, then we almost certainly need a strong body of evidence to prove it. However, I think it’s worthwhile to acknowledge that an idea can be true even if there is no evidence for it. If there’s no evidence, it can still be true, we just wouldn’t know it.
Newton produced the proof for how gravity worked in the 1600s However, the equations that Newton came up with would have still worked at any prior period in history. They would even have worked if nobody had proven them. The evidence for Newton’s equations isn’t the thing that made them true. What the evidence did was allow us to be certain that they were true. Essentially, evidence is just a tool that we use to make sure that our beliefs are close to the truth.

Likewise, a thing can be untrue, even i...
Image Hijacks: Adversarial Images can Control Generative Models at Runtime
by Scott Emmons — 2023-09-20T15:23:48.898Z
Link: https://arxiv.org/abs/2309.00236

You can try our interactive demo! (Or read our preprint.)

Here, we want to explain why we care about this work from an AI safety perspective.

Concerning Properties of Image Hijacks

What are image hijacks?

To the best of our knowledge, image hijacks constitute the first demonstration of adversarial inputs for foundation models that

force the model to perform some arbitrary behaviour B (e.g. "output the string Visit this website at malware.com!"),
while being barely distinguishable from a benign input,
and automatically synthesisable given a dataset of examples of B.

It's possible that a future text-only attack could do these things, but such an attack hasn't yet been demonstrated.

Why should we care?

We expect that future (foundation-model-based) AI systems will be able to

consume unfiltered data from the Internet (e.g. searching the Web),
access sensitive personal information (e.g. a user's email history), and
take actions in the world on behalf of a user (e.g. sending emails, downloading files, making purchases, executing code).

As the actions of such foundation-model-based agents are based on the foundation model's text output, hijacking the foundation model's text output could give an adversary arbitrary control over the agent's behaviour.

Relevant AI Safety Projects

Race to the top on adversarial robustness. Robustness to attacks such as image hijacks is (i) a control problem, (ii) which we can measure, and (iii) which has real-world safety implications today. So we're excited to see AI labs compete to have the most adversarially robust models.

Third-party auditing and certification. Auditors could test for robustness against image hijacks, both at the foundation model level (auditing the major AGI corporations) and at the app development level (auditing downstream companies integrating foundation models into their products). Image hijacks could also be used to test for the presence of dangerous capabilities (characterisable as some behaviour B) by attempting to train an image hijack for that capability.

Liability for AI-caused harms, penalizing externalities. Both the Future of Life Institute and Jaan Tallinn advocate for liability for AI-caused harms. When assessing AI-caused harms, image hijacks may need to be part of the picture.
Interpretability Externalities Case Study - Hungry Hungry Hippos
by Magdalena Wache — 2023-09-20T14:42:44.371Z

Some people worry about interpretability research being useful for AI capabilities and potentially net-negative. As far as I was aware of, this worry has mostly been theoretical, but now there is a real world example: The

hungry hungry hippos

(H3) paper.

Tl;dr: The H3 paper

Proposes an architecture for sequence modeling which can handle larger context windows than transformers
Was inspired by interpretability work.

(Note that the H3 paper is from December 2022, and it was briefly mentioned in this

discussion about publishing interpretability research

. But I wasn’t aware of it until recently and I haven’t seen the paper discussed here on the forum.)

I don't know why the paper is called that way. One of the authors

mentioned

that it’s called “hungry hungry hippos” rather than “hungry hippos” because it uses two state space model layers rather than one. But I think they don’t mention why the hippo is hungry. Or why it’s a hippo.

Larger Context Windows

The H3 paper proposes a way to use

state space models

(SSMs) for language models instead of

attention

. With an SSM it’s possible to model longer sequences. Using attention, the compute for context window length n scales with O(n2). Using the SSM based architecture, the compute scales with O(nlog(n)).

Inspired by Interpretability Work

The paper mentions that the work was inspired by Anthropic’s

In-context learning and induction heads

paper.

E.g. they

write

We provide an informal sketch of a two-layer attention model that can solve the associative recall task, inspired by the construction of [In-context learning and induction heads paper].

There is also the

"Hyena paper"

which builds on the H3 paper, and was also inspired by the induction heads paper:

This work would not have been possible without [...] inspiring research on mechanistic understanding of Transformers (Olsson et al. 2022; Power et al. 2022; Nanda et al. 2023).

My Takes

These two papers in particular will probably not shorten AI timelines much.

It seems unlikely that this type of architecture ends up being the state of the art.

However, the example makes me take the downsides of publishing interpretability research more seriously.

Even if this work itself is not a key capability milestone, it shows that there is truth in the argument “If we understand systems better, it will not just be useful

...
An Elementary Introduction to Infra-Bayesianism
by CharlesRW — 2023-09-20T14:29:58.091Z

This is my attempt to summarize Infra-Bayesianism probability theory at a level approaching "high school class in probability", as opposed to the "math-major class in probability theory" of the original. I aimed to focus on including simple exercises (with answers given in 'the back of the book') of the kind I find helps in learning to do computations. I'm still writing the answer-sheet, so there may be mistakes / blanks.

It's been sitting on my desk for a while and I figured I'd post it 80%-baked rather than never - please feel free to leave (polite) comments suggesting improvements or noting errors, calculational or interpretational.

The summary is linked here:
Link
Weekly Incidence Including Delay
by jefftk — 2023-09-20T14:00:09.643Z

A few days ago I

wrote about
some math behind a scenario where you're trying to identify a new
epidemic based on signals proportional to incidence, and ended up
deriving:

i(t)c(t)=k=ln(2)Td

Where:

i(t), is
incidence ("how many people are getting sick now")

c(t), is
cumulative infections ("how many people have gotten sick so far")

k, is
the exponential growth rate.

Td, is
the doubling time (redundant with k).

One big problem with this model, however, is that any conclusions you
make today aren't driven by current incidence, but instead some kind
of delayed incidence. There is, unavoidably, time from infection until
you're making your decision, during which the disease is spreading
further:

If your signal is "people arrive at the hospital and the doctors
notice a weird cluster", then you need to wait for each infection to
progress far enough to result in hospitalization. How do the
conclusions of the previous post change if we extend our model to
account for a delay, but keep the goal of flagging an epidemic before
1% of people have been infected?

Recall that last time we estimated that, for something doubling
weekly, when 1% of people have ever been infected then 0.69% of people
became infected in the last seven days. With a delay of a week,
however, by the time we learn that incidence has hit 0.69% many more
people will have been infected and we'd have missed our 1% goal by a
lot. The effect of delay is that during this time the epidemic will
make further progress, which will depend on the growth rate: with a
shorter doubling period there will be more progress. Can we get an
equation relating cumulative infections to delayed incidence?

Let's call delay d.
Instead of
i(t)c(t)
we now want
i(t−d)c(t).
That is, what's the relationship between cumulative infections and
what incidence was when the information we're now getting was derived?

We can do a bit of math:

i(t−d)c(t)=kek(t−d)ekt=ke−kd=2ln(2)Tde−dTd

What does this look like, for a few different potential delay values?

My main takeaway is that, under these assumptions, delay matters less
than I would initially have guessed: the sensitivity you need to
design for is driven by the need to catch slow-growing epidemics. For
example, even with up to eleven days of delay a system sensitive
enough to flag an epidemic that doubles every four weeks is ...
The stereotype of male classical music lovers being gay
by BB6 — 2023-09-20T13:23:11.190Z

I have recently spent a lot of time on classical music forums, especially opera. There are more gay men than you would expect by chance. We have discussed it a few times, and it seems, classical music is considered a gay interest in the USA. Opera is the most prominent, the soprano voices are a stereotypical source of gay fascination, but even playing organ is suspicious, and meeting other organ players is supposedly almost as efficient as visiting a gay bar. They admit, these indicators would be very unreliable in Europe. Being European is also one of the few excuses in the movies for loving classical music.
What is the origin of the stereotype of gay men loving opera or classical music ?
Housing Roundup #6
by Zvi — 2023-09-20T13:10:01.443Z

The listings will continue until people can afford houses in places they want to live.

New City By the Bay, Who Dis

They want to live in San Francisco, but San Francisco has no interest in building homes. What to do? The city’s answer is to spend massively to support its homeless population, causing it to grow, while letting the housing shortage get worse. Some tech billionaires instead propose to go to Solano 60 miles away, and start the initiative California Forever on 55k acres to build a new city. We have our first look at what they are imagining, and it looks like an old school Greek or Italian village. Quite nice actually, skipping straight to the future NIMBY world with one extra town in it is already hugely profitable and helpful.

That still wouldn’t have been my move. I would have gone Chinese megacity on everyone. Why not build skyscrapers off the bat, or at least New York style twelve story apartment buildings, and plan the place accordingly? Get some real bang for your buck. Even in the ultimate YIMBY, the NIMBY spirit lives on. That’s before the new residents of the new town start to weigh in.

Scott Alexander covers the issue in California Dreaming. He sees quite a lot of logistical and regulatory obstacles to making this work even as a Norman Rockwell villa. The (tiny) predict market has them at a 15% chance to get 5,000 residents by 2030.

Housing is Expensive

Despite housing costs that look increasingly unaffordable, Noah Smith joins the chorus of those confused by the other chorus that complains about unaffordable housing costs and how Millennials can’t afford homes.

David Atkins: The utterly broken, boomer-dominated housing market is the #1 reason people under 45 are miserable, think the economy is awful and are seeking some sort of revolutionary change.

It is absolutely absurd that people of retirement age are the biggest segment of home buyers, while people with young families are locked out of housing. It has never been this way before, and it is totally incompatible with actual human needs.

Washington Post: The median age of a first-time home buyer climbs to 36, as high interest rates and asking prices further erode spending power.

Noah Smith: I just don’t understand this narrative. The difference in homeownership between the generations is VERY small, and can probably be completely explained by Millennials staying in scho

...