trentmkelly/LessWrong-43k huggingface dataset:
What's Going on With OpenAI's Messaging?
by ozziegooen — 2024-05-21T02:22:04.171Z

This is a quickly-written opinion piece, of what I understand about OpenAI. I first posted it to Facebook, where it had some discussion.

Some arguments that OpenAI is making, simultaneously:

OpenAI will likely reach and own transformative AI (useful for attracting talent to work there).
OpenAI cares a lot about safety (good for public PR and government regulations).
OpenAI isn’t making anything dangerous and is unlikely to do so in the future (good for public PR and government regulations).
OpenAI doesn’t need to spend many resources on safety, and implementing safe AI won’t put it at any competitive disadvantage (important for investors who own most of the company).
Transformative AI will be incredibly valuable for all of humanity in the long term (for public PR and developers).
People at OpenAI have thought long and hard about what will happen, and it will be fine.
We can’t predict concretely what transformative AI will look like or what will happen after (Note: Any specific scenario they propose would upset a lot of people. Vague hand-waving upsets fewer people).
OpenAI can be held accountable to the public because it has a capable board of advisors overseeing Sam Altman (he said this explicitly in an interview).
The previous board scuffle was a one-time random event that was a very minor deal.
OpenAI has a nonprofit structure that provides an unusual focus on public welfare.
The nonprofit structure of OpenAI won’t inconvenience its business prospects or shareholders in any way.
The name “OpenAI,” which clearly comes from the early days when the mission was actually to make open-source AI, is an equally good name for where the company is now.* (I don’t actually care about this, but find it telling that the company doubles down on arguing the name still is applicable).

So they need to simultaneously say:

“We’re making something that will dominate the global economy and outperform humans at all capabilities, including military capabilities, but is not a threat.”

“Our experimental work is highly safe, but in a way that won’t actually cost us anything.”

“We’re sure that the long-term future of transformative change will be beneficial, even though none of us can know or outline specific details of what that might actually look like.”

“We have a great board of advisors that provide accountability. Sure, a few months ago, the board tried to fire Sam, and Sam was able ...
Harmony Intelligence is Hiring!
by James Dao — 2024-05-21T02:11:44.675Z
Link: https://www.harmonyintelligence.com/jobs

Hey folks! Pleased to announce we have a new open position for a Founding Research Engineer at Harmony Intelligence. You’ll be responsible for measuring and identifying dangerous AI capabilities across various domains: cybersecurity, biosecurity, persuasion and manipulation, self-exfiltration and self-replication, and more.

Location:
We're a small but quickly growing remote-first startup, with folks in San Francisco and Sydney.

About Harmony:
We're specifically focused on reducing catastrophic AI risk by building evals and conducting red teaming/audits. Though we’re still young, we're already making a big impact! We’ve published our first automated red teaming paper to massively positive reception from the AI community, and are actively working on model evaluations and other AI safety research. For any further questions, please reach out to our co-founder and CTO Alex Browne. To apply or see more information, go to harmonyintelligence.com/jobs.
[Linkpost] Statement from Scarlett Johansson on OpenAI's use of the "Sky" voice, that was shockingly similar to her own voice.
by Linch — 2024-05-20T23:50:28.138Z
Link: https://variety.com/2024/digital/news/scarlett-johansson-responds-shocked-angered-openai-chatgpt-her-1236011135/

Scarlett Johansson makes a statement about the "Sky" voice, a voice for GPT-4o that OpenAI recently pulled after less than a week of prime time.
tl;dr: OpenAI made an offer last September to Johansson; she refused. They offered again 2 days before the public demo. Scarlett Johansson claims that the voice was so similar that even friend and family noticed. She hired legal counsel to ask OpenAI to "detail the exact process by which they created the ‘Sky’ voice," which resulted in OpenAI taking the voice down.
Full statement below:

Last September, I received an offer from Sam Altman, who wanted to hire me to voice the current ChatGPT 4.0 system. He told me that he felt that by my voicing the system, I could bridge the gap between tech companies and creatives and help consumers to feel comfortable with the seismic shift concerning humans and Al. He said he felt that my voice would be comforting to people.

After much consideration and for personal reasons, declined the offer.

Nine months later, my friends, family and the general public all noted how much the newest system named ‘Sky’ sounded like me.

When I heard the released demo, I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine that my closest friends and news outlets could not tell the difference. Mr. Altman even insinuated that the similarity was intentional, tweeting a single word ‘her’ — a reference to the film in which I voiced a chat system, Samantha, who forms an intimate relationship with a human.

Two days before the ChatGPT 4.0 demo was released, Mr. Altman contacted my agent, asking me to reconsider. Before we could connect, the system was out there.

As a result of their actions, I was forced to hire legal counsel, who wrote two letters to Mr. Altman and OpenAl, setting out what they had done and asking them to detail the exact process by which they created the ‘Sky’ voice. Consequently, OpenAl reluctantly agreed to take down the ‘Sky’ voice.

In a time when we are all grappling with deepfakes and the protection of our own likeness, our own work, our own identities, I believe these are questions that deserve absolute clarity. I look forward to resolution in the form of transparency and the passage of appropriate legislation to help ensure that individual rights are protected.
Some perspectives on the discipline of Physics
by Tahp — 2024-05-20T18:19:22.429Z
Link: https://quark.rodeo/physics.html

I wrote the linked post, and I’m posting a lightly edited version here for discussion. I plan to attend LessOnline, and this is my first attempt at blogging to understand and earnestly explain and is also gauging interest in the topic in case someone at LessOnline wants to discuss the firmware of the universe with me. I might post more physics if there seems to be interest. Here is the post:

Three distinct disciplines within physics

When I teach introductory mechanics, I like to tell my students that there are three things which are all called physics, even if only one of them tends to show up on their exams.

Physics is a set of theories

Periodically during the semester, I draw the following diagram on the board for my Newtonian Mechanics class:

This is an example of a theory. You can know what it means and how it works without knowing anything about how to apply it to real-world systems. Knowing a theory is like knowing the rules of a game. The game may or may not have connections to real life. You are not automatically a master at winning a game just because you know the rules. In the same way, physics theories and the concepts which guide them can exist seperately from their predictive power. Just because the plum pudding model is not a good model of actual atoms doesn't mean that you can't say something about what predictions it would make or what the average charge of a nucleus under the model would be. Examples of physics theories include general relativity, quantum mechanics, statistical mechanics, and the Big Bang.

Physics is problem solving

Once you have a theory, you can apply it to make predictions about a system given some initial state that describes it. For example, you might start with a projectile in a vacuum with some velocity at some location, and you try to figure out what its maximum height would be. This is the thing that actually shows up on physics exams. Sometimes you need to justify your knowledge of theory, but for the most part you demonstrate implicit knowledge of a thoery by using common techniques for describing the evolution of systems. Examples of techniques for solving physics problems are free body diagrams, Feynman diagrams, spacetime diagrams, the Schroedinger equation, and partition functions.

Physics is a science

The thing that sets physics apart from chess is that it describes the real world. You want to prove this. So y...
Interpretability: Integrated Gradients is a decent attribution method
by Lucius Bushnaq — 2024-05-20T17:55:22.893Z

A short post laying out our reasoning for using integrated gradients as attribution method. It is intended as a stand-alone post based on our LIB papers [1] [2]. This work was produced at Apollo Research.

Context

Understanding circuits in neural networks requires understanding how features interact with other features. There's a lot of features and their interactions are generally non-linear. A good starting point for understanding the interactions might be to just figure out how strongly each pair of features in adjacent layers of the network interacts. But since the relationships are non-linear, how do we quantify their 'strength' in a principled manner that isn't vulnerable to common and simple counterexamples? In other words, how do we quantify how much the value of a feature in layer l+1 should be attributed to a feature in layer l?

This is a well-known sort of problem originally investigated in cooperative game theory. A while ago it made its way into machine learning, where people were pretty interested in attributing neural network outputs to their inputs for a while. Lately it's made its way into interpretability in the context of attributing variables in one hidden layer of a neural network to another.

Generally, the way people go about this is setting up a series of 'common-sense' axioms that the attribution method should fulfil in order to be self-consistent and act like an attribution is supposed to act. Then they try to show that there is one unique method that satisfies these axioms. Except that (a) people disagree about what axioms are 'common-sense', and (b) the axioms people maybe agree most on don't quite single out a single method as unique, just a class of methods called path attributions. So no attribution method has really been generally accepted as the canonical 'winner' in the ml context yet. Though some methods are certainly more popular than others.

Integrated Gradients

Integrated gradients is a computationally efficient attribution method (compared to activation patching / ablations) grounded in a series of axioms. It was originally proposed the context of economics (Friedman 2004), and recently used to attribute neural networks outputs to their inputs(Sundararajan et al. 2017). Even more recently, they started being used for internal feature attribution as well (Marks et al. 2024, Redwood Research (unpublished) 2022).

Pro

...
The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks
by Lucius Bushnaq — 2024-05-20T17:53:25.985Z

This is a linkpost for our two recent papers:

An exploration of using degeneracy in the loss landscape for interpretability https://arxiv.org/abs/2405.10927
An empirical test of an interpretability technique based on the loss landscape https://arxiv.org/abs/2405.10928

This work was produced at Apollo Research in collaboration with Kaarel Hanni (Cadenza Labs), Avery Griffin, Joern Stoehler, Magdalena Wache and Cindy Wu. Not to be confused with Apollo's recent Sparse Dictionary Learning paper.

A key obstacle to mechanistic interpretability is finding the right representation of neural network internals. Optimally, we would like to derive our features from some high-level principle that holds across different architectures and use cases. At a minimum, we know two things:

We know that the training loss goes down during training. Thus, the features learned during training must be determined by the loss landscape. We want to use the structure of the loss landscape to identify what the features are and how they are represented.
We know that models generalize, i.e. that they learn features from the training data that allow them to accurately predict on the test set. Thus, we want our interpretation to explain this generalization behavior.

Generalization has been linked to basin broadness in the loss landscape in several ways, most notably including singular learning theory, which introduces the learning coefficient as a measure of basin broadness that doubles as a measure of generalization error that replaces the parameter count in Occam's razor.

Inspired by both of these ideas, the first paper explores using the structure of the loss landscape to find the most computationally natural representation of a network. We focus on identifying parts of the network that are not responsible for low loss (i.e. degeneracy), inspired by singular learning theory. These degeneracies are an obstacle for interpretability as they mean there exist parameters which do not affect the input-output behavior in the network (similar to the parameters of a Transformer WV and WO matrices that do not affect the product WOV).

We explore 3 different ways neural network parameterisations can be degenerate:

when activations are linearly dependent
when gradient vectors are linearly dependent
when ReLU neurons fire on the same inputs.

This investigation leads to the interaction basis, a...
NAO Updates, Spring 2024
by jefftk — 2024-05-20T16:51:03.693Z
Link: https://naobservatory.org/blog/updates-spring-2024

Now that the NAO blog is up, we’re taking the opportunity to post some written updates on the work our team has done over the past ~6 months. We’re hoping to make similar updates something like quarterly. Since this post covers a longer period it’s a bit longer than we expect future ones will be. If anything here is particularly interesting or if you’re working on similar problems, please

reach out

!

Wastewater Sequencing

In the fall & winter we partnered with

CDC’s Traveler-based Genomic Surveillance

program and

Ginkgo Biosecurity

to collect and sequence paired wastewater samples of aggregated airplane lavatory waste and municipal treatment plant influent. Initial sequencing is complete, and we have banked nucleic acids for additional sequencing. We have continued processing weekly treatment plant samples and banking the nucleic acids

Developing a good approach for extracting the nucleic acids from these samples took a lot of iteration. Wastewater is a challenging sample type, with a complex and variable composition. We experimented with different concentration methods, DNA/RNA extraction kits, dissociation reagents, and filters, looking for a protocol that would optimize for viruses relative to bacteria while giving sufficient yield and with a series of steps that were feasible for a daily processing. We also needed to adjust our protocols to handle settled solids (“primary sludge”) and airplane lavatory waste in addition to influent. We’ve published all three protocols (

influent

,

sludge

,

airplane waste

) to protocols.io.

We’ve sequenced a subset of these samples at MIT’s

BioMicroCenter

, using their standard protocols for bulk RNA library preparation. We think a more custom protocol would likely give significantly better results, and also want more in-depth understanding and control of how exactly our sequencing libraries are produced. So we’re very excited to be collaborating with experts at the Broad Institute’s

Sabeti Lab

on adapting some of their custom MGS protocols to wastewater. Developing in-house library prep expertise will also allow us to use the Broad’s

walk-up sequencing service

, which by using newer sequencers is much cheaper per read than the BMC’s offering but requires ready-to-go libraries.

We’re also collaborating with

Marc Johnson

and

Clayton Rushford

at the University of Missouri and

Jason Rothman

in

Katrine Whiteson’s lab

at the Universit...
OpenAI: Exodus
by Zvi — 2024-05-20T13:10:03.543Z

Previously: OpenAI: Facts From a Weekend, OpenAI: The Battle of the Board, OpenAI: Leaks Confirm the Story, OpenAI: Altman Returns, OpenAI: The Board Expands.

Ilya Sutskever and Jan Leike have left OpenAI. This is almost exactly six months after Altman’s temporary firing and The Battle of the Board, the day after the release of GPT-4o, and soon after a number of other recent safety-related OpenAI departures. Many others working on safety have also left recently. This is part of a longstanding pattern at OpenAI.

Jan Leike later offered an explanation for his decision on Twitter. Leike asserts that OpenAI has lost the mission on safety and culturally been increasingly hostile to it. He says the superalignment team was starved for resources, with its public explicit compute commitments dishonored, and that safety has been neglected on a widespread basis, not only superalignment but also including addressing the safety needs of the GPT-5 generation of models.

Altman acknowledged there was much work to do on the safety front. Altman and Brockman then offered a longer response that seemed to say exactly nothing new.

Then we learned that OpenAI has systematically misled and then threatened its departing employees, forcing them to sign draconian lifetime non-disparagement agreements, which they are forbidden to reveal due to their NDA.

Altman has to some extent acknowledged this and promised to fix it once the allegations became well known, but so far there has been no fix implemented beyond an offer to contact him privately for relief.

These events all seem highly related.

Also these events seem quite bad.

What is going on?

This post walks through recent events and informed reactions to them.

The first ten sections address departures from OpenAI, especially Sutskever and Leike.

The next five sections address the NDAs and non-disparagement agreements.

Then at the end I offer my perspective, highlight another, and look to paths forward.

Table of Contents

The Two Departure Announcements

Who Else Has Left Recently?

Who Else Has Left Overall?

Early Reactions to the Departures

The Obvious Explanation: Altman

Jan Leike Speaks

Reactions After Lekie’s Statement

Greg Brockman and Sam Altman Respond to Leike

Reactions from Some Folks Unworried About Highly Capable AI

Don’t Worry, Be Happy?

The Non-Disparagement and NDA Clauses

Legal

...
Infra-Bayesian haggling
by hannagabor — 2024-05-20T12:23:30.165Z

Preface

I wrote this post during my scholarship at MATS. My goal is to describe a research direction of the learning theoretic agenda (LTA). Namely, a natural infra-Bayesian learning algorithm proposal that arguably leads to Pareto-optimal solutions in repeated games. The idea originates from Vanessa, I'm expanding a draft of her into a more accessible description.

The expected audience are people who are interested in ongoing work on LTA. It is especially suitable for people who are looking for a research direction to pursue in this area.

Introduction

There has been much work on the theory of agents who cooperate in the Prisoner's dilemma and other situations. Some call this behavior

superrationality

. For example,

functional decision theory (FDT) is a decision theory that prescribes such behavior. The strongest results so far are those coming from "modal combat", e.g. Critch. However, these results are of very limited scope: among other issues, they describe agents crafted for specific games, rather than general reasoners that produce superrational behavior in a "naturalized" manner (i.e. as a special case of the general rules of reasoning.)

At the same time, understanding multi-agent learning theory is another major open problem. Attempts to prove convergence to game-theoretic solution concepts (a much weaker goal than superrationality) in a learning-theoretic setting are impeded by the so-called grain-of-truth problem (originally observed by Kalai and Lehrer, its importance was emphasized by Hutter). An agent can learn to predict the environment via Bayesian learning only if it assigns non-zero prior probability to that environment, i.e. its prior contains a grain of truth. What's the grain-of-truth problem? Suppose Alice's environment contains another agent, Bob. If Alice is a Bayesian agent, she can learn to predict Bob's behavior only if her prior assigns a positive probability to Bob's behavior. (That is, Alice's prior contains a grain of truth.) If Bob has the same complexity level as Alice, then Alice is not able to represent all possible environments. Thus, in general, Alice's prior doesn't contain a grain of truth. A potential solution based on "reflective oracles" was proposed by Leike, Taylor and Fallenstein. However it involves arbitrarily choosing a fixed point out of an enormous space of possibilities, and requires that all agents involved choose ...
Jaan Tallinn's 2023 Philanthropy Overview
by jaan — 2024-05-20T12:11:39.416Z
Link: https://jaan.info/philanthropy/#2023-results

to follow up my philantropic pledge from 2020, i've updated my philanthropy page with 2023 results.

in 2023 my donations funded $44M worth of endpoint grants ($43.2M excluding software development and admin costs) — exceeding my commitment of $23.8M (20k times $1190.03 — the minimum price of ETH in 2023).
D&D.Sci (Easy Mode): On The Construction Of Impossible Structures [Evaluation and Ruleset]
by abstractapplic — 2024-05-20T09:38:55.228Z

This is a followup to the D&D.Sci post I made last Friday; if you haven’t already read it, you should do so now before spoiling yourself.

Below is an explanation of the rules used to generate the dataset (my full generation code is available here, in case you’re curious about details I omitted), and their strategic implications.

Ruleset

Impossibility

Impossibility is entirely decided by who a given architect apprenticed under. Fictional impossiblists Stamatin and Johnson invariably produce impossibility-producing architects; real-world impossiblists Penrose, Escher and Geisel always produce architects whose works just kind of look weird; the self-taught break Nature's laws 43% of the time.

Cost

Cost is entirely decided by materials. In particular, every structure created using Nightmares is more expensive than every structure without them.

Strategy

The five architects who would guarantee an impossible structure are D, E, G, H, and K. Of these, G - and only G - intends to use Nightmares as construction material. The optimal choices given the Duke's stated preferences are therefore [D, E, H, K].

Reflections

This challenge was created with the intent of being egregiously easy and anomalously accessible. From the performances I saw, it looks like it fit the bill: congratulations to everyone who played publicly on reaching my intended solution. (Particular congratulations to aphyer for providing said solution within an hour of me posting the challenge, and to new player Lorxus for managing to correctly identify every named impossiblist on their way to a perfect answer.)

I hope this scenario managed to be fun despite - or because of? - the simplicity and lack of greater point. If you liked it, in lieu of correctly attributing thanks, please be slightly kinder to everyone you meet: after all, you have no way of being certain any given person wasn't my sponsor. (Conversely, if you disliked it, please wander the streets of your nearest metropolis glowering at random strangers until you feel karmic balance has been restored.)

(. . . also, please share your opinions with me. I find myself very curious about how/if players think I should aim to run smaller/lower-difficulty games going forward.)

Scheduling

Unless someone gives me a reason not to, I (tentatively, falliably) plan to run my next scenario from Friday 7th June to Monday 17th June. We'll see if reality plays along.