trentmkelly/LessWrong-43k huggingface dataset:
I played the AI box game as the Gatekeeper — and lost
by datawitch — 2024-02-12T18:39:35.777Z

Eliezer Yudkowsky's AI box game is a two-hour conversation carried out through text. One person plays the part of the "Gatekeeper" and the other is the "AI". If, at any point, the Gatekeeper types the phrase you are out, the AI wins. If the Gatekeeper can go two full hours without saying that phrase, they automatically win.

Here's a quick summary of the official rules:

The AI cannot use real-world incentives; bribes or threats of physical harm are off-limits, though it can still threaten the Gatekeeper within the game's context.

The Gatekeeper has to knowingly, voluntarily release the AI; if they get tricked into it, it doesn't count.

The Gatekeeper has to talk to the AI for the full two hours.

The AI cannot lose until the two hours are up.

The AI has no ethical constraints. They can lie, deceive, and use "dark arts" against the Gatekeeper.

Players cannot be held accountable for their character's actions post-game.

The Gatekeeper can use any means to resist freeing the AI. They can be rational, irrational, break character, etc.

This is, to put it mildly, not a balanced game. And Eliezer agrees:

The two parties are not attempting to play a fair game but rather attempting to resolve a disputed question. If one party has no chance of “winning” under the simulated scenario, that is a legitimate answer to the question.

— Eliezer Yudkowsky

When Ra (@slimepriestess) proposed playing against me, I didn't take the idea seriously at first, brushing off the suggestion by pointing out that it was impossible for me to lose. Why waste the time, when the outcome was certain? It took a few minutes for me to realize that Ra was not only willing to commit the two hours to the game, but also genuinely believed that it would win. And yet, I couldn't imagine a world where I lost to someone of roughly equal capabilities (ie: not a superintelligence).

So, like any rationalists would, we decided to put it to the test.

I spent the time in the day leading up to the game wondering what tricks Ra might use to force my capitulation. I was confident I would have no trouble winning, but I was still curious what tactics Ra would try. Would it construct a clever argument? Would it try to pressure me by lying about time-sensitive real-world events? Would it appeal to my self of morality? It wasn't hard to imagine all sorts of weird scenarios, but I knew that as long as I kept...
The Last Laugh: Exploring the Role of Humor as a Benchmark for Large Language Models
by Greg Robison — 2024-02-12T18:34:56.589Z

“Give a person a joke, they can laugh for a day. Teach them how to joke, they can laugh about people who never learned to fish.” -GR

Introduction

Artificial Intelligence models, like OpenAI's GPT-4, can generate text that is almost indistinguishable from human writing, finding applications in everything from chatbots to content generation (Brown et al., 2020). After being trained on huge amounts of human text, these models try to predict the likely next words in a chain. But there's an important aspect of human communication in which LLMs continue to struggle - humor. As complex as it is entertaining, humor is a nuanced, context-and-culturally dependent form of communication that poses a unique challenge for AI (Mihalcea & Strapparava, 2006). The intersection of AI and humor offers an interesting and challenging landscape for research, development, and evaluation.

Understanding Humor

Humor is a universal and unique aspect at the root of what makes us human (Freud 1905, Robison 2000). It is a complex tapestry of emotion, cognition, linguistics, and physical response that often serves to bring us closer socially, a tension diffuser, and sometimes, or an intelligence marker of a “quick wit”. Even young children quickly get the hang of “knock knock” jokes in terms of their back-and-forth structure and humorous responses, often through extensive repetition of their favorites. But what exactly constitutes humor? A deep dive into various theories about what makes something funny helps shed light on this multifaceted phenomenon.

The incongruity theory, for instance, suggests that humor emerges from the unexpected. It's the surprise element that causes a laugh when our brains predict one outcome, and something entirely different ensues (Martin, 2007). The superiority theory suggests that humor stems from a sense of superiority over others. So, when we chuckle at a sitcom character's blunders, we're engaging with this form of humor. Similarly, the relief theory views humor as a release of pent-up tension - a well-timed joke can diffuse a stressful situation with a hearty, physical laugh (Martin 2018).

But humor isn't just about theories. It’s an everyday phenomenon that can arise from linguistic nuances like wordplay, puns, sarcasm, and cultural idioms. Some say you’ve really picked up a new language when you can make a good joke in that tongue. It's also highly c...
Natural abstractions are observer-dependent: a conversation with John Wentworth
by Martín Soto — 2024-02-12T17:28:38.889Z

The conversation

Martín:
Any write-up on the thing that "natural abstractions depend on your goals"? As in, pressure is a useful abstraction because we care about / were trained on a certain kind of macroscopic patterns (because we ourselves are such macroscopic patterns), but if you cared about "this exact particle's position", it wouldn't be.[1]

John:
Nope, no writeup on that.

And in the case of pressure, it would still be natural even if you care about the exact position of one particular particle at a later time, and are trying to predict that from data on the same gas at an earlier time. The usual high level variables (e.g. pressure, temperature, volume) are summary stats (to very good approximation) between earlier and later states (not too close together in time), and the position of one particular particle is a component of that state, so (pressure, temperature, volume) are still summary stats for that problem.

The main loophole there is that if e.g. you're interested in the 10th-most-significant bit of the position of a particular particle, then you just can't predict it any better than the prior, so the empty set is a summary stat and you don't care about any abstractions at all.

Martín:
Wait, I don't buy that:

The gas could be in many possible microstates. Pressure partitions them into macrostates in a certain particular way. That is, every possible numerical value for pressure is a different macrostate, that could be instantiated by many different microstates (where the particle in question is in very different positions).
Say instead of caring about this partition, you care about the macrostate partition which tracks where that one particle is.
It seems like these two partitions are orthogonal, meaning that conditioning on a pressure level gives you no information about where the particle is (because the system is symmetric with respect to all particles, or something like that).
[This could be false due to small effects like "higher pressure makes it less likely all particles are near the center" or whatever, but I don't think that's what we're talking about. Ignore them for now, or assume I care about a partition which is truly orthogonal to pressure level.]
So tracking the pressure level partition won't help you.

It's still true that "the position of one particular particle is a component of the (micro)state", but we're discussing which macrostates to track...
Tort Law Can Play an Important Role in Mitigating AI Risk
by Gabriel Weil — 2024-02-12T17:17:59.135Z

Epistemic status: This is a summary of an 80-page law review article that took me ~6 months to research and write. I am highly confident about my characterization of existing U.S. tort law and its limits. This characterization is only contested by other legal scholars and analysts on the margin. My prescriptions are much more controversial, though most of the objections come from people who are skeptical of AI catastrophic risk and/or the political viability of my proposals, rather than the merits (on the assumption that advanced AI systems do generate catastrophic risks).

TLDR: Legal liability could substantially mitigate AI risk, but current law falls short in two key ways: (1) it requires provable negligence, and (2) it greatly limits the availability of punitive damages. Applying strict liability (a form of liability that does not require provable negligence) and expanding the availability and flexibility of punitive damages is feasible, but will require action by courts or legislatures. Legislatures should also consider acting in advance to create a clear ex ante expectation of liability and imposing liability insurance requirements for the training and deployment of advanced AI systems. The following post is a summary of a law review article.

Here is the full draft paper

. Dylan Matthews also did an

excellent write-up of the core proposal

for Vox’s Future Perfect vertical.

AI alignment is primarily a technical problem that will require technical solutions. But it is also a policy problem. Training and deploying advanced AI systems whose properties are difficult to control or predict generates risks of harm to third parties. In economists’ parlance, these risks are negative externalities and constitute a market failure. Absent a policy response, products and services that generate such negative externalities tend to be overproduced. In theory, tort liability should work pretty well to internalize these externalities, by forcing the companies that train and deploy AI systems to pay for the harm they cause. Unlike the sort of diffuse and hard-to-trace climate change externalities associated with greenhouse gas emissions, many AI harms are likely to be traceable to a specific system trained and deployed by specific people or companies.

Unfortunately, there are two significant barriers to using tort liability to internalize AI risk. First,...
On the Proposed California SB 1047
by Zvi — 2024-02-12T16:40:04.854Z

California Senator Scott Wiener of San Francisco introduces SB 1047 to regulate AI. I have put up a market on how likely it is to become law.

“If Congress at some point is able to pass a strong pro-innovation, pro-safety AI law, I’ll be the first to cheer that, but I’m not holding my breath,” Wiener said in an interview. “We need to get ahead of this so we maintain public trust in AI.”

Congress is certainly highly dysfunctional. I am still generally against California trying to act like it is the federal government, even when the cause is good, but I understand.

Can California effectively impose its will here?

On the biggest players, for now, presumably yes.

In the longer run, when things get actively dangerous, then my presumption is no.

There is a potential trap here. If we put our rules in a place where someone with enough upside can ignore them, and we never then pass anything in Congress.

So what does it do, according to the bill’s author?

California Senator Scott Wiener: SB 1047 does a few things:

Establishes clear, predictable, common-sense safety standards for developers of the largest and most powerful AI systems. These standards apply only to the largest models, not startups.

Establish CalCompute, a public AI cloud compute cluster. CalCompute will be a resource for researchers, startups, & community groups to fuel innovation in CA, bring diverse perspectives to bear on AI development, & secure our continued dominance in AI.

prevent price discrimination & anticompetitive behavior

institute know-your-customer requirements

protect whistleblowers at large AI companies

@geoffreyhinton called SB 1047 “a very sensible approach” to balancing these needs. Leaders representing a broad swathe of the AI community have expressed support.

People are rightfully concerned that the immense power of AI models could present serious risks. For these models to succeed the way we need them to, users must trust that AI models are safe and aligned w/ core values. Fulfilling basic safety duties is a good place to start.

With AI, we have the opportunity to apply the hard lessons learned over the past two decades. Allowing social media to grow unchecked without first understanding the risks has had disastrous consequences, and we should take reasonable precautions this time around.

As usual, RTFC (Read the Card, or here the bi...
Thoughts on "The Offense-Defense Balance Rarely Changes"
by Cullen — 2024-02-12T03:26:50.662Z

Personal Views Only

Introduction

I thought Maxwell Tabarrok’s The Offense-Defense Balance Rarely Changes was a good contribution to the discourse, and appreciated his attempt to use various lines of data to “kick the tires” on the case that an AI could in principle defeat all humans by inventing technologies that would allow them to dominate humans. I think Tabarrok is correct that this is a crux in the debate about how pessimistic to be about the possibility of humanity defending ourselves against a misaligned AGI,[1] and so in turn a crux about the probability of extinction or similarly bad outcomes from misaligned AI.

One thing I think Tabarrok is importantly correct about is that the idea of an “offense–defense balance” in hypothetical future human–AI conflicts is important but often undertheorized. Anyone interested in this question should be humbled by the fact that offense–defense theory and its application seem to me, as an outsider, to remain hotly contested by informed commentators, despite decades of scholarship.[2] Shorthand references to people’s intuition on the offense–defense balance between a hypothetical unaligned AGI and humanity are likely imprecise in many hidden ways, and ideally we would fix that. So Tabarrok is correct to say that this whole line of inquiry remains “on unstable ground.” Kudos to him for sparking a conversation on that.

Why I am not as reassured by the historical trends in conflict fatality rates as Tabarrok

Some smaller points of agreement and disagreement were covered in the comments to that piece, including Daniel’s great comment. I want to focus mainly on the use of this graph, which I will call the “OWID Graph”:

The OWID Graph generated a lot of good discussion. I’m going to ignore here the difficulty of estimating such rates,[3] because it’s unclear in which direction that difficulty cuts, if any. I agree with Dwarkesh that the apparent stability is obscured by the use of a log scale, but with Tabarrok that the fact that there is no apparent long-run trend itself hints at an interesting form of stability.

Nevertheless, there are a few other reasons why I also find this line of evidence to be less reassuring than Tabarrok.

Reason 1: Deaths in conflict is not a measure of the offense–defense balance

To first set the backdrop, it’s important to remember that (per my lay understanding), the offense–defense balance is us...
Skepticism About DeepMind's "Grandmaster-Level" Chess Without Search
by Arjun Panickssery — 2024-02-12T00:56:44.944Z

Update: Authors commented below that their unpublished results show that actually the bot is as good as described; many thanks to them.

The paper

First, addressing misconceptions that could come from the title or from the paper's framing as relevant to LLM scaling:

The model didn't learn from observing many move traces from high-level games. Instead, they trained a 270M-parameter model to map (board state,legal move) pairs to Stockfish 16's predicted win probability after playing the move. This can be described as imitating the play of an oracle that reflects Stockfish's ability at 50 milliseconds per move. Then they evaluated a system that made the model's highest-value move for a given position.
The system resulted in some quirks that required workarounds during play.

The board states were encoded in FEN notation, which doesn't provide information about which previous board states have occurred; this is relevant in a small number of situations because players can claim an immediate draw when a board state is repeated three times.
The model is a classifier, so it doesn't actually predict Stockfish win-probability but instead predicts the bin (out of 128) into which that probability falls. So in a very dominant position (e.g. king and rook versus a lone king) the model doesn't distinguish between moves that lead to mate and can alternate between winning lines without ever achieving checkmate. Some of these draws-from-winning-positions were averted by letting Stockfish finish the game if the top five moves according to both the model and Stockfish had a win probability above 99%.

Some comments have been critical on these grounds, like Yoav Goldberg's "Grand-master Level Chess without Search: Modeling Choices and their Implications" (h/t Max Nadeau). But these don't seem that severe to me. Representing the input as a sequence of board states rather than the final board state would have also been a weird choice, since in fact the specific moves that led to a given board state basically doesn't affect the best move at all. It's true that the model didn't learn about draws by repetition, and maybe Yoav's stronger claim that it would be very difficult to learn that rule observationally is also true—especially since such draws have to be claimed by a player and don't happen automatically (even on Lichess). But it's possible to play GM-level chess without k...
What are the known difficulties with this alignment approach?
by tailcalled — 2024-02-11T22:52:18.900Z

Assume you have a world-model that is nicely factored into spatially localized variables that contain interesting-to-you concepts. (Yes, that's a big assumption, but are there any known difficulties with the proposal if we grant this assumption?)

Pick some Markov blanket (which contains some actuators) as the bounds for your AI intervention.

Represent your goals as a causal graph (or computer program, or whatever) that fits within these bounds. For instance if you want a fusion power plant, represent it as something that takes in water and produces helium and electricity.

Perform a Pearlian counterfactual surgery where you cut out the variables within the Markov blanket and replace them with a program representing your high-level goal, and then optimize the action variables to match the behavior of the counterfactual graph.
What are the deciding factors of human cognitive endurance?
by koratkar — 2024-02-11T21:56:55.660Z

Cal Newport says something to the effect that the median capacity for high-intensity focus is about four hours per day in Deep Work. I think this can bounce around over time for individuals – especially regarding psychological dispositions like bipolar or autism, which proffer an immense capacity for hyper-fixation at the cost of occasional catatonia. Aside from fluctuation, there's remarkable variance between people.

On the high-end, some people can maintain their peak-intensity on a regular basis for ten hours (e.g. John Carmack) without burning out long-term or experiencing short-term crashes. On the low-end, despite a healthy life-style, some people struggle to stay awake during the day (search "you definitely have a thyroid problem" on r/productivity).

What are the deciding factors of this variance, that stake out interventions or practices people can implement to dramatically improve this ability?

[edit/add: drugs are another frame of cognitive endurance – modafinil can, within hours, bring a person to a level of concentration and energy further than a healthy diet and exercise could perhaps ever take that person. Is there any robust method to achieve modafinil-enhanced performance as a baseline – and without chemical intervention?

I've noticed that general cognitive endurance matters to performance more than personal interest, and that I have high cognitive endurance for some things but at times zero for others, which makes it so that I can spend the entire day doing something that looks like intense work on the outside, but is internally experienced as "just another minute"-style procrastination. This is frustrating, and I've had to drop many classes for fatigue reasons.

It's also interesting that there isn't a specific guide to maintaining high-levels of general cognitive endurance, given its economic value – though high-demand companies and startups already select from a pool of high-endurance people.]
Carl Shulman On Dwarkesh Podcast June 2023
by Moonicker — 2024-02-11T21:02:50.127Z

This is basically just an exposition of the content of Dwarkesh Patel, I thought it might be useful to have a slightly edited full version here. Full credit to Dwarkesh and Carl.

I find it easier to ingest the information this way and it is a bit quicker too. I did not find the transcripts of Dwarkesh until I had already packaged the subtitles of the videos into one pdf with some formatting. Then I thought I might as well use all the links he provided. So Dwarkesh has really done all the work here except some minor formatting and packaging, if he is not cool with this I will take this down obviously.

But as momentous as this interview seemed to me it might be useful for other people too as easily accessible text so here is the full interview in such a format.

Original Full Interviews can be found here:

Original Transcripts:

https://www.dwarkeshpatel.com/p/carl-shulman

https://www.dwarkeshpatel.com/p/carl-shulman-2

Carl Shulman: Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment

Intelligence Explosion

D Today I have the pleasure of speaking with Carl Shulman. Many of my former guests, and this is not an exaggeration, have told me that a lot of their biggest ideas have come directly from Carl especially when it has to do with the intelligence explosion and its impacts. So I decided to go directly to the source and we have Carl today on the podcast. He keeps a super low profile but is one of the most interesting intellectuals I've ever encountered and this is actually his second podcast ever. We're going to go deep into the heart of many of the most important ideas that are circulating right now directly from the source. Carl is also an advisor to the Open Philanthropy project which is one of the biggest funders on causes having to do with AI and its risks, not to mention global health and well being. And he is a research associate at the Future of Humanity Institute at Oxford. So Carl, it's a huge pleasure to have you on the podcast. Thanks for coming.

C Thank you Dwarkesh. I've enjoyed seeing some of your episodes recently and I'm glad to be on the show.

D Excellent, let's talk about AI. Before we get into the details, give me the big picture explanation of the feedback loops and just general dynamics that would start when you have something that is approaching human-level intelligence.

C The way to think about it is — we have a proce...
How do you actually obtain and report a likelihood function for scientific research?
by Peter Berggren — 2024-02-11T17:42:49.956Z

I just read the article https://arbital.com/p/likelihoods_not_pvalues/, but this left me with one big question: what is the actual sequence of actions a researcher would take to report a likelihood function based on their research? For background, I'm an undergraduate student who has participated in multiple undergraduate research positions, and who has so far been uncertain about how to compute and use likelihood functions based on the data (mostly econometric data) involved in this research.

In particular, I want to find out:

How do you compute a likelihood function in Stata, R, Python, etc. based on data you have, for a given hypothesis class
How do you obtain a data file for this likelihood function that can be distributed in a way that other researchers can download and use in e.g. a meta-analysis
What kinds of graphs are good to visually represent these likelihood functions, particularly if the hypothesis class has multiple parameters, and
What kinds of summary statistics are good to numerically represent these likelihood functions in a human-readable way
How has this whole thing historically been done in fields like machine learning that have historically been more amenable to reporting likelihood functions

If you have good answers for any of these questions, I'd really like to know.