trentmkelly/LessWrong-43k huggingface dataset:
AI Safety Newsletter #42: Newsom Vetoes SB 1047 Plus, OpenAI’s o1, and AI Governance Summary
by Corin Katzke — 2024-10-01T20:35:32.399Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-42-newsom-vetoes

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Listen to the AI Safety Newsletter for free on Spotify or Apple Podcasts.

Subscribe here to receive future versions.

Newsom Vetoes SB 1047

On Sunday, Governor Newsom vetoed California’s Senate Bill 1047 (SB 1047), the most ambitious legislation to-date aimed at regulating frontier AI models. The bill, introduced by Senator Scott Wiener and covered in a previous newsletter, would have required AI developers to test frontier models for hazardous capabilities and take steps to mitigate catastrophic risks. (CAIS Action Fund was a co-sponsor of SB 1047.)

Newsom states that SB 1047 is not comprehensive enough. In his letter to the California Senate, the governor argued that “SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data.” The bill requires testing for models that use large amounts of computing power, but he says “by focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security.”

Sponsors and opponents react to the veto. Senator Wiener released a statement calling the veto a "missed opportunity" for California to lead in tech regulation and stated that “we are all less safe as a result.” Statements from other sponsors of the bill can be found here. Meanwhile, opponents of the bill, such as venture capitalist Marc Andreessen celebrated the veto. Major Newsom donors such as Reid Hoffman and Ron Conway, who have financial interests in AI companies, also celebrated the veto.

OpenAI’s o1

OpenAI recently launched o1, a series of AI models with advanced reasoning capabilities. In this story, we explore o1’s capabilities and their implications for scaling and safety. We also cover funding and governance updates at OpenAI.

o1 models are trained with reinforcement learning to perform complex reasoning. The models are trained to produce long, hidden chains of thought before responding to the user. This allows them to break down hard problems into simpler steps, notice and correct its own mistakes, and test different problem-solving approaches.

o1 models outperform previous models—and sometimes rival human experts—in solving hard p...
Retrieval Augmented Genesis
by João Ribeiro Medeiros — 2024-10-01T20:18:01.836Z

a prototype and some thoughts on semantics

Before reading this article I strongly encourage you to Checkout the RAGenesis App!
Full code available at https://github.com/JoaoRibeiroMedeiros/RAGenesisOSS.

Screenshot of Verse Uni Verse page

Project Goals

Send a message of unity and mutual understanding between different cultures through a “generative book” compiling the wisdom of many different traditions.
Develop a framework for transparency of embedded knowledge bases and evaluation of main messaging in a RAG application.
Define Semantic Similarity Networks as special case of Semantic Networks and derive Main Chunk methodology based on elementary Graph theory.
Create a showcase of my technical capacity as an AI engineer, data scientist, product designer, software developer and writer.
Make something cool.

Introduction

Having led the development of several software products based on RAG engines for the last year, I have been consistently investigating the scope of semantic similarity for a plethora of different use cases.

Turns out this exercise got me wondering about some of the more fundamental aspects of what AI-driven semantic similarity can accomplish, in the years to come.

I figured that through semantic similarity algorithms the current AI engines have opened up a pathway to explore texts and documentation in a radically innovative way, which has implications for education, research and everything that stems from those two foundational areas of human activity.

In this article, I present ragenesis.com platform and explore the thought process behind its development, which I have accomplished as a solo full-stack AI engineering, software development and data science project. Beyond that, I also used this exercise as a platform to collect thoughts around the importance of RAG tools, and how there is still a lot of opportunity when it comes to the value that this kind of application can produce.

Finally, I showcase some results for the semantic similarity network associated with the knowledge base I built for this project, centered around the verses of some of the greatest books of all time: the Torah, the Bhagavad Gita, the Analects, the Bible and the Quran.

See diagrams for main messaging of the Bible (New Testament), the Analects, the Quran, the Bhagavad Gita and the Torah. The concepts used to produce these diagrams will be unraveled in the sections to come,

...
Likelihood calculation with duobels
by Martin Gerdes — 2024-10-01T16:21:01.268Z

The post An Intuitive Explanation of Bayes's Theorem explains that using Bayesian reasoning we can much better see how likely something is (by calculating conditional probabilities), and thus update our own beliefs as efficiently as possible.
However, the essay shows us how to calculate those probabilities using lots of math (duh!). Which is fine if you just want to grasp the concept, or need to evaluate a medical study or something.

But what about applying it in our daily lives? Nobody has time for divisions while evaluating evidence...
The article gives 2 hints to help. The first using odds ratios, the second using decibels. But odds ratios keep getting multiplied and thus quickly get unwieldy, while decibels would be great, if not for the fact that I have no clue what the log10 values of different ratios are, and I really dislike the thought of memorizing them.

I think I can offer a more intuitive way to approach this. The key insight is: We don't need exact probabilities. In daily life, we are not in the position of a doctor calculating precise likelihoods from study data. We are taking gut level probabilities and want to continue reasoning with the results. So all we need are ballpark numbers anyway. We want to know whether something is 8% or 80% likely. The exact percentages are just noise though, our intuition isn't that precise.

I propose to encode the likelihood information using duobels, rather than decibels. Log2 instead of log10. Then every change of that number by one halves or doubles one side of the odds ratio.
The table for that would look like this:

duobel	odds ratio	ratio	%
-2	1:4	1/5	20
-1	1:2	1/3	33
0	1:1	1/2	50
+1	2:1	2/3	67
+2	4:1	4/5	80

(For in between values, we can use n.6 (.585) to get a x1.5 instead of a x2 increase: 0.6 is 3:2, -0.6 is 2:3, 1.6 is 3:1, and so on.
n.2 is 1.15, n.4 is 1.3, n.8 is 1.75 -> this is in most cases already too precise for daily needs)

All we need to remember to calculate the ratios in our head is this:

0 is 1:1, even odds, a coin-flip
positive numbers have odds 2n:1, negative numbers have odds 1:2|n|
the odds ratio x:y is equal to the division xx+y

That's it, we can translate any duobel number into an odds ratio or a ratio! And percentages for stuff like 2/3 we have already memorized, if we even need percentages in the first place.

Now let's talk likelihood shifts, where we can take advantage of the logarithmic measure of likelin...
Is Text Watermarking a lost cause?
by egor.timatkov — 2024-10-01T16:20:51.113Z

Summary:

A text watermark is some edit to a piece of text that makes its creator easy to identify. The main use for this would be to watermark Large Language Model outputs. This post explores how effective any text watermark could be in general. It does this by analyzing the smallest amount of text that a watermark could fit inside of. It is addressed to anyone working on watermarks, or interested in how feasible they could be. It provides a formula for testing watermark effectiveness and concludes that certain qualities that we would want from a watermark, especially robustness, significantly and negatively impact a watermark’s efficiency.

Introduction:

Ideally we want a watermark to meet three criteria

1) The watermark should be decisive: It should be almost impossible for normal text to appear marked by accident.
2) The watermark should be invisible, or at least not intrusive to someone just reading the text.
3) The watermark should be robust and able to withstand alterations to the text. Perhaps even from a malicious actor trying to destroy the watermark.

Suppose for a moment that watermarking text in this way was impossible. Perhaps it is always unreliable, or always circumventable. How would we know that this is the case? This post tries to examine a hypothetical text watermark by assuming the watermark consists of distinct changes peppered throughout the text called “markings.” It is assumed these “markings” can occur naturally in normal text, but occur more frequently in marked text. By simply counting the number of locations in the text that have been marked or could have been marked, we can determine the total number of markings that need to be embedded in the text in order for the text to be decisively marked. This is a proxy for how long the text needs to be, for any given watermark (after all, a better watermark is one that can be hidden in a smaller piece of text). In many cases, this required number of markings is reasonable, however it can get very high if you want your watermark to be resistant to tampering. This means that secure text watermarks- ones which require rewriting much of the text to undo, might either: be difficult to make, change the text so much that it is clearly no longer regular text, or require a very large amount of text to work.

Index:

Summary:

Introduction:

Index:

Trouble imagini...
Information dark matter
by Logan Kieller — 2024-10-01T15:05:41.159Z
Link: https://logankieller.substack.com/p/information-dark-matter

I started collecting a list of once “secret” documents that were, in one way or another, broadly released (whether it be through legal process, government disclosures, leaks, or the ideas themselves eventually turned into books):

The

analysis

done by Deutsche Bank investors to predict the 2008 financial crisis (dramatized as the Jenga scene from the movie “The Big Short”)
A

consulting project

completed by BCG for the city of Dallas on their loose dog problem
Deeply

insightful tech emails

surfaced through court cases written by people like Bill Gates, Jeff Bezos, Mark Zuckerberg, etc capturing some of their strategies, views, or ideas
The book,

Bulletproof Problem Solving

started as a professional development document within McKinsey and was eventually refined and published
MrBeast’s

guide

to content production
Netflix’s famous

culture document

, which Sheryl Sandberg has called one of the most important documents ever to come out of Silicon Valley
Facebook’s

little red book

With all of these examples, we can see that significant human capital is funneled into fields like tech, finance, and consulting (or law, startups, VC, government/statecraft, etc), and it is therefore unsurprising that much of the work produced there is made to be valuable. In fact, it is very valuable, but it is largely inaccessible to external audiences.

This also happens more broadly; in any human endeavor involving any intellectual effort at all.

The takeaway is that there is an incomprehensible amount of amazing work being done that is nearly, or even completely, impossible for others to find.

This is information dark matter.

It’s out there, and now we know it’s out there, but it is not something we can go and meaningfully interact with.

And while information created for internal company use is just one example, I began to consider where else we might find information dark matter and how we can make it accessible.

Why is this important?

It’s simple: with more information accessible to us, the more we can expand our intellectual arsenals for tackling bigger problems.

To solve this problem of information dark matter, the questions I hope to answer are:

What are the main types of information dark matter?
Where can they be found?
How can we more effectively make the information accessible?

To start answering the first ...
Conventional footnotes considered harmful
by dkl9 — 2024-10-01T14:54:01.732Z
Link: https://dkl9.net/essays/footnote_harm.html

Writers use footnotes — equally, endnotes — intending that they be optional for the reader. A note will hold a citation, technicality, or explanation, any of which is of interest to only some readers. This is a useful tactic, in principle.

Footnotes are indicated with ordinal symbols. A cue to a note may be a number, letter, or sequential symbol (commonly, *, †, ‡, etc). In any case, the reference only indicates where the note is, in the sequence of all notes present, rather than anything of the note's content.

So, if you wonder whether you'd care for the content of a note, you have to look at the note, switching to the bottom of the page and breaking your focus. Thus the notion that footnotes are optional is an illusion. The false option is even worse in the case of endnotes in printed works; there, to get to the note, you have to flip across many pages.

Good solutions exist, but are underused:

Decide, for each detail, whether to include it in the main text or leave it out entirely, rather than leaving some as "optional" in notes. I usually do this myself. Parentheses and em-dashes can help when a detail is hard to work in grammatically.
Include, in the cue to each note, a hint as to its content, besides just the ordinal pointer. A one-letter abbreviation, standardised thruout the work, may work well, e.g.:

"c" for citation supporting the marked claim
"d" for a definition of the marked term
"f" for further, niche information extending the marked section
"t" for a pedantic detail or technicality modifying the marked clause

Commit to only use notes for one purpose — say, only definitions, or only citations. State this commitment to the reader.
Newsom Vetoes SB 1047
by Zvi — 2024-10-01T12:20:06.127Z

It’s over, until such a future time as either we are so back, or it is over for humanity.

Gavin Newsom has vetoed SB 1047.

Newsom’s Message In Full

Quoted text is him, comments are mine.

To the Members of the California State Senate: I am returning Senate Bill 1047 without my signature.

This bill would require developers of large artificial intelligence (Al) models, and those providing the computing power to train such models, to put certain safeguards and policies in place to prevent catastrophic harm. The bill would also establish the Board of Frontier Models – a state entity – to oversee the development of these models.

It is worth pointing out here that mostly the ‘certain safeguards and policies’ was ‘have a policy at all, tell us what it is and then follow it.’ But there were some specific things that were requires, so Newsom is indeed technically correct here.

California is home to 32 of the world’s 50 leading Al companies, pioneers in one of the most significant technological advances in modern history. We lead in this space because of our research and education institutions, our diverse and motivated workforce, and our free-spirited cultivation of intellectual freedom. As stewards and innovators of the future, I take seriously the responsibility to regulate this industry.

Queue the laugh track. No, that’s not why California leads, but sure, whatever.

This year, the Legislature sent me several thoughtful proposals to regulate AI companies in response to current, rapidly evolving risks – including threats to our democratic process, the spread of misinformation and deepfakes, risks to online privacy, threats to critical infrastructure, and disruptions in the workforce. These bills, and actions by my Administration, are guided by principles of accountability, fairness, and transparency of AI systems and deployment of AI technology in California.

He signed a bunch of other AI bills. It is quite the rhetorical move to characterize those bills as ‘thoughtful’ in the context of SB 1047, which (like or hate its consequences) was by far the most thoughtful bill, was centrally a transparency bill, and was clearly an accountability bill. What you call ‘fair’ is up to you I guess.

SB 1047 magnified the conversation about threats that could emerge from the deployment of AI. Key to the debate is whether the threshold for regulat

...
Will AI and Humanity Go to War?
by Simon Goldstein — 2024-10-01T06:35:22.374Z

[This post is the introduction to my full paper, available here https://philpapers.org/rec/GOLWAA. This post was partially inspired by a LW comment thread between @Matthew Barnett and @Wei Dai.]

Abstract. This paper offers the first careful analysis of the possibility that AI and humanity will go to war. The paper focuses on the case of artificial general intelligence, AI with broadly human capabilities. The paper uses a bargaining model of war to apply standard causes of war to the special case of AI/human conflict. The paper argues that information failures and commitment problems are especially likely in AI/human conflict. Information failures would be driven by the difficulty of measuring AI capabilities, by the uninterpretability of AI systems, and by differences in how AIs and humans analyze information. Commitment problems would make it difficult for AIs and humans to strike credible bargains. Commitment problems could arise from power shifts, rapid and discontinuous increases in AI capabilities. Commitment problems could also arise from missing focal points, where AIs and humans fail to effectively coordinate on policies to limit war. In the face of this heightened chance of war, the paper proposes several interventions. War can be made less likely by improving the measurement of AI capabilities, capping improvements in AI capabilities, designing AI systems to be similar to humans, and by allowing AI systems to participate in democratic political institutions.

Keywords: AI safety, the bargaining model, information failures, power shifts, focal points

1. Introduction

Many in the AI safety community have worried that future AI systems may enter into strategic conflict with humanity. Such AI systems may be misaligned, so that their goals conflict with humanity’s. In addition, the collective power of such systems could match or exceed the power of humanity. In such a future, AI systems may go to war with humanity. Here, we would have two powerful parties vying for control of scarce resources. The two parties may have very different values and very different perspectives on how to achieve their goals.

While conceptually possible, this risk scenario has a blind spot: most conflicts do not end in war. War offers each party a chance of victory, but also comes with costs: some resources will be spent on guns that could have been spe...
AMA: International School Student  in China
by Novice — 2024-10-01T06:00:16.282Z

Exigence:
1. Observed US/Euro-centricity of LW in general.

Context:
1. Author's lack of much national allegiance(banal nationalism) towards any given nation-state(the term banal nationalism comes from a book of the same title)

Discussion Proposals:
1. International schools in general
2. Discussion about nationalism/cosmopolitanism in general
3. Discussion of the forum's inherent biases
[This is my first post, so please offer criticism.]
AGI Farm
by Rahul Chand — 2024-10-01T04:29:58.606Z

This post discusses Joe Carlsmith’s views on how to approach the problem of AI risk as interspecies interaction and how humans can use it navigate future AI development better. The essay is divided into three parts. First I give my understanding of Carlsmith's views, then I build upon some of his ideas by relating them to the field of superalignment and how the logical conclusion of his ideas might lead to scenarios where "too good" becomes bad.

The Prior - A brief summary of Joe Carlsmith’s views

In “Artificial Other” Carlsmith talks about two things

How to frame the discussion of potential AI risk as inter species interaction & the issues & opportunities that arise with the framing.
What our vast human experiences interacting with other species can teach us about what is to come.

In Carlsmith's views discussing AI risk becomes more accessible when framed as interaction with another species. This framing gives us more grounded context. Why? First, because we can relate it to our vast experiences interacting with different species on earth, and second because we can picture what our interactions with future super-intelligent AI's might look like by looking at the skewed power dynamics we see in our current interactions. e.g. with wildlife, with other humans (e.g. talking to a toddler), and in our popular culture (e.g. lot of fiction is based on what would happen if aliens landed on Earth). This framing however requires you to anthropize AI to an extent. How do you view it as an other species? It's easier to anthropize some future super AI, but what about current AIs? Is GPT-4 conscious or sentient? An example that Carlsmith talks about that I want to extend upon is that of a jellyfish. The jellyfish, an extremely simple organism with just two cell layers, is in many ways more 'alive' than GPT-4 or Claude, despite these AIs being much more intelligent. How do I anthropize an intelligent “other” which feels less alive than a jellyfish? Carlsmith touches on the weird foray we are getting ourselves into with

“I say all this partly because I want us to be prepared for just how confusing and complicated “AI Otherness” is about to get”

Foster’s Octopus or Herzog’s Bear? Yin or Yang?

In the rest of “Artificial other” & “Loving a World you don't trust” Carlsmith touches on topics ranging from what our experiences interacting with animals can tel...