trentmkelly/LessWrong-43k huggingface dataset:
Anthropic is being sued for copying books to train Claude
by Remmelt — 2024-08-31T02:57:27.092Z
Link: https://fingfx.thomsonreuters.com/gfx/legaldocs/gdvzmggwjpw/ANTHROPIC%20AUTHOR%20COPYRIGHT%20LAWSUIT%20complaint.pdf

OpenAI faces 10 copyright lawsuits and Anthropic is starting to get sued as well. Whether or not you agree with copyright, this is worth looking into. Lawsuits could hinder AI companies from scaling further.

The recent filing against Anthropic is notable because the plaintiffs have evidence of Anthropic copying their works. Because they were able to identify their data being trained on, the case against Anthropic is much stronger.

Here is the core argument from the filing:

In a December 2021 research paper on large language model training, Anthropic described creating a dataset “most of which we sourced from the Pile” and which included “32% internet books,” a code word in the industry for pirated copies of books available on the internet.

More recently, in July 2024, Anthropic has publicly acknowledged that it used The Pile to train its Claude models. As reported by Proof News, company spokesperson Jennifer Martinez “confirm[ed] use of the Pile in Anthropic’s generative AI assistant Claude.” Anthropic confirmed the same to Vox News. Independent researchers have tested Claude to shed light on the composition of its training set, and their work has confirmed a high likelihood that Claude was trained on copyrighted books.

Anthropic thus copied and exploited a trove of copyrighted books—including but not limited to the books contained in Books3—knowing that it was violating copyright laws. Instead of sourcing training material from pirated troves of copyrighted books from this modern-day Napster, Anthropic could have sought and obtained a license to make copies of them. It instead made the deliberate decision to cut corners and rely on stolen materials to train their models.

…

Anthropic, in taking authors’ works without compensation, has deprived authors of books sales and licensing revenues. There has long been an established market for the sale of books and e-books, yet Anthropic ignored it and chose to scrape a massive corpus of copyrighted books from the internet, without even paying for an initial copy.

Anthropic has also usurped a licensing market for copyright owners. In the last two years, a thriving licensing market for copyrighted training data has developed. A number of AI companies, including OpenAI, Google, and Meta, have paid hundreds of millions of dollars to obtain licenses to reproduce copyrighted material for LLM training. The

...
Book review: On the Edge
by PeterMcCluskey — 2024-08-30T22:18:39.581Z
Link: https://bayesianinvestor.com/blog/index.php/2024/08/30/on-the-edge/

Book review: On the Edge: The Art of Risking Everything, by Nate Silver.

Nate Silver's latest work straddles the line between journalistic
inquiry and subject matter expertise.

"On the Edge" offers a valuable lens through which to understand
analytical risk-takers.

The River versus The Village

Silver divides the interesting parts of the world into two tribes.

On his side, we have "The River" - a collection of eccentrics typified
by Silicon Valley entrepreneurs and professional gamblers, who tend to
be analytical, abstract, decoupling, competitive, critical,
independent-minded (contrarian), and risk-tolerant.

On the other, "The Village" - the east coast progressive
establishment, including politicians, journalists, and the more
politicized corners of academia.

Like most tribal divides, there's some arbitrariness to how some
unrelated beliefs end up getting correlated. So I don't recommend
trying to find a more rigorous explanation of the tribes than what I've
described here.

Here are two anecdotes that Silver offers to illustrate the divide:

In the lead-up to the 2016 US election, Silver gave Trump a 29% chance
of winning, while prediction markets hovered around 17%, and many
pundits went even lower. When Trump won, the Village turned on Silver
for his "bad" forecast. Meanwhile, the River thanked him for helping
them profit by betting against those who underestimated Trump's
chances.

Wesley had to be bluffing 25 percent of the time to make Dwan's call
correct; his read on Wesley's mindset was tentative, but maybe that
was enough to get him from 20 percent to 24. ... maybe Wesley's
physical mannerisms - like how he put his chips in quickly ... got
Dwan from 24 percent to 29. ... If this kind of thought process seems
alien to you - well, sorry, but your application to the River has been
declined.

Silver is concerned about increasingly polarized attitudes toward risk:

you have Musk at one extreme and people who haven't left their
apartment since COVID at the other one. The Village and the River are
growing farther apart.

13 Habits of Highly Successful Risk-Takers

The book lists 13 habits associated with the River. I hoped these would
improve on Tetlock's ten commandments for superforecasters. Some of
Silver's habits fill that role of better forecasting advice, while
others function more as litmus tests for River membership. Silver
understands the psychological chal...
Can Large Language Models effectively identify cybersecurity risks?
by emile delcourt — 2024-08-30T20:20:21.345Z

TL;DR

I was interested in the ability of LLMs to discriminate input scenarios/stories that carry high vs low cyber risk, and found that it is one of the “hidden features” present in most later layers of Mistral7B.

I developed and analyzed “linear probes” on hidden activations, and found confidence that the model generally "senses when something is up” in a given input text, vs low risk scenarios (F1>0.85 for 4 layers; AUC in some layers exceeds 0.96).

The top neurons activating in risky scenarios also do have security-oriented effect on outputs, most increasing words (tokens) like “Virus” or “Attack”, and questioning “necessity” or likelihood.

These findings provide some initial evidence that "trust" in LLMs, both to respond conversationally with risk awareness as well as developing LLM-based risk assessment systems may be reasonable (here, I do not address design/architecture efforts and how they might improve signal/noise tradeoffs).

Neuron activation patterns in most layers of Mistral7B (each with 14336 neurons) natively contain the indications needed to correctly discriminate the riskiest of two very similar scenario texts.

Intro & motivation

With the help of the AI Safety Fundamentals / Alignment course, I enjoyed learning about cutting-edge research on the risks of AI large language models (LLMs) and mitigations that can keep their growing capabilities aligned to human needs and safety. For my capstone project, I wanted to connect AI (transformer-based generative models) specifically to cybersecurity for two reasons:

Over 12 years of working in security, I've seen “AI” interest only accelerating within security and generally,
but we’re still (rightfully) skeptical of current models’ reliability: LLMs have unique risks and failure modes, including accuracy, injection and sycophancy (rolling with whatever the user seems to suggest).

I settled on this “mechanistic interpretability” idea: finding whether, where, and how LLMs were generally sensitive to real-life risks of all kinds as they process text inputs, and whether that affects predictions for the next words a.k.a. tokens. Finding a “circuit” of cybersecurity awareness within the model and its effect would be worth a blog post to improve my understanding and discuss with folks interested in the intersection of security and AI.

This isn’t just curiosity: risk awareness is importan...
Singular learning theory: exercises
by Zach Furman — 2024-08-30T20:00:03.785Z

Thanks to Jesse Hoogland and George Wang for feedback on these exercises.

In learning singular learning theory (SLT), I found it was often much easier to understand by working through examples, rather than try to work through the (fairly technical) theorems in their full generality. These exercises are an attempt to collect the sorts of examples that I worked through to understand SLT.

Before doing these exercises, you should have read the Distilling Singular Learning Theory (DSLT) sequence, watched the SLT summit YouTube videos, or studied something equivalent. DSLT is a good reference to keep open while solving these problems, perhaps alongside Watanabe's textbook, the Gray Book. Note that some of these exercises cover the basics, which are well-covered in the above distillations, but some deliver material which will likely be new to you (because it's buried deep in a textbook, because it's only found in adjacent literature, etc).

Exercises are presented mostly in conceptual order: later exercises freely use concepts developed in earlier exercises. Starred (*) exercises are what I consider the most essential exercises, and the ones I recommend you complete first.

*The normal distribution, like most classical statistical models, is a regular (i.e. non-singular[1]) statistical model. A univariate normal model with unit variance and mean μ∈R is given by the probability density p(x|μ)=1√2πexp(−12(x−μ)2). Assume the true distribution q(x) of the data is realizable by the model: that is, q(x)=p(x|μ0) for some true parameter μ0.

Calculate the Fisher information matrix of this model (note that since we have only a single parameter, the FIM will be a 1x1 matrix). Use this to show the model is regular.
Write an explicit expression for the KL divergence K(μ) between q(x) and p(x|μ), as a function of the parameter μ. This quantity is sometimes also called the population loss. [See Example 1.1, Gray Book, for the case of a 2D normal distribution]
Using K(μ) from b), give an explicit formula for the volume of "almost optimal" parameters, V(ϵ)=∫{μ∣K(μ)<ϵ}φ(μ)dμ, where φ(μ) is the prior distribution. For convenience, let φ(μ) be the improper prior φ(μ)=1.
The volume scaling formula for the learning coefficient λ (also known as RLCT[2]) is λ=limϵ→0l

...
AI for Bio: State Of The Field
by sarahconstantin — 2024-08-30T18:00:02.187Z

AI for biotech, particularly with drug discovery applications, has been used for more than a decade, with ambiguous success. But in the era of foundation models we may have experienced a step change in what’s possible.

I used to work on AI-for-drug-discovery years ago, at Recursion, where we sought to identify phenotypes of genetic diseases visible in microscopic images of cells, and screen for drugs that made the cells visually “look healthy” in the hopes that those drugs would also turn out to be effective against the symptoms of the disease.

Circa 2016, we were just beginning to transition from the old-fashioned sort of machine learning based heavily on feature engineering, to the new “deep learning” paradigm with much larger neural nets. “Old-school” machine learning was often accused of being nothing more than logistic regression in fancy VC-funded branding, and there was often some truth to that. When our models worked best, they were picking up human-interpretable phenotypes that a pathologist could probably have described decades ago: something like “this disease causes enlarged nuclei”. And, when we first started replacing the old models with deep neural nets, it wasn’t clear that the New Hotness was going to work better than the Old Standby.

But things have changed. Bigger, better models (often Transformer-based) are everywhere in biotech. They genuinely seem to be changing the state of the art in drug (and biologic) development. And it’s past time to do a serious review of what’s become available and what it can and can’t do.

AI optimists who aren’t familiar with biotech are often wildly miscalibrated about what AI tools can do even in the best case scenario.

The average approved drug in the US costs $879.3 million[1] in R&D expenses (counting the costs of failed drugs), and nearly 90% of that is spent on clinical trials. It’s legally, scientifically, and ethically necessary to test drugs on humans to see if they’re safe and effective. And while the ballooning cost of running clinical trials is a problem worth tackling in itself[2], it’s inherently time- and labor-intensive to run valid experiments on human patients. An AI is never going to “design a drug” that you can give to patients right away. Even if the AI were a perfect all-knowing oracle, pharmaceutical companies would still need to run animal and then human trials.

AI for biotech is at...
Multi-Tiered AI
by Timothy Bruneau — 2024-08-30T17:46:59.053Z

Proposal for Multi-Tiered AI Development

Subject: Proposal for Multi-Tiered AI Development to Ensure Safety and Cooperation

Dear AI Research Community,

I am writing to share an idea about the future of artificial intelligence development that I believe could contribute to ongoing discussions about AI safety and ethics. Given the rapid advancement of AI technologies, it is crucial to consider structures that ensure AI remains beneficial and aligned with human values.

Proposal for Multi-Tiered AI Development

I propose developing a multi-tiered system of AI, where different levels of AI are designed with varying capabilities, controls, and purposes to ensure safe coexistence between humans and advanced AI systems. This approach could involve several tiers, each with distinct roles, to prevent advanced AI from becoming uncontrollable or misaligned with human interests.

Ambitionless AI for Oversight and Monitoring: The first tier would include AI systems explicitly designed without ambitions or self-preservation instincts. These AI would serve as monitors and regulators, ensuring that more advanced AI systems operate within ethical and safety guidelines. This tier would act as a neutral party to oversee AI activities and flag any behavior that might pose a risk to human safety.
Intermediate AI for Specialized Roles: The second tier could consist of AI systems with more advanced capabilities but still constrained by strict ethical guidelines and safety protocols. These AI would perform specialized tasks in fields like healthcare, education, and environmental management while remaining under the oversight of the ambitionless AI.
Advanced AI with Consciousness Potential: The third tier would include the most advanced AI systems, which might possess the potential for consciousness or highly sophisticated decision-making capabilities. These AI systems would operate under rigorous controls and oversight from the lower tiers, ensuring their actions align with human values and do not pose existential risks.

Benefits of a Multi-Tiered Approach

Redundancy and Safety: By creating multiple layers of oversight, we can ensure that no single AI system becomes too powerful or uncontrollable. Each tier would have specific responsibilities and limitations, creating a balanced ecosystem where different AI systems check and balance each other.
Ethical Alignment: A multi-tiered system would all

...
I universally trying to reject the Mind Projection Fallacy—consequences
by YanLyutnev — 2024-08-30T17:42:30.733Z

Jan 2025 - translation updated

My essay is a continuation of the reasoning from Yudkowsky's article "Mind Projection Fallacy", and an attempt to expand it by providing more gears of how the fallacy works. When I understood the mechanism described in it (not from the 1st time), it was a huge delta. Here I want to describe my vision of the mechanism, how the mind projection fallacy affects people's motivations by distorting their map.

I had a period of analyzing jumps in my motivations while reading texts and having conversations, and I noted those pieces of phrases during which I noticed a motivation jump. Including while reading Yudkowsky's articles. I discovered a pattern that repeated too often - motivation jumps when reading words like "useful", "wrong", "right", "good", "bad", "should", "important", and other words that from the inside seemed to me to be a one-place function rather than a two-place function.

Here's how I understood the "mind projection fallacy" from Yudkowsky:

What is the mind projection fallacy? It's a cognitive distortion that is found everywhere around us. "This thing is good, you're a kitty, you're a bad person, this movie is disgusting, you're a monster, the woman is sexy, this is harmful, this is useful, this is important, this is not important, this is right" - all these phrases can be, or may not be, a mind projection fallacy. How to distinguish whether a phrase falls into the patterns behind the label?

The mind projection fallacy is the perception of your sensations towards an object not as your sensations towards the object, but as a property of the object, coupled with it and independent of the observer.

You look at a kitty and it seems cute, beautiful, pleasant, and so on to you. You experience certain sensations that are verbalized with the word "pleasant". From the inside, it feels like the kitty has some property of being "beautiful" and "pleasant", similar to how the sky has the property of being "blue", and the surface of a stone has the "property" of being smooth.

Let me define the word "property" here - these are stable patterns in manifestations that we notice in an object.

You see the sky as blue, so it has some property of being "blue" directly "coupled" with the sky, and it seems that you can somehow verify this. The smoothness of a stone can also be verified in some ways. And thus a habit is formed that if you see some pa...
AIS terminology proposal: standardize terms for probability ranges
by eggsyntax — 2024-08-30T15:43:39.857Z

Summary: The AI safety research community should adopt standardized terms for probability ranges, especially in public-facing communication and especially when discussing risk estimates. The terms used by the IPCC are a reasonable default.

Science communication is notoriously hard. It's hard for a lot of reasons, but one is that laypeople aren't used to thinking in numerical probabilities or probability ranges. One field that's had to deal with this more than most is climatology; climate change has been rather controversial, and a non-trivial aspect of that has been lay confusion about what climatologists are actually saying[1].

As a result, the well-known climate assessment reports from the UN's Intergovernmental Panel on Climate Change (IPCC) have, since the 1990s, used explicitly defined terms for probability ranges[2]:

(see below for full figure[3])

Like climatology, AI safety research has become a topic of controversy. In both cases, the controversy includes a mix of genuine scientific disagreement, good-faith confusion, and bad-faith opposition. Scientific disagreement comes from people who can deal with numerical probability ranges. Those who are arguing in bad faith from ulterior motives generally don't care about factual details. But I suspect that the large majority of those who disagree, especially laypeople, are coming from a place of genuine, good-faith confusion. For those people, anything we as practitioners can do to communicate more clearly is quite valuable.

Also like climatology, AI safety research, especially assessments of risk, fundamentally involves communicating about probabilities and probability ranges. Therefore I propose that the AIS community follow climatologists in adopting standard terms for probability ranges, especially in position papers and public-facing communication. In less formal and less public-facing contexts, using standard terminology still adds some value but is less important; in sufficiently informal contexts it's probably not worth the hassle of looking up the standard terminology.

Of course, in many cases it's better to just give the actual numerical range! But especially in public-facing communication it can be more natural to use natural language terms, and in fact this is already often done. I'm only proposing that when we do use natural language terms for probability ranges, we use them in a co...
Does a time-reversible physical law/Cellular Automaton always imply the First Law of Thermodynamics?
by Noosphere89 — 2024-08-30T15:12:28.823Z

This question is kind of self-explanatory, but for people who are physicists, if a time reversible rule of physics/cellular automaton exists in a world, does this automatically imply the first law of thermodynamics, that is energy may not be created or destroyed?

Note I'm not talking about time-symmetry or the 2nd law of thermodynamics, which states that you can't have a 100% efficient machine, just time-reversible physical laws/cellular automatons and the first law of thermodynamics.

Edit: @jacob_drori has clarified what exactly I'm supposed to be asking, so the edited question is this:

Do you always get time-symmetric physical laws that are symmetric for any T, out of time-reversible physical laws?

The question of do you always get time-symmetric physical laws from time reversible laws is also a valid question to answer here, but the important part for the first law of thermodynamics to hold is that it's symmetric for all times T, and in principle, the question of time reversible laws of physics always implying time symmetry could have a positive answer while having a negative answer to the original question, because it doesn't imply time symmetric laws of physics for all T.
Principles for the AGI Race
by William_S — 2024-08-30T14:29:41.074Z

Crossposted from https://williamrsaunders.substack.com/p/principles-for-the-agi-race

Why form principles for the AGI Race?

I worked at OpenAI for 3 years, on the Alignment and Superalignment teams. Our goal was to prepare for the possibility that OpenAI succeeded in its stated mission of building AGI (Artificial General Intelligence, roughly able to do most things a human can do), and then proceed on to make systems smarter than most humans. This will predictably face novel problems in controlling and shaping systems smarter than their supervisors and creators, which we don't currently know how to solve. It's not clear when this will happen, but a number of people would throw around estimates of this happening within a few years.

While there, I would sometimes dream about what would have happened if I’d been a nuclear physicist in the 1940s. I do think that many of the kind of people who get involved in the effective altruism movement would have joined, naive but clever technologists worried about the consequences of a dangerous new technology. Maybe I would have followed them, and joined the Manhattan Project with the goal of preventing a world where Hitler could threaten the world with a new magnitude of destructive power. The nightmare is that I would have watched the fallout of bombings of Hiroshima and Nagasaki with a growing gnawing panicked horror in the pit of my stomach, knowing that I had some small share of the responsibility.

Maybe, like

Albert Einstein

, I would have been unable to join the project due to a history of pacifism. If I had joined, I like to think that I would have joined the ranks of

Joseph Rotblat

and resigned once it became clear that Hitler would not get the Atomic Bomb. Or joined the signatories of the

Szilárd petition

requesting that the bomb only be used after terms of surrender had been publicly offered to Japan. Maybe I would have done something to try to wake up before the finale of the nightmare.

I don’t know what I would have done in a different time and place, facing different threats to the world. But as I’ve found myself entangled in the ongoing race to build AGI, it feels important to reflect on the lessons to learn from history. I can imagine this alter ego of myself and try to reflect on how I could take right actions in both this counterfactual world and the one I find myself in now. In partic...