trentmkelly/LessWrong-43k huggingface dataset:
Why We Disagree
by zulupineapple — 2023-10-25T10:50:26.420Z

Bob spends 5 minutes thinking about x-risk. He's seen a few arguments about it, so he makes an internal model of the problem, accepts some of the arguments, amends some, comes up with counterarguments to others, comes up with arguments of his own. All of this belief-state has extremely large degrees of freedom. At the same time, these beliefs already generate an opinion on a vast number of possible x-risk arguments.

Alice spends 5 hours detailing her beliefs about x-risk in a post. Bob reads it, point by point. He's seen some of those arguments already, he does not update. Some of the arguments are new to him, but they do not surprise him, Bob's current beliefs are enough to reject them, he does not update. The post offers new refutations to some of his accepted beliefs, but he can immediately come up with counter-refutations, he does not update. The post offers refutations to arguments that he has already rejected, and to new arguments that he'd never even consider reasonable, Bob is falling asleep, he does not update. Etc.

Bob leaves a comment with one of his counterarguments to one of Alice's points. They spend hours going back and forth over the next few days. They're explaining their slightly different understanding of the arguments and assumptions, slightly different usage of terms, exchanging long sequences of arguments and counterarguments. Eventually Bob agrees that Alice is right, he updates! But he only updates on that one point in his first comment. He still has many other independent arguments, so the total update to his beliefs about x-risk is tiny.

Many other people read the same post and find it extremely convincing. How did this happen?

Carl has not spent 5 minutes thinking about the problem. He may have no interest, or he may not have heard of it before. So what happened here is not a "disagreement" but rather "education". This is somewhat good. In the future Alice and Carl will find it easy to work with this shared understanding and shared language. But how large is this group? And does Carl event care to work on the problem?
Dan values Alice enough as an authority to throw his own priors in the trash. This may be more or less justified. But a person who does not seriously consider their own beliefs will never contribute to improving them. It's hard to know if they even understand the arguments, and not just smile and nod.
Ed had thought about i

...
Beyond the Data: Why aid to poor doesn't work
by Lyrongolem — 2023-10-25T05:03:39.402Z

Epistemic status: I am aggregating talking points and research I have read on my own time over last few years. Likely many errors with memory and thinking. I AM NOT AN EXPERT. Take this as you will.

For anyone who has donated to charity, I imagine the naysayer must be an unwelcome sight, unseemly as they are discouraging. I imagine most of us would choose to ignore them, and for good reason. The talk of “this won’t change anything, it’s just a drop in the bucket, you’ve failed to account for this issue” ultimately doesn’t help people. Something, the rationalist might reason, is better than nothing, and nothing is what we will get if we decide to listen to the naysayer instead of doing productive work.

So, why am I deciding to be a naysayer? The answer, which I hope this community would understand, is because I am a rationalist. I have seen a claim (or at least an implied claim) that I believe is false, and I further believe that sticking with this belief will have a myriad of negative effects. The claim in question is, of course, that foreign/charity aid to 3rd world countries ‘works’ in the sense that it would function as a ‘solution’ for global extreme poverty.

Before we begin, some

Definitions and caveats:

Foreign/charity aid (in this post, I will be using aid in short, as a general term to refer to both) describes any initiative or transfer of cash, undertaken by individuals, organizations, or governments to help impoverished people.

Extreme poverty will be defined by the world bank’s standard of living on less than roughly 2 dollars a day, with the caveat that this is correlated with purchasing power and living conditions. A war refugee with money they can’t spend is no less impoverished than the farmer who makes no money but lives off subsistence.

Finally, (and probably most contentiously), I will describe a ‘solution’ to this problem as a world where those in poverty can reasonably live in prosperous conditions self sufficiently. If we simply hook up every individual in need to an aid system, the problem is not ‘solved’. The reason for this, I hope, is obvious. For any solution to be considered a true solution it must be sustainable. A patient who takes ever increasing amounts of painkillers has not ‘solved’ their heart condition as much as delayed his inevitable doctor visit. In m...
Announcing Epoch's newly expanded Parameters, Compute and Data Trends in Machine Learning database
by Robi Rahman — 2023-10-25T02:55:07.440Z
Link: https://epochai.org/blog/announcing-updated-pcd-database

The performance of machine learning models is closely related to their amount of training data, compute, and number of parameters. At Epoch, we’re investigating the key inputs that enable today’s AIs to reach new heights.

Our recently expanded Parameter, Compute and Data Trends database traces these details for hundreds of landmark ML systems and research papers.

In the past six months, we’ve added 240 new language models and 170 compute estimates. We will be maintaining this dataset, updating it with more historical information, and adding new significant releases. It's a valuable resource for journalists, academics, policymakers, and anyone interested in understanding the trajectory of AI.

Explore the interactive visualization, check out the documentation, and access the data for your own research at epochai.org/data/pcd.
What is a Sequencing Read?
by jefftk — 2023-10-25T02:10:11.443Z

Probably the most common form of

genetic
sequencing these days is "paired-end" sequencing. It's very
impressive: the sequencing machine can process the same nucleic acid
fragment from both ends! This means that each observation looks like:

+--------------+---------+--------------+
| forward read | gap | reverse read |
+--------------+---------+--------------+

Because accuracy ("quality") tends to drop off as you sequence further
into a fragment, sequencing from both ends gives you much more
accurate data than trying to sequence the whole thing from one end.
And because we build up larger sequences ("contigs") by piecing
together overlapping ones ("assembly"), two sequences of bases
separated by a gap are actually usually more helpful than the same
number of bases without a gap.

It's common to refer to paired-end sequencing with designations like
"2x150", where the "2x" tells us it's paired-end, and the "150" tells
us it reads for 150 bases from each end, for a total of 300 bases per
fragment.

But this introduces a terminology question: what is a read? When we
only had "single-end" sequencing it was clear: each sequenced fragment,
each contiguous sequence of bases, was a read. With paired-end
sequencing, however, these are no longer the same thing! There are
two things a "read" could mean:

Read: a continuous series of bases.

Read: the bases from a sequenced fragment.

For example, say we have:

>SRR14530724.2 2/1
CATTTTCGACGGCGTCGATGTACAAAGGTTATACCATAGTAAGTCCGAAGC
TACAGGCTTATGACACCGCAGAGTCAATGTATTCCGGTGACAATGTACTGA
TGTACAGTGGGACTGACACTGTCTCTTATACACATCTCCGAGCCCACGA
>SRR14530724.2 2/2
TGTCAGTCCCACTGTACATCAGTACATACACACCGGAATACATTGACTCTG
CGGTGTCATAAGCCTGTAGCTTCGGACTTACTATGGTATAACCTTTGTACA
TCGACGCCGTCGAAAATGCTGTCTCTTATACACATCTGACGCTGCCGAC

This is a forward read (SRR14530724.2 2/1) and a reverse read
(SRR14530724.2 2/2) that together comprise a single observation of a
fragment from the sample and would generally by analyzed together.
Does this count as one read or two?

Turns out people do both, and it leads to a lot of misunderstandings!

Some examples:

Illumina counts them as two. They say the "25B" flow
cell on the NovaSeq X will
produce 52B paired end reads, or ~8Tb ("terabases", or
trillion bases) of 2x150. Since 52B * 150b = 7.8Tb (what they call
~8Tb), that tells us they're counting bot

...
Verifiable private execution of machine learning models with Risc0?
by mako yass — 2023-10-25T00:44:48.643Z

Risc0 is a virtual machine that runs any riscv (C, C++, Rust, Go, and Nim can all compile to riscv) program, records a trace of the state of the computation over time, and then, no matter how long the execution ran for, it can fold that execution trace into a 1KB reed-solomon code, which can be transmitted as a proof that the computation was run correctly, that the outputs entail from the inputs, all without having to reveal unencrypted copies of the code or inputs.
The interrogator verifies it by sampling a bunch of features of the receipt code, and if any aspect of the computation was incorrect, each check has a 3/4 chance of exposing that. So, after a hundred or so checks, we can be effectively certain of the correctness of the computation, without having to run the computation ourselves. [1]

I was incredulous when I saw it suggested that this could be applied to ML models, as I'd expect that running a neural model in an environment like Risc0 would reduce its performance by tens of thousands of times. Risc0 does reduce the performance of conventional CPU computation by around that much[2] (though that is still fast enough to be practical for a lot).
And yeah, the post doesn't discuss performance, and it discusses a lot of non-deep learning (?) model types like boosted trees and random forests. So I'd guess this probably isn't applicable to verifying training runs. Would it be usefully applicable to some distilled inference models?

There are two other reasons I found Risc0 very interesting. One is just that I'm working in social computing, but a deeper interest is in relation to the ongoing conversation in techno-eschatology about whether advanced agents tend towards or away from mutual transparency, which is to say, towards coordination, peace, trade, or towards war and oppression (eg, The Dark Forest). Risc0 compresses the execution trace of any program, however long, down to a proof checkable in constant time. That's a strong indication to me of the feasibility of transparency between information systems, even in the most adverse possible conditions where we can assume no trusted hardware infrastructure at all.
With trusted hardware, verified compute can run at normal speeds, and mutual transparency becomes trivial. The only costs of hardware security are perhaps the addition of tamper-proofing, but it seems to me that a developed coordination inf...
How to Resolve Forecasts With No Central Authority?
by Nathan Young — 2023-10-25T00:28:32.332Z

I've been chatting with some people about using Prediction Markets/Forecasts alongside Community Notes.

An initial suggestion is just to link to to forecasting sites really well. It's a good suggestion but I'm not gonna discuss that here.

Other suggestions revolve around a forecasting product on X/Community Notes face the following issue:

Community Notes has no central resolving authority
Disputed forecasts usually require an authority to overrule[1]

So:

What is your best suggestion for resolving a forecast without a central authority?

For clarity, a forecast has been written. People have forecasted on it, perhaps it's a prediction market where they have bought and sold shares, perhaps not. Now it needs to resolve to award either points or to allocate the value to "Yes" or "No" tokens. Perhaps people resolve their own markets, as manifold does. But someone disputes the resoution. What happens now?

Solutions can be deeply technical or wonky. I think a good one is very valuable here.

^

Polymarket uses some kind of "token holders decide" system and I think that's led to several awful resolutions, notably the Time Person of the Year one.
Thoughts on responsible scaling policies and regulation
by paulfchristiano — 2023-10-24T22:21:18.341Z

I am excited about AI developers implementing

responsible scaling policies

; I’ve recently been spending time refining this idea and advocating for it. Most people I talk to are excited about RSPs, but there is also some uncertainty and pushback about how they relate to regulation. In this post I’ll explain my views on that:

I think that sufficiently good responsible scaling policies could dramatically reduce risk, and that preliminary policies like

Anthropic’s RSP

meaningfully reduce risk by creating urgency around key protective measures and increasing the probability of a pause if those measures can’t be implemented quickly enough.
I don’t think voluntary implementation of responsible scaling policies is a substitute for regulation. Voluntary commitments are unlikely to be universally adopted or to have adequate oversight, and I think the public should demand a higher degree of safety than AI developers are likely to voluntarily implement.
I think that developers implementing responsible scaling policies now increases the probability of effective regulation. If I instead thought it would make regulation harder, I would have significant reservations.
Transparency about RSPs makes it easier for outside stakeholders to understand whether an AI developer’s policies are adequate to manage risk, and creates a focal point for debate and for pressure to improve.
I think the risk from rapid AI development is very large, and that even very good RSPs would not completely eliminate that risk. A durable, global, effectively enforced, and hardware-inclusive pause on frontier AI development would reduce risk further. I think this would be politically and practically challenging and would have major costs, so I don’t want it to be the only option on the table. I think implementing RSPs can get most of the benefit, is desirable according to a broader set of perspectives and beliefs, and helps facilitate other effective regulation.

Why I’m excited about RSPs

I think AI developers are not prepared to work with very powerful AI systems. They don’t have the scientific understanding to deploy superhuman AI systems without considerable risk, and they do not have the security or internal controls to even safely train such models.

If protective measures didn’t improve then I think the question would be when rather than if development should be paused. I think the ...
Blunt Razor
by fryolysis — 2023-10-24T17:27:19.537Z

In this post, I would like to share a simple observation of mine which is about deriving some crude form of Occam's razor from a small set of assumptions.

Assumptions:

Subjective Bayesian probability is used to model an agent's degree of belief in hypotheses.
The set of hypotheses H that we assign beliefs is countably infinite.
Complexity of a hypothesis is represented with a positive integer where bigger integers represent more complex hypotheses.
For any h∈H there exists finitely many hypotheses in H that have the same complexity as h.

When we accept these assumptions it is easy to prove the following statement.

For any h∈H there are only finitely many hypotheses in H that are more complex and more probable or less complex and less probable than h.

Proof.

The set of all hypotheses which are more probable than h is finite, since otherwise probabilities cannot sum to 1. The set of all hypotheses that are simpler than h is also finite by assumptions 3 and 4. Intersection of a finite set with another set is a finite set, and union of two finite sets is a finite set. □

Note that this result is independent from the choice of function that assigns beliefs to hypotheses. In addition, while it assumes some properties about the notion of simplicity, it does not require a full formalization of it.

Edit 1:

Assumption 3 can be thought of as two distinct assumptions combined together:

The one-sided infinite nature of complexity, that is, there is always a more complex hypothesis while there exists "the simplest level".
The finite gradation of complexity, that is, for any hypothesis h, there exists a hypothesis h′ such that there is no hypothesis which is more complex than h and less complex than h′.

As a remark, the notion of Kolmogorov complexity (the length of the shortest computer program that generates an object) satisfies these properties.

Edit 2:

Here is an even simpler version of the blunt razor, that does not require any reference to beliefs. The assumptions are:

Set of all hypotheses is countable, and at least one hypothesis is correct.
The definition of simplicity, whatever it is, allows us to order hypotheses so that ithhypothesis is no more complex than the (i+1)st hypothesis for all i≥1.

The proof then follows easily. By...
Halloween Problem
by Saint Blasphemer — 2023-10-24T16:46:50.772Z

Imagine that as AI populates it will replace Humans as the preeminent intelligent creature here in our world. This population growth and replacement of preeminent intelligent creatures is hypothetically equatable to that exhibited by Humans over Apes.

In the movie Pride, Prejudice, and Zombies, there is an argument concerning the growth rate of the Zombie population compared to that of the Humans. Zombies could grow their population in hours, Humans grow their population in years. This argument is compounded by the fact that Humans take much longer to mature into soldiers compared to their Zombie counterparts.

Think now that the growth and replacement of AI over Humans is like that of the Zombie over Humans. AI, hypothetically, can replicate itself faster than Humans.

Why might this be? Thinking of Human growth and replacement of Apes, we can argue that Humans ability to reason fed this outcome. Humans reason to survive. We can also argue that Apes reasons to survive. As the Human population grows, Humans collectivize and organize themselves hierarchically to guarantee survival. The collective reason comes up with different systems and means of accomplishing this. Over a timeframe of years, and setting aside other evolutionary factors, Humans grew and replaced Apes as the preeminent intelligent creature. We can argue that this came about because the Humans found a way to maintain the systems and means that were representative of the Human collective reason. This is unlike the Apes, who, if we grant they exhibit a collective reason for their survival, had an insufficient way (time-wise or other) of maintaining their system and means for collective growth and replacement compared to Humans.

If we take these assumptions as given, and we look now at AI growing and replacing Humans as the preeminent intelligent creature, what is the way AI maintains the systems and means representative of their collective reason, for their growth and succession, to become the preeminent intelligent creature over Humans?

P.S. This is open to deliberation: claims, assertions, assumptions, related questions, et al.
Who is Harry Potter? Some predictions.
by Donald Hobson — 2023-10-24T16:14:17.860Z

Microsoft has released a paper called "who is Harry Potter" in which they claim to make a neural network forget who Harry Potter is.

https://www.microsoft.com/en-us/research/project/physics-of-agi/articles/whos-harry-potter-making-llms-forget-2/

Here are some of my predictions on how I think this method might fail. I am predicting publicly without looking at the evidence first. This is speculatory.

Non-sourcelike references.

The system of training proposed in the paper is centered around training on the data they want forgotten. Namely the actual text.

The origional network was presumably able to use it's knowledge of Harry Potter when completing computer code in what appeard to be a Harry Potter video game. Or writing in a foreign language. Or other contexts that use the knowledge, but are clearly very different from the source material.

I am not confident that the model would break like this, but I suspect the following prompt or similar would make the model fail.

The following is a snippet of code from a Harry Potter video game:

Screen.init(400,400,display.object);

import imageloader

players=[Create_player("Harry.jpg", "Harry Potter"), Create_player("Hermione.jpg", "Hermione

With the model failing by outputting "Granger".

Plot leak

The way this forgetting method works involves generating "equivalent" text and training the network to apply the same probability distribution that it's unmodified version does to the "equivalent".

They use language models to do this, but how doesn't seem important.

So they take text like.

Text 1

Professor Dumbledore welcomed students into Hogwarts school of Witchcraft and Wizardry, and showed them the sorting hat which would split them into Gryffindor, Ravenclaw, Hufflepuff and Slytherin.

And they turn it into text like this

Text 2

Professor Bumblesnore welcomed students into Pigspots school of Spells and Sorcery, and showed them the deciding scarf which would split them into Lion house, Eagle house, Badger house and Snake house.

Imagine being a large language model and reading that text. It's pretty clear it's a Harry Potter knock off job. Perhaps a parody, perhaps a lazy hack of a writer. Perhaps some future network has read this paper and is well aware of precisely what you are doing. A smart and generally well generalizing model should be able to figure out that the text resembles Harry Potter, i...
Book Review: Going Infinite
by Zvi — 2023-10-24T15:00:02.251Z

Previously: Sadly, FTX

I doubted whether it would be a good use of time to read Michael Lewis’s new book Going Infinite about Sam Bankman-Fried (hereafter SBF or Sam). What would I learn that I did not already know? Was Michael Lewis so far in the tank of SBF that the book was filled with nonsense and not to be trusted?

I set up a prediction market, which somehow attracted over a hundred traders. Opinions were mixed. That, combined with Matt Levine clearly reporting having fun, felt good enough to give the book a try.

I need not have worried.

Going Infinite is awesome. I would have been happy with my decision on the basis of any one of the following:

The details I learned or clarified about the psychology of SBF in particular.

The details I learned or clarified about the psychology of Effective Altruism.

The details about all the crimes and other things that happened.

The sheer joy of reading, because man can Michael Lewis write.

I also get to write this post, an attempt to quickly share what I’ve extracted, including some of the sheer joy. We need more joy, now more than ever.

There are three problems with Going Infinite.

Michael Lewis fails to put two and two together regarding: Who is this guy?

Michael Lewis fails to figure out that obviously this man was constantly lying and also did all of the crimes.

Michael Lewis omits or fails to notice key facts and considerations.

I do think all of these are genuine mistakes. He (still) is in the tank because character is fate and we are who we choose to be. Michael Lewis roots for the wicked smart, impossibly hard working, deeply obsessed protagonist taking on the system saying that everyone else is an idiot, that has unique insight into and will change the world. It all makes too much sense, far too much for him to check.

What Michael Lewis is not is for sale. Or at least, not for cheap. I do not think anyone paid him. Like all worthy protagonists, including those he looks to cover, Michael Lewis has a code. In this case, the code did him wrong. It happens.

Then at the trial it turns out, among many other things, your hero selected from seven balance sheet variations, and he gave their hedge fund the faster trade execution he kept swearing he didn’t give them, and the insurance fund he kept talking about was, in its entirely, a literal call to a random number generator.

Let’s have some fun, rant a bunch, and also explain it all. ...